Expert guide to speculative decoding for accelerating LLM inference: draft model selection, acceptance rates, rejection sampling, tree-based speculation… From the article "Speculative Decoding: Theory and Implementation" by Synor, published on Misar.Blog.
In this episode:
0:00 Speculative Decoding Accelerates LLM Inference by Using a Draft Model
0:49 At Acceptance Rates of 70-90% Speculative Decoding Yields 1.5-3× End-To-End
1:24 The Speedup Formula Is (Γ × Acceptance_rate)/
1:59 Rejection Sampling Guarantees the Output Distribution Is Identical
2:32 Draft Model Family Matters
3:05 Optimal Speculation Length Γ Is 4-8 Tokens
3:16 Tree-Speculation
Links from this episode:
Assisters.dev
MisarReach
Misar.AI
Surfer
Kit
MailerLite
Some of the links above are affiliate links. If you buy through them, Misar.Blog may earn a commission at no extra cost to you.
More episodes:
Related reads:
Read the article: https://www.misar.blog/@synor/articles/speculative-decoding-implementation.
Read the articles by this Author: https://www.misar.blog/@synor.
Brought to you by Misar.AI: https://www.misar.ai.