What is speculative decoding, and will it make your local LLM faster?

The mechanism, in one pass

Ordinary generation commits one token per forward pass of the model. To produce four tokens the model runs four times, each run seeing everything produced so far. That serial loop is what limits generation speed on most local setups.

Speculative decoding splits the job into proposing and checking. A small draft model proposes a short run of candidate tokens quickly. The main model then evaluates that entire run in one forward pass and decides, position by position, whether it agrees.

Suppose the prompt is a weather sentence and the draft proposes four tokens. The main model accepts the first two, disagrees at the third and substitutes its own token there, and discards the fourth because it came after the first disagreement. Three committed tokens for one pass of the main model instead of three passes. That is the entire idea.

• Draft model proposes several candidate tokens.
• Main model verifies the whole run in one pass.
• Accepted candidates commit; the first rejection is replaced by the main model.
• Everything after the first rejection is discarded.

Why it does not change your output quality

This is the part worth being precise about, because it is the difference between an optimisation and a compromise. The original paper introduces the method as exact: it accelerates sampling without changing the distribution the model samples from, and reports identical outputs alongside a 2x to 3x acceleration on the model it tested.

The reason is structural. A draft token survives only if the main model agrees with it, so nothing enters your output that the main model would not have produced on its own. LM Studio states the same constraint plainly in its documentation, and adds that after the last accepted draft token the main model always produces one additional token itself.

One requirement follows from this design: the draft and main models must share a vocabulary, because they have to be talking about the same tokens. That is why draft models are normally a smaller member of the same family. Some serving stacks now offer an opt-in cross-vocabulary mode, but it is an explicit setting rather than the default.

When it actually speeds things up, and when it does not

The speed-up comes from two multiplied factors: how much cheaper the draft model is than the main model, and how often its proposals are accepted. Both have to hold, and the second one changes prompt by prompt.

LM Studio publishes explicit sizing guidance for the largest draft model likely to help: roughly 1B for a 7B main model, 3B for a 14B main model, and 7B for a 32B main model, with no recommendation at all for a 3B main model because there is nothing meaningfully smaller to pair with it. It also states directly that if the draft model is not fast enough or not effective enough, generation speed will not increase and could decrease.

Acceptance rate is the part you cannot configure. A prompt asking for the quadratic formula has essentially one continuation, so even a very small draft model proposes what the large one would; almost every token is accepted. A prompt asking for a story continuation has hundreds of valid next words, so proposals are rejected constantly and the draft compute is wasted.

Workload shape matters too. vLLM's documentation frames the technique as reducing inter-token latency under medium-to-low request rates on memory-bound workloads, and its own selection table shows model-based methods giving high gain when optimising latency and generally lower gain when optimising throughput under heavy traffic. If your GPU is already saturated serving many concurrent users, the spare capacity that speculation exploits is not there.

Draft models, MTP, EAGLE: the same idea in different packages

The settings menus are more confusing than the concept. Everything in this family follows the same draft-and-verify loop and differs only in where the proposals come from.

A plain draft model is a separate small checkpoint you load alongside the main one. Multi-token prediction, or MTP, is an auxiliary prediction path built into the main model architecture itself, which is why it often costs less extra memory. Target-conditioned draft networks such as EAGLE-3, DFlash and DSpark are purpose-trained speculators that read the main model's internal hidden states rather than only its tokens. Lighter options such as n-gram and suffix decoding skip the extra model entirely and propose from patterns already present in the text.

For a local setup, the practical hierarchy is simple. If your model ships native MTP, that is usually the cheapest thing to try. If not, a smaller sibling model as a draft is the standard path. The purpose-trained speculators mostly matter if you are running a serving stack that supports them.

How to try it without fooling yourself

In LM Studio, switch to Power User mode or higher, load your model, and select a Draft Model in the Speculative Decoding section of the chat sidebar. Pair down a size tier within the same family, for example an 8B main model with a 1B draft.

In vLLM, pass --speculative-config as a JSON object with method, model, and num_speculative_tokens. In llama.cpp, speculative decoding is a documented server feature, so it is available underneath much of the consumer tooling as well.

Then measure properly, because this is a setting that can silently cost you speed. Fix the model, the context length, and the sampling settings, and record tokens per second on the same set of prompts with speculation off and on. Use prompts from your real work, and include at least one open-ended generation alongside your structured ones, because those two cases can move in opposite directions.

Raising the number of speculative tokens is not a free dial either. More candidates per round means more committed tokens when acceptance is high, but also more sequential drafting work that is thrown away when acceptance is low.

• Same family, same vocabulary, much smaller draft.
• Benchmark with speculation off and on, everything else fixed.
• Test a structured prompt and an open-ended prompt separately.
• Watch memory: the draft model occupies space your context could have used.

What this answer does not claim

RepoRadar has not benchmarked speculative decoding on any specific hardware. The vendor and framework figures cited here are reproduced with attribution; the published measurements come with their own disclosed hardware and software configurations, and results on your machine will differ.

Reported speed-ups in this field are typically measured on particular model pairs and particular workloads. The community record includes at least one documented case of a pairing being net-negative on consumer hardware, which is a reason to measure rather than to assume a multiplier.

The exactness property applies to the algorithm as published. It is a property of the draft-and-verify design, not a certificate that every implementation in every framework is bug-free.

A useful next action

Check whether the model you already run locally has a smaller sibling in the same family at roughly one tenth its size. If it does, that is your draft model candidate and the experiment costs one download.

Run ten of your own prompts with speculation off, then the same ten with it on, and compare tokens per second. If the gain is under about 20 percent, the memory the draft model occupies is probably worth more to you as additional context.