What Is Speculative Decoding and Why It Makes AI Faster

Two AIs, One Answer

Speculative decoding is a speed trick for AI models. Normally, a big AI writes its answer one word at a time, slowly thinking about each word before moving on. With speculative decoding, a smaller, faster "helper" AI guesses the next few words first, and the big AI just checks whether those guesses are good. When the guesses are right, the big AI gets to write several words all at once.

Imagine a slow, careful writer working with a fast typist. The typist drafts a sentence quickly. The careful writer reads it, keeps the parts that are correct, and only fixes the parts that are wrong. The final answer is just as good as if the careful writer did every word themselves — but it took a fraction of the time.

AI That Doesn't Feel Slow

Big AI models are powerful, but they're also slow. When you ask one a question, it has to think hard about every single word it writes. That delay is what makes some chatbots feel laggy, what makes code tools freeze while you wait, and what makes running AI at scale expensive.

Speculative decoding can make the same big model run 2 to 3 times faster without changing the quality of a single answer. That means snappier apps, lower cloud bills, and AI tools that feel instant instead of sluggish. It's one of the most useful tricks in modern AI engineering — and you can use it without retraining anything.

The Draft-and-Verify Loop

Speculative decoding uses two models working together. The small "drafter" model is fast but less accurate. The big "verifier" model is slow but very smart. The drafter writes a few possible next words, then the verifier looks at all of them in a single pass. The verifier keeps the words it agrees with and throws out the rest. Then the drafter tries again from the last good word.

Here's the typical flow:

If the drafter guesses right most of the time, the big model can leap forward by 3-4 words per round instead of just 1. That's where the speed-up comes from — the big model isn't doing less work, it's just doing the same work in fewer, larger steps.

Speculative Decoding in Code

Here's a simple Python example showing the idea. The real libraries (like vLLM, Hugging Face Transformers, and TensorRT-LLM) handle the math, but the pattern looks like this:

# Normal decoding: one token at a time, slow but safe
def normal_decode(prompt, big_model):
    tokens = []
    for _ in range(100):
        next_token = big_model.predict_one(prompt, tokens)
        tokens.append(next_token)
    return tokens

# Speculative decoding: draft 5, verify all, keep the good ones
def speculative_decode(prompt, big_model, small_model):
    tokens = []
    while len(tokens) < 100:
        # 1. Small fast model drafts K tokens at once
        draft = small_model.predict_k(prompt, tokens, k=5)

        # 2. Big model checks every draft token in one pass
        accepted = big_model.verify_batch(prompt, tokens, draft)

        # 3. Keep the accepted prefix, throw out the rest
        tokens.extend(accepted)
    return tokens

The verify_batch call is the magic. Even though the big model is still doing the real thinking, it gets to review 5 words at once instead of picking just 1. As long as the drafter is right most of the time, the whole pipeline finishes much faster — and the answer is exactly the same as if the big model had worked slowly the entire time.

Knowledge Check

Test what you learned with this quick quiz.