Speculative Decoding: Accelerating Large Language Model Inference

About this tutorial

Large Language Models (LLMs) have achieved impressive success across a wide range of applications, but deploying them efficiently remains challenging due to the high latency and computational cost of autoregressive generation. Speculative decoding has emerged as an effective solution to this challenge by using a smaller draft model to predict multiple future tokens and a larger target model to verify these predictions in parallel. In practice, speculative decoding can provide 2–4× inference speedups while maintaining the same generation quality.

This tutorial provides an overview of the key ideas, algorithms, and recent developments in speculative decoding. We will introduce how speculative generation works, discuss recent advances in draft model design, verification mechanisms, and discuss practical considerations for applying speculative decoding to real-world LLM systems. Through this tutorial, we aim to provide a comprehensive understanding of the current landscape of speculative decoding and highlight its potential for building faster and more efficient LLM applications, as well as the challenges that remain for future research.

Standard decoding 8 target-model calls

Speculative decoding makes inference much faster than before

Speculative decoding 2 target-model calls

Speculative decoding makes inference much ~~slower~~faster than before

Both rows produce the same text. Speculative decoding drafts ahead (dashed), verifies a whole block in one call (accepted, corrected), and re-drafts whatever follows a correction.

Schedule

A half-day tutorial of three hours. The outline below is provisional, and times will be filled in once the CSoNet 2026 programme is announced.

Time	Section	Presenter
TBA – TBA	Part I · Foundations Why autoregressive decoding is slow, and how draft-then-verify speeds it up without changing the output.	Nguyen Le Minh
TBA – TBA	Part II · A taxonomy of methods: drafting and verification The main families of draft models, and the ways a draft can be verified.	Le Nguyen Khang
TBA – TBA	Q & A session I
TBA – TBA	Coffee break
TBA – TBA	Part III · Advanced methods Recent algorithms that raise acceptance rates, and the limits they run into.	Do Dinh Truong
TBA – TBA	Part IV · Downstream applications & adaptations Adapting speculation to tasks beyond plain text generation.	Le Nguyen Khang
TBA – TBA	Part V · Systems, serving & practice What changes when speculative decoding meets a real serving stack.	Do Dinh Truong
TBA – TBA	Final remarks, open problems & Q & A session II	All presenters

Reading List

Bold papers are discussed in detail during our tutorial.

For a broader introduction, see the community Reading List on speculative decoding. Our own work in this area is SPECTRA (ACL 2025), AdaSpec (AAAI 2026) and UniSpec (ACL 2026).

Part I — Foundations & history

Blockwise Parallel Decoding for Deep Autoregressive Models (Stern, Shazeer & Uszkoreit, NeurIPS 2018)
Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman & Matias, ICML 2023)
Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al. (DeepMind), arXiv 2023)
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (Xia et al., Findings of EMNLP 2023)
Speculative Decoding with Big Little Decoder (Kim et al., NeurIPS 2023)
Better & Faster Large Language Models via Multi-token Prediction (Gloeckle et al., ICML 2024)
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (Xia et al., Findings of ACL 2024)
Decoding Speculative Decoding (Yan et al., NAACL 2025)

Part II — A taxonomy of methods: drafting and verification

Drafting with lightweight heads

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., ICML 2024)
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li et al., ICML 2024)
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (Li et al., EMNLP 2024)
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding (Ankner et al., COLM 2024)
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding (Du et al., ICML 2024)
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (Bhendawade et al. (Apple), EMNLP 2025)

Self-drafting and early exit

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (Zhang et al., ACL 2024)
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (Elhoushi et al. (Meta), ACL 2024)
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting (Liu et al., NeurIPS 2024)
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (Xia et al., ICLR 2025)

Training-free drafting: retrieval, suffix automata and n-grams

REST: Retrieval-Based Speculative Decoding (He et al., NAACL 2024)
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (Fu et al., ICML 2024)
SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation (Le, Do & Nguyen, ACL 2025)
SAM Decoding: Speculative Decoding via Suffix Automaton (Hu et al., ACL 2025)
Turning Trash into Treasure: Accelerating Inference of LLMs with Token Recycling (Luo et al., ACL 2025)
DReSD: Dense Retrieval for Speculative Decoding (Ruzzetti et al., Findings of ACL 2025)
RASD: Retrieval-Augmented Speculative Decoding (Quan et al., Findings of ACL 2025)

Training and aligning a drafter

DistillSpec: Improving Speculative Decoding via Knowledge Distillation (Zhou et al., ICLR 2024)
Online Speculative Decoding (Liu et al., ICML 2024)
CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter (Miao et al., ACL 2025)
FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling (Zhao et al., ACL 2025)

Verification: acceptance rules and draft trees

SpecTr: Fast Speculative Decoding via Optimal Transport (Sun et al., NeurIPS 2023)
Block Verification Accelerates Speculative Decoding (Sun et al., ICLR 2025)
Traversal Verification for Speculative Tree Decoding (Weng et al., NeurIPS 2025)
PEARL: Parallel Speculative Decoding with Adaptive Draft Length (Liu et al., ICLR 2025)

Part III — Advanced methods

State-of-the-art drafters

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (Li et al., NeurIPS 2025)
Mixture of Attentions for Speculative Decoding (Zimmer et al., ICLR 2025)
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding (Li et al., ICML 2025)
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (Hu et al., NeurIPS 2025)
Mamba Drafters for Speculative Decoding (Choi et al., Findings of EMNLP 2025)
STree: Speculative Tree Decoding for Hybrid State-Space Models (Wang et al., NeurIPS 2025)
CLaSp: In-Context Layer Skip for Self-Speculative Decoding (Wang et al., ACL 2025)
CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Acceleration (Yang et al., NeurIPS 2025)

Learning to draft better

UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (Do, Le & Nguyen, ACL 2026)
SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications (Oliaro et al., NeurIPS 2025)
AdaSpec: Adaptive Multilingual Speculative Decoding with Self-Synthesized Language-Aware Training and Vocabulary Simplification (Do, Le & Nguyen, AAAI 2026)
AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders (Hu et al., NeurIPS 2025)
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (Timor et al., ICML 2025)
Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (Zhang et al., ICLR 2026)
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026)
Flatter Tokens are More Valuable for Speculative Draft Model Training (ICLR 2026)

Multi-draft sampling and theoretical limits

Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits (Khisti et al., ICLR 2025)
Towards Optimal Multi-draft Speculative Decoding (Hu et al., ICLR 2025)
Faster Cascades via Speculative Decoding (Narasimhan et al., ICLR 2025)
Distributed Speculative Inference: Speculation Parallelism for Provably Faster Lossless Inference (Timor et al., ICLR 2025)
List-Level Distribution Coupling with Applications to Speculative Decoding and Lossy Compression (Phan et al., NeurIPS 2025)
Speculative Sampling via Exponential Races (Kobus & Nazer, Findings of ACL 2025)
Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization (ICLR 2026)
Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding (ICLR 2026)

Relaxing the lossless guarantee

Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (Bachmann et al. (Meta), ICLR 2025)
Fuzzy Speculative Decoding for a Tunable Accuracy–Runtime Tradeoff (Holsman et al., Findings of ACL 2025)
Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026)

Part IV — Downstream applications & adaptations

Reasoning and test-time scaling

Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026)
Reward-Guided Speculative Decoding for Efficient LLM Reasoning (Liao et al. (Salesforce), ICML 2025)
Scaling Speculative Decoding with Lookahead Reasoning (Fu et al., NeurIPS 2025)
Accelerating Large Language Model Reasoning via Speculative Search (Wang et al., ICML 2025)
Accelerated Test-Time Scaling with Model-Free Speculative Sampling (Baek et al., EMNLP 2025)
SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026)
FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning (ICLR 2026)

Multimodal, vision and action

LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (Jang et al., ICLR 2025)
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding (Teng et al., ICLR 2025)
ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (Wang et al., NeurIPS 2025)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding (Zhang et al., NeurIPS 2025)
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning (Liu et al., EMNLP 2025)
Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance (Wang et al., EMNLP 2025)

Diffusion, retrieval, agents and alignment

Accelerated Diffusion Models via Speculative Sampling (De Bortoli et al. (DeepMind), ICML 2025)
Diffusion Models are Secretly Exchangeable: Parallelizing DDPMs via Auto Speculation (Chen et al., ICML 2025)
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (Christopher et al., NAACL 2025)
Speculative Actions: A Lossless Framework for Faster Agentic Systems (ICLR 2026)
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (Wang et al. (Google), ICLR 2025)
Speculative Safety-Aware Decoding (Zhang et al., EMNLP 2025)
Reward-Shifted Speculative Sampling Is an Efficient Test-Time Weak-to-Strong Aligner (Zhang et al., EMNLP 2025)
Constrained Decoding with Speculative Lookaheads (Nakshatri et al., NAACL 2025)

Part V — Systems, serving & practice

SpecInfer: Accelerating Generative LLM Serving with Tree-based Speculative Inference and Verification (Miao et al., ASPLOS 2024)
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding (Chen et al., NeurIPS 2024)
TurboSpec: Closed-loop Speculation Control for Optimizing LLM Serving Goodput (Liu et al., arXiv 2024)
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding (Sun et al., COLM 2024)
MagicDec: Breaking the Latency–Throughput Tradeoff for Long Context Generation (Chen et al., ICLR 2025)
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding (Wu et al., ACL 2025)
MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (Yan et al., NeurIPS 2025)
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache (Tiwari et al., ICML 2025)
QSpec: Speculative Decoding with Complementary Quantization Schemes (Zhao et al., EMNLP 2025)
RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (Chen et al., ICML 2025)
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (Jia et al., NeurIPS 2025)
OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding (Nallapati et al. (Qualcomm), NeurIPS 2025)
Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM Inference (Zhang et al., EMNLP 2025)
Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based Decoding (NeurIPS 2025)
Speculate Deep and Accurate: Lossless Training-Free Acceleration for Offloaded LLMs (Choi et al., NeurIPS 2025)
Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation (Liu et al., ICML 2025)

BibTeX

@misc{csonet2026-speculative-decoding-tutorial,
  author = {Nguyen, Le-Minh and Le, Nguyen-Khang and Do, Dinh-Truong},
  title  = {Speculative Decoding: Accelerating Large Language Model Inference},
  note   = {Tutorial at the 15th International Conference on
            Computational Science and Network Intelligence (CSoNet 2026)},
  year   = {2026},
  url    = {https://csonet-conf.github.io/csonet26/}
}

CSoNet 2026 Tutorial Speculative Decoding: Accelerating Large Language Model Inference