OpenBrowser / Engineering notesView study source
On this page Agreement saves target passes, but drafting still costs time
GEMMA 4 E2B-IT · BATCH=1 · GAMMA=4 · n=3

When does multi-token prediction make Gemma 4 faster?

At batch=1, MTP stayed faster than target-only decoding in all three runs for four of twelve vLLM GPU-and-prompt combinations. Agreement alone did not predict the result.

A draft can agree with a language model and still take longer than asking that model to generate the answer alone. Gemma 4's multi-token prediction, or MTP, adds a small assistant that proposes tokens for a larger target model to check. The assistant earns its cost only when accepted proposals save enough later target work.

This study follows that trade through the model, the serving system, and three kinds of prompts. It compares Gemma 4 E2B-it with its official assistant on A10, A100-80GB, H100, and B200 GPUs. Two comparisons matter throughout: MTP against target-only decoding inside an engine, and vLLM against the transformers reference implementation. An advantage in the second comparison is not evidence of an advantage in the first.

Updated . Benchmark runs: . Study source and implementation.

Agreement saves target passes, but drafting still costs time

Target-only decoding runs the target model for each next token. Speculative decoding first runs a smaller drafter sequentially, then checks several proposals together with the target. The comparison below shows the dependency that changes: one target verification can replace several later target passes. The drafter's own sequential work remains, so the drawing does not establish a speedup.

Fewer target passes require an additional drafting stage

Target-only decoding runs the target for successive tokens. Speculative decoding runs an assistant for several proposals before one target verification.
The diagram scrolls horizontally on narrow screens.Target-only and draft-model decoding have different execution dependencies. The illustration does not show measured execution times or a guaranteed throughput gain.

Four proposals also do not mean four finished tokens. In the next example, the assistant proposes d1, d2, d3, and d4. The target accepts d1 and d2, finds a mismatch at d3, and discards the remaining draft. It supplies correction c3, so generation continues from d1, d2, c3. The accepted prefix saves work only relative to what target-only decoding would have cost.

The target keeps an accepted prefix and corrects the first mismatch

The assistant proposes four tokens. The target accepts the first two, rejects the rest, and supplies the third output token.
The diagram scrolls horizontally on narrow screens.Two accepted proposals and one target correction illustrate a speculative round. This example is not a measured acceptance rate.

The number of proposals per round is gamma. This comparison fixes gamma at 4 so the runs have the same draft length. A heuristic schedule changes that length as generation proceeds, so it can change both drafting cost and the amount of work the target can avoid. 1

Acceptance measures agreement between proposals and the target, not the time spent obtaining that agreement. A useful explanation of speed needs both sides of the system. The model sets compute and memory requirements, while the engine schedules drafting and verification.

The model's memory footprint changes the hardware question

The assistant is small compared with the target, but both models occupy memory. The model inventory below records parameter counts and BF16 artifact sizes from the study's Hugging Face API snapshot. Those stored bytes help explain deployment requirements. They are not a count of the arithmetic performed for every generated token, and they have not been freshly verified for this rewrite.

Study snapshot of the target and assistant models. Parameter counts and BF16 artifact sizes were source-reported from the Hugging Face API, not reverified today.
Model
google/gemma-4-E2B-it5,123,178,05110,246,621,918 B (9.5429 GiB)
google/gemma-4-E2B-it-assistant (drafter)77,993,476157,565,344 B (0.1467 GiB)

The gap between stored parameters and per-token arithmetic comes partly from Per-Layer Embeddings, or PLE. E2B is a dense transformer with embedding lookups, not a mixture-of-experts model. A lookup reads the row selected by a token ID rather than multiplying every stored embedding weight. The comparison below distinguishes that memory capacity from the routed feed-forward computation of MoE. The study uses 1.91 billion active parameters for utilization estimates, while the model card cites 2.3 billion effective parameters. That accounting difference remains unresolved.

PLE and MoE reduce the ratio of active to stored parameters for different reasons. PLE uses token-indexed lookups, while MoE selects expert computation.
MoE (e.g. Gemma 4 26B-A4B)PLE (Gemma 4 E2B / E4B)
What's storedN expert FFN weight matricesPer-token embedding vectors per layer
Op typeMatmul (gated FFN)Gather (lookup by token id)
FLOPs/tokenNonzero (top-k experts × FFN)Zero (memory read only)
Routed byLearned router over hidden stateToken id (deterministic index)
Why "active" < totalTop-k of N experts selectedOnly one row per layer touched
AddsCompute capacity (sparsely activated)Memory capacity (deterministically indexed)

Those memory reads make hardware bandwidth relevant beside peak arithmetic throughput. The four measured GPUs span Ampere, Hopper, and Blackwell, with different memory systems and capacities. Their specifications below are the study's recorded hardware snapshot, not new measurements. The B200 peak-compute entry depends on a provisional architecture constant, so its displayed precision does not make that estimate definitive.

Hardware specifications recorded in the study from NVML telemetry and architecture-based calculations. This is a source-reported snapshot, not a fresh verification. B200 peak FP16 throughput uses a provisional constant.
GPUArch
A10Ampere867238462.48600.1024
A100-80GB PCIeAmpere801085120155.931935.3680
H100 80GB HBM3Hopper901325120535.273352.3280
B200Blackwell10014876802382.407672.32180

Memory capacity is only one deployment constraint. A CUDA wheel must also contain code that the GPU can execute. The study's vLLM 0.21.0 wheel discussion identifies sm_75 as the oldest included target, followed by sm_80, sm_86, sm_89, sm_90, sm_100, and sm_120. The SM version identifies a GPU's compute capability, not its number of streaming multiprocessors. A driver compatibility package cannot supply missing kernels for an older architecture. 2

Precision introduces a separate constraint. BF16 and FP16 are two-byte floating-point formats with different precision and exponent ranges. The model sizes above describe BF16 artifacts, while the hardware table labels its compute denominator as peak FP16 throughput. Neither label means that every GPU has the same native arithmetic support. Native BF16 tensor-core support starts with Ampere. Binary compatibility alone does not establish a BF16 execution path for the guide's Turing candidates. 3

The deployment tiers below are broader than the four-GPU experiment. The source states a 16 GiB BF16 memory floor with max_model_len=4096, but also lists an 11 GiB RTX 2080 Ti. Those statements do not establish a consistent minimum, and the unmeasured candidates are not additional benchmark evidence. The study's CUDA-wheel guidance also specifies an R535+ driver baseline, with cuda-compat-12-9 or cuda-compat-13-0 where needed. Such packages address driver compatibility, not missing SASS or memory capacity.

Source deployment tiers for vLLM 0.21.0 and Gemma 4 MTP, including compute capability, VRAM, and suggested context limits. These are compatibility recommendations, not measured performance rankings. The 11 GiB entry conflicts with the stated 16 GiB floor, and Turing BF16 support needs separate validation.
TierExample GPUsm_vLLM 0.21.0 Gemma 4 MTP
MinimumT4 / RTX 2080 Tism_7516 / 11 GiBWorks at BF16 with max_model_len <= 4096; T4 is the cheapest cloud option
RecommendedA10 / L4sm_86 / sm_8924 GiBHeadroom for longer context and MTP draft cache
RecommendedRTX 4090sm_8924 GiBBest single-card consumer option
ProductionA100 40/80GBsm_8040 / 80 GiBFull MTP + batching + PagedAttention
ProductionH100 / H200sm_9080 / 141 GiBHighest acceptance throughput; MTP scales with batch

The experiment also depends on access to the model weights and an authenticated service, not just a suitable card. In the companion model-serving project, local preparation creates an environment with HF_TOKEN for model access and MODEL_API_KEY for the service. Installing the benchmark dependencies at this stage keeps the load generator available to the same deployment workflow. The preserved example shows those dependencies rather than a new benchmark configuration.

cd multi-token-prediction
cp .env.example .env
# fill HF_TOKEN, MODEL_API_KEY (any opaque string)
uv sync --extra bench

Modal then supplies persistent storage and deployment credentials around those model artifacts. The bootstrap script below configures CLI authentication, registers the model-api-key and hf-token secrets, and creates the gemma-models volume. Its idempotent setup makes the same weights available across redeployments, which matters when comparing MTP and baseline containers rather than comparing different download states.

# Idempotent: token, secrets, and persistent volume for weights.
bash deploy/modal/setup_modal.sh

The serving engine changes what a saved pass is worth

Once the models fit, the engine determines how much time each draft and verification costs. The transformers reference service calls generate(assistant_model=...), while the vLLM deployment uses its own serving stack. The component map below separates model execution from the API, load generation, telemetry, and deployment scripts. That separation is necessary to attribute a difference to speculative decoding rather than to a different request path.

Component responsibilities in the companion model-serving project. The inference engine and API serve requests, while benchmark, GPU-probe, utilization, and Modal deployment modules control and measure the comparison.
PathRole
server/mtp_engine.pyLoads target + drafter, calls generate(assistant_model=...) per the Gemma 4 reference. Exposes generate() and stream_generate(). Tracks accept/propose counters.
server/api.pyOpenAI-compatible FastAPI service: /v1/chat/completions (stream + non-stream), /v1/models, /healthz, /metrics. X-API-Key and Bearer auth.
bench/load_runner.pyConcurrent SSE benchmark; persists JSON + Prometheus per run. Captures acceptance rate.
bench/gpu_probe.pyLive VRAM and peak FP16 TFLOPS via NVML.
bench/mfu.pyExact MFU = 2 * N_active * tokens/sec / peak_TFLOPS.
deploy/modal/modal_app.pyModal app: GPU container, ASGI mount, persistent volume for weights. Transformers MTP engine.
deploy/modal/vllm_modal_app.pyModal app: vLLM v0.21.0 serving stack with num_speculative_tokens env propagation.
deploy/modal/run_ab.shOne-shot A/B sweep: deploys MTP, warms, benches, redeploys baseline, benches, restores MTP.
deploy/modal/run_const.shSame flow with MTP_SCHEDULE=constant, N=4 for apples-to-apples vLLM parity.
deploy/modal/vllm_run_ab.shvLLM A/B: num_speculative_tokens=4 vs no-spec-config baseline.

The architecture uses two same-type GPU containers, but only the server GPU runs inference. The bench GPU runs load_runner and NVML-probes its own board. It does not execute the model and does not directly measure the inference server's utilization. The diagram makes that boundary visible: deployment scripts start the service and the benchmark separately, and results pass through the bench-results volume into metrics/runs/. Matching GPU types supplies hardware denominators, not server-side utilization telemetry.

modal deploy

modal run

HTTPS .modal.run

modal volume get

deploy scripts

Server GPU
transformers or vLLM

Bench GPU
load_runner + NVML

bench-results volume

metrics/runs/

The server GPU runs transformers or vLLM. A separate same-type bench GPU runs load_runner and probes its own hardware with NVML. Deployment scripts retrieve persisted results into metrics/runs/.

The engines also manage attention and cached tokens differently. The KV cache stores attention keys and values from earlier tokens, so later decoding steps can reuse them. PagedAttention organizes that storage, while FlashAttention reduces memory traffic during attention computation. Verification has its own execution costs. These mechanisms can explain an engine advantage, but this experiment does not isolate their individual contributions. In particular, concurrency=1 results do not measure a gain from serving many requests through continuous batching. 4

Version choice defines which speculative paths are available. The study pins vllm/vllm-openai:v0.21.0. Its recorded release history places base Gemma 4 support in PR #38826, merged on 2026-04-02 and released in v0.20.0 on 2026-04-27. Gemma 4 MTP support followed in PR #41745, merged on 2026-05-06 and released in v0.21.0 on 2026-05-15. This is the study's version boundary, not a claim about the latest release today.

The preserved launch example below belongs to the deployment guidance, not to the measured configuration. It explicitly selects method=mtp, proposes two tokens, and limits context to 4096. The benchmark instead passes the separate assistant model without a method key and fixes num_speculative_tokens=4. Treating the example as the benchmark command would change both the proposer selection and the draft length.

vllm serve google/gemma-4-E2B-it \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-E2B-it-assistant","num_speculative_tokens":2}' \
  --dtype bfloat16 --max-model-len 4096

DFlash is outside that comparison. The frozen version notes place its initial support in PR #36847 and v0.20.0, but identify later fixes absent from v0.21.0. These include the FP8 KV-cache fix #42692 and lookahead-slot allocation fix #43733, associated with the v0.22.0 release line. They also list #40391 and #41703 as unresolved on 2026-05-28. Those dated qualifications prevent the batch=1 draft-model result from becoming a compatibility claim for DFlash with an FP8 KV cache.

Before either engine serves requests, pre-downloading both models removes Hugging Face transfer time from subsequent container starts. The weight-warming script below populates gemma-models with the target and assistant. That removes one source of cold-start delay, but it cannot remove every later kernel compilation or graph-capture cost. A warmed weight volume and a warmed speculative execution path are different conditions.

bash deploy/modal/warm_weights.sh

Requests then reach an OpenAI-compatible interface with either X-API-Key or Authorization: Bearer authentication. The example below shows the message payload and sampling controls. Its max_tokens=256 and temperature=1.0 illustrate API use, not the study's greedy, 128-token workload. In the custom transformers service, response usage includes accepted_tokens, proposed_tokens, and acceptance_rate under speculative_decoding. The native vLLM server exposes acceptance through cumulative metrics instead, a difference that limits prompt-level analysis later.

curl https://<random>.modal.run/v1/chat/completions \
  -H "X-API-Key: ${MODEL_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-E2B-it",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a short joke about saving RAM."}
    ],
    "max_tokens": 256,
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 64
  }'

A service can answer a health check without demonstrating that its draft path is fast or correctly disabled in a baseline. The smoke scripts below deploy each stack, print its HTTPS endpoint, check /healthz, and tear it down. They establish service reachability. The measurement workflow needs additional warmup and zero-proposal checks before an MTP comparison is meaningful.

GPU=h100 bash deploy/modal/smoke_test.sh        # transformers MTP
GPU=h100 bash deploy/modal/vllm_smoke_test.sh   # vLLM stack

The comparison holds the draft schedule and workload fixed

The reported runs use batch=1, concurrency=1, 16 requests, max_tokens=128, and temperature=0.0. Each regime rotates eight prompts on Modal. On 2026-06-04, every combination of four GPUs, three regimes, two engines, and two decoding modes was measured in three independent cold runs, n=3. All 144 runs succeeded. The summaries exclude the first timed request, idx=0, because speculative kernel compilation can survive the preliminary warmup. This makes the headline a warm-throughput comparison, not a first-request latency claim.

Throughput and latency answer different questions about those requests. System throughput divides total completion tokens by wall-clock time. TTFT, the time to first token, measures the wait until the first streamed token. TPOT, the time per output token, divides the remaining latency by completion_tokens minus one. TPOT is undefined for a one-token completion. A high token rate therefore does not necessarily mean a short request or a low startup delay. 5

A comparable MTP trial must also hold the draft schedule fixed. The first script below uses the default heuristic schedule and is not the source of the constant-gamma headline. run_const.sh fixes the transformers draft at N=4, matching vLLM's num_speculative_tokens=4. Each A/B workflow deploys MTP, warms and measures it, redeploys the baseline, measures again, and restores MTP. The transformers baseline uses NUM_ASSISTANT_TOKENS=0 even if its assistant remains loaded. The vLLM baseline omits speculative configuration entirely.

# Transformers MTP A/B (default heuristic schedule):
GPU=h100 bash deploy/modal/run_ab.sh

# Transformers MTP, schedule=constant, N=4 (apples-to-apples vLLM parity):
GPU=h100 bash deploy/modal/run_const.sh

# vLLM A/B (num_speculative_tokens=4 vs no-spec-config baseline):
GPU=h100 bash deploy/modal/vllm_run_ab.sh

Each run warms three short requests before timing. Baseline validation separately checks that proposed_tokens stays at zero for three independent requests, rather than trusting a mode label. Within an engine, the reported MTP/baseline ratio is the mean of paired per-run throughput ratios. Across engines, a throughput ratio divides their mean throughputs. These operations are different, so dividing two rounded table means need not reproduce a paired-ratio cell. Mean ± standard deviation describes the three runs, not a confidence interval or a significance test.

The load generator provides a second way to see why settings matter. The preserved command below sends 64 requests at concurrency=4 with a 256-token limit. That is an example invocation, not a reproduction of the reported 16-request, concurrency=1, 128-token experiment. bench/load_runner.py consumes SSE responses and persists JSON plus Prometheus data. Its measurements still depend on the workload supplied to it.

uv run python -m bench.load_runner \
  --base-url https://<endpoint>.modal.run \
  --api-key "${MODEL_API_KEY}" \
  --requests 64 --concurrency 4 --max-tokens 256 \
  --label mtp_on

Prompt selection is the remaining controlled change. PROMPT_SET chooses generic, code, or structured prompts for the same deployment scripts. The example below selects code across all four GPUs and shows separate baseline runs as well as vLLM A/B runs. Mode filters are MODES for transformers and VLLM_MODES for vLLM. Regime labels keep the outputs separate, with the generic suffix omitted for compatibility with existing run names.

# Transformers MTP constant N=4 (PROMPT_SET in {generic,code,structured}):
PROMPT_SET=code bash deploy/modal/run_const.sh H100
PROMPT_SET=code bash deploy/modal/run_const.sh A100-80GB
PROMPT_SET=code bash deploy/modal/run_const.sh A10
PROMPT_SET=code bash deploy/modal/run_const.sh B200

# Transformers baseline (N=0, MTP off):
for GPU in H100 A100-80GB A10 B200; do
  PROMPT_SET=code MODES=baseline bash deploy/modal/run_ab.sh "$GPU"
done

# vLLM A/B (mtp + baseline):
for GPU in H100 A100-80GB A10 B200; do
  PROMPT_SET=code bash deploy/modal/vllm_run_ab.sh "$GPU"
done

The deployment environment adds limits that are not decoding effects. The source records an eight-app web-function cap for a free Modal workspace, so an unrestricted parallel sweep can exhaust deployment slots. The scripts stop deployments they own between cells. The recorded image branches also differ: B200 uses torch==2.9.1+cu128 for sm_100 kernels, while the other GPUs use torch==2.7.0+cu126. Hardware comparisons therefore describe these deployed stacks, not a controlled substitution of the GPU alone.

Only four vLLM combinations stayed above target-only decoding

Google's published results establish why MTP is worth testing, but they do not set a speedup floor for this workload. The table below spans different model sizes and hardware. Google qualifies these numbers as up to, depending on tasks, at batch_size=1 and gamma=4. The 1.5x A100 result concerns the 26B variant, while the larger claims include mobile accelerators and other Gemma variants. None is a matched control for E2B on this four-GPU deployment.

Google's published task-dependent maximum speedups at batch_size=1 and gamma=4. These model and hardware combinations are external context, not replications of this study.
VariantHardwarePublished speedup
Gemma 4 E2BSamsung S26 mobile GPUup to 1.8x
Gemma 4 E4BSamsung S26 mobile GPUup to 2.2x
Gemma 4 E2BPixel TPUup to 2.8x
Gemma 4 E4BPixel TPUup to 3.1x
Gemma 4 31BApple M4up to 2.5x
Gemma 4 26BNVIDIA A100up to 1.5x
Gemma 4 31BNVIDIA A100up to 3.0x

The matched vLLM comparison gives a narrower answer. A10 stays above target-only throughput in all three runs for generic, code, and structured prompts. A100-80GB code does so as well. The other eight GPU-and-regime cells have observed ranges that cross 1.0, even though their means exceed it. The table below summarizes paired per-run ratios. Its emphasis marks an observed three-run range entirely above 1.0, not statistical certainty about future requests.

vLLM MTP/baseline throughput ratios at constant gamma=4, mean ± standard deviation across three runs. Emphasized cells have their full observed three-run range above 1.0.
GPU
A101.38x ± 0.062.06x ± 0.411.79x ± 0.29
A100-80GB1.06x ± 0.141.53x ± 0.251.07x ± 0.07
B2001.41x ± 0.421.18x ± 0.291.22x ± 0.37
H1001.05x ± 0.271.28x ± 0.431.37x ± 0.59

The plot makes the size of that variation easier to compare across prompt regimes. A10 code has the largest mean ratio, 2.06x ± 0.41, but H100 structured shows why a positive mean is insufficient. Its 1.37x ± 0.59 summarizes runs ranging from 0.72x to 2.14x. Those runs include a slowdown and a speedup. The plot's standard-deviation marks and range-based classification describe different aspects of those three observations.

The same draft length produces different throughput ratios

The chart compares vLLM throughput with MTP against vLLM throughput without MTP. A ratio of 1x is break-even.

A101.38x ± 0.06
A100-80GB1.06x ± 0.14
B2001.41x ± 0.42
H1001.05x ± 0.27
Measured vLLM ratios for generic, code, and structured prompts. Error marks show mean ± standard deviation across three runs, not confidence intervals. The positive classification requires the entire observed three-run range to exceed 1.0.

The transformers reference path does not reproduce the same pattern of gains. Its summary below includes an A10 generic slowdown and an H100 code slowdown, while structured output produces its clearest gains on A10 and B200. That change across engines matters: the assistant's agreement rate alone cannot explain performance when the runtime changes the cost of obtaining and verifying each draft.

Transformers reference-path MTP/baseline throughput ratios, mean ± standard deviation across three constant-gamma runs. The frozen source disagrees about the classification of H100 structured output.
GPU
A100.78x ± 0.080.90x ± 0.071.16x ± 0.11
A100-80GB0.93x ± 0.261.01x ± 0.190.97x ± 0.12
B2001.24x ± 0.231.20x ± 0.301.40x ± 0.26
H1000.85x ± 0.240.83x ± 0.101.04x ± 0.01

The source overview counts seven positive and two negative cells across both engines, leaving fifteen ranges that cross 1.0. That count includes H100 transformers structured as positive, while the detailed structured discussion calls it breakeven despite retaining 1.04x ± 0.01. This classification conflict remains unresolved. It does not change the four positive vLLM combinations. The regime-by-regime evidence below explains those comparisons without treating the disputed transformers label as settled.

Generic prompts leave little room for the drafter

The generic regime rotates eight prose prompts. On the transformers path, aggregate acceptance spans 34.6-35.7% across GPUs, so most proposals do not survive verification. The first table pairs that agreement with warm system throughput. A10 falls from 11.67 to 9.14 tokens/s with MTP, giving a paired ratio of 0.78x ± 0.08. Its full run range stays below 1.0. The other three generic transformers ratios cross 1.0, including B200's positive mean.

Generic prompts on the four measured GPUs, transformers MTP versus baseline. Warm system throughput is in tokens/s, with mean ± standard deviation across three runs.
GPUArch
NVIDIA A10Ampere8.660011.67 ± 0.119.14 ± 0.890.78x ± 0.0835.7%
NVIDIA A100 80GB PCIeAmpere8.019359.84 ± 2.538.46 ± 0.190.93x ± 0.2635.5%
NVIDIA B200Blackwell10.0767217.87 ± 4.2021.21 ± 0.981.24x ± 0.2335.2%
NVIDIA H100 80GB HBM3Hopper9.0335216.12 ± 2.3913.16 ± 1.510.85x ± 0.2434.6%

vLLM changes the absolute token rate much more than enabling MTP changes it within either engine. In the table below, vLLM MTP is 8.9x to 16.9x the transformers MTP mean throughput. Yet its within-engine MTP gain stays above 1.0 in every run only on A10. The two ratio columns answer different questions. With MTP disabled in both engines, the same generic data give a separate engine-throughput ratio of 6.3x to 13.9x. That advantage does not come from accepting draft tokens.

Generic warm system throughput across engines. MTP/baseline entries summarize paired per-run ratios. Cross-engine entries divide mean throughputs.
GPU
A109.14100.4372.971.38x ± 0.0611.0x37.3%
A100-80GB8.46142.73136.761.06x ± 0.1416.9x36.5%
B20021.21189.62143.521.41x ± 0.428.9x37.1%
H10013.16166.44164.961.05x ± 0.2712.6x37.2%

Output length complicates the interpretation of those cross-engine rates. In the generic workload, vLLM emits 2048 completion tokens across 16 requests at the 128-token limit. The source reports shorter transformers output and attributes that difference to an interaction between assisted decoding, rejection sampling, and EOS. Tokens per wall-clock second still describes the work each stack produced. It does not establish equal completion lengths or equivalent output quality, so end-to-end request latency must remain beside throughput.

The first timed request introduces another difference. In the per-prompt comparison below, cold idx=0 is a particular request occurrence, not a reason to discard that prompt's later warm occurrence. H100's cold MTP/baseline ratio is 0.44, compared with 0.98 for the warm occurrence of prompt 0. A10 retains warm ratios above 1.0 across the prompts. The split exposes startup cost without turning the remaining near-breakeven values into dependable gains.

Generic vLLM MTP/baseline ratios by prompt, with the first cold occurrence separated from the later warm occurrence of prompt 0. Values summarize three runs.
GPU
H1000.440.981.041.031.061.030.901.050.99
A100-80GB0.711.061.061.041.081.040.921.151.04
A100.901.391.491.351.421.321.261.481.35
B2001.171.291.381.351.351.261.281.371.29

Averaging those prompt-level ratios shows how one cold occurrence can change a descriptive result. H100 moves from 0.95 when cold idx=0 is included to 1.01 when it is dropped. A10 remains above 1.0 either way. These are means of prompt ratios, not replacements for the run-level aggregate throughput comparison. Their different aggregation explains why the warm figures below need not equal the headline ratios.

Means of generic per-prompt vLLM ratios with and without the first cold occurrence. This prompt-level aggregation differs from the headline paired run-throughput statistic.
GPU
H1000.951.010.90 to 1.06
A100-80GB1.011.050.92 to 1.15
A101.331.381.26 to 1.49
B2001.301.321.26 to 1.38

The engine advantage persists when the same generic prompts are compared individually. The table below holds speculative draft length at four in both implementations and divides vLLM throughput by transformers throughput. The cold occurrence produces a smaller ratio on every GPU than the listed warm occurrences. Warm ratios span 8.1x to 21.7x across the grid. That pattern supports an engine-level difference without identifying which attention kernel or scheduler feature causes it.

Generic per-prompt throughput ratios between vLLM MTP and transformers MTP, both at draft length four. Cold and warm request occurrences remain distinct.
GPU
H1006.613.511.714.313.410.915.810.6
A100-80GB8.118.015.419.416.414.821.715.2
A108.012.310.112.310.19.913.410.1
B2007.19.68.610.18.48.111.18.1

Agreement varies more by prompt than by the GPU running it. In the transformers prompt grid below, mean acceptance across the eight prompts spans 35.1-36.1% by GPU, while the within-GPU prompt spread spans 16.7-27.3 percentage points. These prompt means differ from the aggregate acceptance statistic above. Binary search at idx=1 is more predictable for the drafter than the PagedAttention-versus-FlashAttention explanation at idx=5. The prompt's content changes the opportunity for speculation before hardware determines its cost.

Transformers generic acceptance by prompt, averaged across three runs. The mean column averages prompt percentages and is not the run-level accepted/proposed ratio.
GPU
A1033.0%44.6%33.2%38.9%34.7%27.9%40.4%36.4%36.1%16.7
A100-80GB32.0%47.4%33.3%36.2%37.6%28.8%39.5%32.6%35.9%18.6
B20033.0%50.9%34.4%40.4%32.4%23.6%39.5%34.3%36.1%27.3
H10033.0%43.2%26.3%41.6%32.9%29.7%39.5%34.9%35.1%16.9

There is no matching vLLM per-prompt acceptance grid. Its native server exposes spec_decode_num_accepted_tokens_total and spec_decode_num_draft_tokens_total through process-cumulative /metrics, not per-response usage.speculative_decoding. The study obtains vLLM acceptance from counter deltas over the benchmark wall interval. Those aggregates cannot establish which individual prompt accepted which proposals. Code prompts test the next question at the regime level: whether more predictable token sequences create a larger useful draft.

Code raises acceptance, but only some GPUs turn it into speed

The code regime rotates two_sum, merge_sort, is_balanced, LRUCache, quicksort, Dijkstra, flatten, and binary_tree. Familiar syntax gives the drafter more repeated structure to predict. In the comparison below, transformers acceptance rises to 50.0-51.0%, but its throughput barely improves or falls relative to generic prompts. vLLM MTP gains more throughput on A10 and A100-80GB. The same shift in agreement therefore has different consequences in the two engines.

Code versus generic warm MTP throughput in both engines, with transformers acceptance. Cross-regime throughput ratios compare workloads, not MTP against its baseline.
GPU
A109.149.731.06x100.43136.041.35x35.7%50.5%
A100-80GB8.468.380.99x142.73203.261.42x35.5%51.0%
B20021.2119.370.91x189.62190.831.01x35.2%50.7%
H10013.1613.461.02x166.44189.641.14x34.6%50.0%

vLLM's aggregate code acceptance is 52.5% on A10, 52.6% on A100-80GB, 52.3% on B200, and 52.4% on H100. Combining those values with the displayed transformers means gives a per-GPU code acceptance range of 50.0-52.6%. The frozen executive summary instead reports 49.8-52.6%. That discrepancy is unresolved. The available means support higher agreement on code than on generic prompts, not a silently corrected lower endpoint.

Code can change target-only throughput as well, so comparing code MTP with generic MTP does not isolate speculation. The baseline table below removes the drafter's proposals from both engines. Its code-versus-generic changes are not uniform across GPUs. vLLM's baseline engine advantage on code spans 6.1x to 15.7x over transformers, independent of MTP. That is a separate measured range from the 8.9x to 26.8x cross-engine MTP range across all regimes.

Target-only throughput for code and generic prompts in both engines. MTP is off in both arms, so the engine difference cannot be attributed to speculative acceptance.
GPU
A1011.6710.850.93x72.9766.620.91x
A100-80GB9.848.550.87x136.76133.880.98x
B20017.8716.570.93x143.52167.521.17x
H10016.1216.020.99x164.96157.220.95x

Dividing MTP by its own baseline gives the relevant code speedup comparison. A10 vLLM code reaches 2.06x ± 0.41 with an observed range of 1.54x to 2.52x. A100-80GB reaches 1.53x ± 0.25, ranging from 1.25x to 1.86x. Both ranges stay above 1.0. B200's 0.91x to 1.59x and H100's 0.96x to 1.90x cross 1.0. On transformers, no code cell is a consistent gain across the runs, and H100's 0.83x ± 0.10 is a consistent slowdown in this sample.

Paired per-run MTP/baseline ratios for code and generic prompts, mean ± standard deviation across three runs. Only A10 and A100-80GB show code gains across all three vLLM runs.
GPU
A100.78x ± 0.080.90x ± 0.071.38x ± 0.062.06x ± 0.41
A100-80GB0.93x ± 0.261.01x ± 0.191.06x ± 0.141.53x ± 0.25
B2001.24x ± 0.231.20x ± 0.301.41x ± 0.421.18x ± 0.29
H1000.85x ± 0.240.83x ± 0.101.05x ± 0.271.28x ± 0.43

The acceptance grid helps distinguish a repeatable prompt property from a repeatable timing gain. Code prompt means remain near 50% across the four transformers deployments. Their within-GPU spreads are smaller than the generic spreads, although H100 reaches 12.1 percentage points, above the 6.7-9.5 range stated in the original prose. The grid retains that H100 value. More uniform agreement is a plausible reason code helps the drafter, but the timing comparison still has to cover drafting and verification costs.

Transformers code acceptance by prompt across three runs. Indices follow two_sum, merge_sort, is_balanced, LRUCache, quicksort, Dijkstra, flatten, and binary_tree.
GPU
A1056.3%54.4%48.9%48.8%48.7%48.0%50.6%48.3%50.5%8.3
A100-80GB57.2%54.4%49.4%47.7%49.1%51.5%50.6%48.3%51.0%9.5
B20053.1%54.4%47.7%48.8%49.1%51.5%52.4%48.3%50.7%6.7
H10055.1%54.4%47.7%43.0%49.1%49.7%50.6%50.0%50.0%12.1

The per-prompt engine comparison below shows that vLLM's advantage is not confined to one code task. On A100-80GB, the listed warm ratios range from 22.2x to 26.9x. B200 has a smaller gap, including a 9.0x warm ratio for one prompt. These prompt-level values should not be replaced by a claim that every warm code prompt exceeds 10x. They also remain engine comparisons, not the within-vLLM benefit of enabling MTP.

Code per-prompt throughput ratios between vLLM MTP and transformers MTP at draft length four, with the first cold request shown separately.
GPU
H10011.716.115.614.213.812.713.413.3
A100-80GB7.725.526.923.024.323.922.222.6
A109.816.015.613.913.812.313.213.3
B2006.611.010.59.710.39.09.69.5

End-to-end latency gives the corresponding request-level view. The table below includes the first cold request instead of hiding it inside a throughput average. On A100-80GB, vLLM's warm mean is 644 ms, compared with 2267 ms for its cold request and 4729 ms for the transformers warm mean. That distinction matters for an interactive caller who waits for a response. It still does not remove the need to examine completion lengths when comparing engines.

Code request end-to-end latency in milliseconds, averaged across three runs. Cold idx=0 and warm means are separate for each engine.
GPU
A10407916194089964
A100-80GB456122674729644
B200228318302121689
H100310510293204714

Code establishes that higher agreement can coincide with a useful MTP gain, especially in the measured A10 and A100-80GB vLLM stacks. It does not establish a threshold acceptance rate that guarantees a gain on every GPU. Structured prompts raise agreement further and make that limitation more visible.

Structured output makes agreement easier, not speed certain

The structured regime uses eight JSON, YAML, TOML, OpenAPI, and GeoJSON skeleton prompts. The hypothesis is that constrained syntax makes the target's next token easier for the assistant to predict. Aggregate acceptance rises to 52.5-58.0% across engines and GPUs. In vLLM it reaches 57.0-58.0%, yet only A10 keeps its MTP/baseline ratio above 1.0 in every run. The table below puts the agreement and timing measures together so that neither substitutes for the other.

Structured warm throughput, paired MTP/baseline ratios, and aggregate acceptance across three runs. The first timed cold request is excluded from the warm summaries.
GPU
A107.226.281.16x ± 0.11111.5564.151.79x ± 0.2955.3%57.1%
A100-80GB5.836.110.97x ± 0.12156.37147.271.07x ± 0.0754.7%58.0%
B20011.658.321.40x ± 0.26182.74157.661.22x ± 0.3752.5%57.0%
H1009.328.981.04x ± 0.01211.05156.561.37x ± 0.5952.8%57.2%

A10's structured vLLM ratio is 1.79x ± 0.29, with a range of 1.42x to 2.13x. A100-80GB's 1.07x ± 0.07 includes a run at 0.97x. H100 is more variable: its runs are 2.14x, 1.25x, and 0.72x, summarized as 1.37x ± 0.59 in the source. The observed 0.72x to 2.14x range supports neither a dependable gain nor a claim of established breakeven. Higher aggregate agreement has not removed timing uncertainty.

The engine comparison remains much larger than most within-engine changes. For structured prompts, the ratios of mean MTP throughputs are 15.5x on A10, 26.8x on A100-80GB, 15.7x on B200, and 22.6x on H100. With MTP off, the corresponding baseline engine ratios are 10.2x, 24.1x, 18.9x, and 17.4x. These are distinct comparisons. The overall MTP engine range of 8.9x to 26.8x must not be described as a universal range for both decoding modes.

Individual structured tasks are less uniform than their regime label suggests. The fully specified JSON object at idx=0 accepts 94.2% of proposals on every transformers deployment, while GeoJSON coordinates at idx=5 fall as low as 37.8%. The grid below has a within-GPU spread of 52.5-56.4 percentage points, the widest of the three regimes. Its mean column averages prompt percentages, which is different from dividing all accepted tokens by all proposed tokens in a run.

Transformers structured acceptance by prompt. Indices identify a JSON object, JSON book array, Kubernetes YAML, HTTP-200 JSON, user-record JSON, GeoJSON points, TOML crate configuration, and an OpenAPI path.
GPU
A1094.2%44.0%62.8%57.1%74.2%41.7%48.5%56.4%59.9%52.5
A100-80GB94.2%44.0%62.8%54.6%74.2%37.8%52.1%56.4%59.5%56.4
B20094.2%44.0%62.8%54.6%56.4%37.8%48.5%58.6%57.1%56.4
H10094.2%44.0%62.8%54.6%56.4%37.8%52.1%56.4%57.3%56.4

High acceptance does not even rank these prompts by throughput. In the per-prompt transformers results below, the almost fully accepted JSON object is not the fastest task on any GPU. On B200 it reaches 12.8 tokens/s, while the OpenAPI path at idx=7 reaches 16.6 tokens/s with much lower acceptance. Request length, prefill, drafting, and verification can all affect system tokens/s. The table shows why an agreement percentage cannot replace a direct time measurement.

Transformers structured system throughput in tokens/s by prompt, averaged across three runs. Prompt indices match the preceding acceptance grid.
GPU
A108.26.08.66.05.75.98.59.8
A100-80GB7.84.87.44.94.84.66.38.1
B20012.89.814.29.78.19.514.116.6
H10011.08.111.87.96.67.610.413.0

The tradeoff can now be stated without assuming that more accepted tokens always mean faster generation. Every speculative round pays for the assistant and target verification. Accepted proposals can avoid later target passes, but only the balance of those costs determines elapsed time. The illustration below summarizes that causal account. It is qualitative, not a fit to these measurements or an acceptance threshold inferred from them.

Accepted proposals must save more time than drafting and verification add

A speculative round includes drafting and verification overhead. Accepted tokens save later target work only if those savings exceed the added cost.
The diagram scrolls horizontally on narrow screens.Higher acceptance creates more opportunity to avoid later target work. The illustration does not measure overhead, predict a break-even rate, or explain the observed variance by itself.

The available evidence stops short of a complete cost breakdown. vLLM per-prompt acceptance is unavailable in every regime because only process-cumulative counters were collected. Per-prompt MFU, MBU, and TPOT detail is not tabulated either. The structured results support a relation between prompt predictability and agreement. They do not attribute each prompt's measured runtime to a specific GPU operation.

Cold requests and utilization estimates set the limits

The headline excludes cold idx=0 because a short warmup does not necessarily compile the speculative kernels used by the measured request. Container initialization is outside the benchmark wall clock, but the first timed request can still pay for drafter initialization and kernel JIT. The cold-versus-warm table below measures that residual cost on a system-throughput basis. In the generic vLLM MTP cells, cold throughput is 0.35-0.74 of warm throughput, compared with the source's 0.82-0.98 baseline range. The penalty is asymmetric.

Residual first-request cost in selected vLLM MTP cells, mean ± standard deviation across three runs. Cold throughput uses the first timed request. Warm throughput aggregates the remaining requests on a system-throughput basis.
vllm_mtp cell
A10 generic63.9 ± 14.2100.4 ± 3.40.64x
A100-80GB generic50.1 ± 20.6142.7 ± 7.00.35x
B200 generic140.1 ± 38.4189.6 ± 22.00.74x
H100 generic68.6 ± 12.3166.4 ± 22.70.41x
A100-80GB structured64.1 ± 36.2156.4 ± 5.50.41x

For this comparison, cold tokens/s is idx=0 completion_tokens divided by its end-to-end seconds. Warm tokens/s divides total completion tokens by total end-to-end seconds over idx=1 through idx=15. That is not the mean of per-request decode_tokens_per_sec, which excludes TTFT and changes the weighting. Removing one cold occurrence also does not remove the remaining variability. The source describes warm-throughput spread commonly at 10-40% of the mean. Three runs cannot establish a confidence interval for a production workload.

Utilization estimates require more assumptions than measured throughput. MFU compares modeled computation with peak compute rate. MBU compares modeled memory traffic with peak memory bandwidth. Repeated weight reads at batch=1 make bandwidth a plausible constraint, so MBU can be more informative than a low MFU alone. The study models these quantities from output measurements and hardware denominators. They are not direct observations of inference-server utilization, because the separate benchmark GPU probes its own board. 6

The MFU formula below uses the dense forward-pass estimate of two operations per active parameter per token. The study substitutes N_active=1.91 billion, excluding PLE lookup storage from the matrix-multiplication count. Its model card's 2.3 billion effective count is not reconciled with that choice. At fixed throughput and peak, the displayed equation makes MFU proportional to N_active. The frozen source's reciprocal rescaling note conflicts with that equation, so it is not a reliable correction to the reported estimates.

FLOPs/token = 2 * N_active                    (Kaplan 2020)
achieved_TFLOPS = FLOPs/token * tokens/sec / 1e12
MFU = achieved_TFLOPS / peak_TFLOPS

The bandwidth estimate below similarly depends on modeled bytes per output token. The study sets param_bytes=10,246,621,918 and kv_cache_bytes=0, explicitly a lower bound that omits KV-cache traffic. TPOT must be expressed in seconds for the displayed conversion to GB/s. B200 adds another qualification to utilization interpretation: its compute peak uses a provisional 8192 FP16 operations per cycle per SM, yielding the table's 2382.40 TFLOPS. That compute constant affects MFU's denominator, not the measured throughput, TPOT, latency, or acceptance. The MBU equation has separate byte and peak-bandwidth assumptions.

bytes/token = param_bytes + kv_cache_bytes
achieved_GBps = bytes/token / TPOT / 1e9
MBU = achieved_GBps / peak_HBM_GBps

The proposed explanation for A10's gains is that a slower target decode leaves more time for accepted proposals to repay drafting overhead. The measurements are consistent with that explanation, but they do not isolate its cause. Different engine implementations, GPU-specific images, output lengths, and residual startup behavior remain part of the comparison. No experiment here establishes the same result at higher concurrency, another context length, a different gamma, or a different prompt distribution.

The useful decision is whether the target work saved by accepted drafts exceeds the cost of producing and checking them in the intended service. This study makes that comparison for one small, greedy, single-request workload. A broader deployment claim would need matched output-length checks, a warmup that covers the measured speculative path, and repeated measurements under the service's actual traffic. Acceptance helps explain where to look. Elapsed time determines whether MTP helped.