domains / families / devops / jetson-speculative-decoding

jetson-speculative-decoding

separate

Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.

When to use

• Tune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.
• TPOT/ITL is the bottleneck (TTFT is fine, output is just slow).
• Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.
• Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.

When not to use

• High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.
• Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix".
• After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.
• Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.
• High concurrency can erase the benefit because continuous batching already keeps the GPU busy.
• Orin Nano/NX usually lack enough memory headroom for both target and draft models.
• Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.

Skill page

Source