Speculative decoding can help AI chatbots improve throughput and reduce hardware demand by using a smaller model to draft tokens that a larger model validates.
Per-agent tok/s is the aggregate output throughput divided by the concurrency level. Note how it collapses from roughly 40 (single user) to roughly 10 (8 concurrent) in both modes. The GB10 is ...
"sweep": "HIP_VISIBLE_DEVICES=1 PYTHONPATH=. python3 scripts/qwen35_concurrency_decode_sweep.py --model /models/hipengine/Qwen3.6-35B-A3B-PARO-full4096-e5-packed-MTP ...
Some results have been hidden because they may be inaccessible to you
Show inaccessible results