Cohere has developed a new method for accelerating large language model (LLM) inference called hardware-aware dynamic speculative decoding (DSD). This approach adapts the number of draft tokens based on hardware constraints, optimizing performance across varying batch sizes. Speculative decoding (SD) has been used to speed up LLM inference by generating multiple tokens simultaneously and verifying them in a single step, but its effectiveness has been limited by hardware bottlenecks. Cohere's DSD addresses these limitations by dynamically adjusting the number of tokens processed, improving efficiency and throughput.
The technique leverages the tradeoff between compute and memory bandwidth on GPUs. At small batch sizes, inference is memory bandwidth-bound, while at larger sizes, it becomes compute-bound. DSD adapts by increasing the number of draft tokens when memory bandwidth is the limiting factor and decreasing them when compute is the bottleneck. This adaptability ensures that DSD remains effective across a wide range of workloads, including dynamic batch sizes and reinforcement learning rollouts. The method also simplifies the optimization process by using goodput as a metric to determine the optimal number of draft tokens.
Cohere tested DSD on the MT-Bench dataset and found that it outperformed traditional SD and vanilla models in several scenarios. For dense models, DSD achieved up to 23% faster performance at high batch sizes, while for MoE models, it maintained similar speedups to fixed-K SD. These results highlight the potential of DSD to enhance inference efficiency without sacrificing quality or accuracy. Cohere also integrated DSD into the vLLM framework, ensuring compatibility with existing optimizations like async scheduling and CUDA Graph.
Source: cohere
