VIDRAFT, a team competing in the Fast Gemma Challenge, achieved a throughput of 510.58 tokens per second (TPS) using the Google Gemma-4-E4B-it model on an NVIDIA A10G GPU. This result was verified by the challenge organizers, maintaining a PPL (perplexity) score of 2.3930, which is below the quality threshold of 2.42. The team emphasized that their goal was not just to achieve the highest TPS but to do so without compromising model quality.
The team's configuration, detailed in a public manifest.json file, includes several optimizations such as a sliding window of 188 tokens, a centroid top-k value of 49, and speculative decoding with a multi-token prediction drafter. These settings were chosen to balance throughput and PPL, with the team sweeping through various configurations to find the optimal balance. The synthetic warmup bridge was also used to ensure consistent performance during the timed run by pre-warming the CUDA graph capture and JIT compiler.
The challenge required participants to maximize inference speed on identical hardware without altering the model or disabling features. The team's submission was verified by the organizers using a private prompt set, ensuring the results were reproducible and reliable. The team acknowledged that while there were faster runs, they exceeded the PPL threshold and did not pass verification.
Source: huggingface
