A 0.9M parameter model trained with a 222,000:1 token-per-parameter ratio showed performance that peaked at 20B tokens before declining significantly. The model was trained using a configuration that included a 6-layer architecture with 96 hidden dimensions and SwiGLU intermediate layers, along with a Muon optimizer and AdamW for parameter optimization. The training data came from FineWeb-HQ and Cosmopedia v2, with a total budget of 200B tokens. The model was evaluated on the Open SLM benchmark suite, including ARC-Easy, ARC-Challenge, HellaSwag, and PIQA, as well as ArithMark-2/3, aggregated into an INT Index score. The INT Index peaked at 4.55 at 20B tokens and then declined monotonically to 3.31 at 180B tokens — a −27.3% loss over the last 160B tokens of training. Individual benchmarks showed similar trends, with three of four benchmarks performing worse at 180B tokens than at 20B. Only ARC-Easy improved slightly, but by less than the losses observed in other benchmarks. The model's performance was compared to a Chinchilla control, which was trained on 18M tokens at a 20:1 ratio, resulting in an INT Index of 1.53, essentially at chance level. This indicates that the useful token-per-parameter range for a Pico-tier model is between 22K and 30K tokens per parameter. Models trained at these ratios typically show healthy scaling curves, while those trained beyond this range experience performance degradation. The study highlights the importance of starting with a smaller budget and evaluating performance before extending training. The findings suggest that blindly scaling to higher token-per-parameter ratios can result in a worse model than a more moderate approach. The research was conducted by Banaxi-Tech, with additional insights from the community on the Open SLM Leaderboard.
Source: huggingface
