Hugging Face researchers explored methods to enhance a 15M parameter French language model by modifying its learning process rather than increasing its size. The project aimed to improve the model's performance without adding parameters, focusing on how the model processes information. The team tested several approaches, including changes to optimizers and layer structures, but found these methods ineffective. Instead, they turned to a novel approach involving looping and entropy-based halting to improve the model's ability to handle complex tasks.
The researchers built upon two academic papers that introduced the concept of looping in transformer models. By reusing the same block multiple times, they increased the model's effective depth without adding parameters. They also implemented entropy-based halting, which allows the model to stop processing tokens when it reaches a certain confidence level. This approach was tested in three variants, each with different halting strategies, and showed improvements in perplexity. The results, however, were based on a single training run and require further validation with multiple seeds.
The project highlights the challenges of improving small-scale language models and the importance of exploring alternative training methods. The findings suggest that modifying the computation shape can yield better results than simply increasing model size. The researchers emphasized the need for further testing to validate their results and ensure the robustness of their approach.
Source: huggingface
