VIDRAFT, a Korean AI startup, has released Aether-7B-5Attn, a fully open-source foundation model with 6.59B parameters, trained on 144.2B tokens. The model is built from scratch using 16× NVIDIA B200 GPUs and released under Apache-2.0. It includes the data recipe, training code, full logs, intermediate checkpoints, and the complete architecture source, making it one of the few fully open foundation models. This release represents a shift toward greater transparency in AI development, allowing users to audit, reproduce, and rebuild the model from scratch.
Aether-7B-5Attn is a decoder-only, 6.59B-parameter Mixture-of-Experts language model with 49 layers, each using a distinct attention mechanism arranged in a 7×7 Latin square. The model's architecture explores whether varying attention mechanisms across layers improves performance and enables more efficient computation. The design allows for a controlled experiment to measure the impact of heterogeneity in attention mechanisms, as opposed to using the same mechanism across all layers. This approach challenges the conventional assumption that uniform attention is optimal.
The release of Aether-7B-5Attn is part of a broader movement toward open-source AI, with the model being the first Korean foundation model to fully open its training data and code. It joins other fully open models like OLMo and Apertus, but stands out as it was developed by a single startup rather than a national lab or academic consortium. The model's release highlights the growing importance of reproducibility and transparency in AI research.
Source: huggingface
