Hugging Face released a comprehensive evaluation of 13 open-source large language models (LLMs) for Swiss legal tasks in German, French, and Italian. The evaluation tested models on three benchmarks with 55,861 scored samples each, highlighting the importance of task-specific performance over general rankings. The results show that no single model excels across all legal tasks, emphasizing the need to choose models based on specific use cases.
The top-performing models include NVIDIA's Nemotron 3 Ultra 550B, which leads the composite score at 59.0, and Gemma 4 31B, which scores 54.8 overall and excels in multiple-choice questions. GLM 5.2 is the best translator, scoring 66.2, while DeepSeek V4 Pro and Kimi K2.6 closely follow. The evaluation also reveals that the top five models are effectively tied, with a 2.1-point gap between the first and fifth place, and uncertainty ranges of 3.2 to 3.6 points, indicating the need for further research.
The benchmarks include SLDS (summarization), SwiLTra-Bench (translation), and LEXam (open-ended legal reasoning and multiple-choice questions). Each benchmark uses different metrics, with some relying on human judges and others on automated scoring. The evaluation underscores that performance varies significantly by task, with no single model dominating all areas. The findings suggest that model selection should be task-specific, with different models excelling in translation, summarization, and legal reasoning.
Source: huggingface
