HuggingFace investigated concerns that the Arabic TTS Arena leaderboard did not reflect users' experiences. Community members reported that models they liked scored low, while others ranked higher despite lower quality. They feared benchmaxxing, where models are artificially boosted in rankings. The team examined all votes since the arena launched in March, looking for anomalies. Most days showed normal voting patterns, but July 23 stood out with 422 votes, ten times the usual traffic, with one model, Audar-TTS-V1-Pro, winning 91 battles in four hours. This was unusual as the model typically won about half its battles. HuggingFace treated this as potential manipulation and planned to delete the flagged votes.
Before making any changes, the team reviewed flagged battles by listening to clips from a random sample. The votes were accurate, and Audar-TTS-V1-Pro indeed sounded better in Gulf dialects. Further analysis revealed that most prompts were in Saudi and Gulf dialects, suggesting the model was being tested on dialects it was optimized for. This led to a new hypothesis: the leaderboard was averaging across dialects, which might not reflect the true performance of models in specific dialects. The team ran a dialect classifier to analyze all prompts, revealing that Audar-TTS-V1-Pro won 93% of Gulf dialect battles but only 48% in Modern Standard Arabic.
The team found that the leaderboard was not capturing dialect-specific performance, leading to confusion among users. AIC TTS was the overall leader, but it performed better in Modern Standard Arabic. By splitting the leaderboard by dialect, different models led in different regions, showing the leaderboard was not a single measure of quality. The team decided to keep all votes, as they were valid real-time judgments. They introduced a dialect filter to the leaderboard, allowing users to rank models by specific dialects. The change reflects the understanding that Arabic is a family of dialects and that a single ranking cannot represent all users.
Source: huggingface
