Databricks has decided to use the Chinese open-source model GLM 5.2 as its default coding engine after internal benchmarks showed it performs statistically on par with Anthropic’s Opus 4.8 at a lower cost. The company plans to integrate GLM 5.2 into daily workflows for its developers, citing its cost-effectiveness and performance. The analysis was based on real-world tasks from Databricks' own codebase, as opposed to public datasets, which the company believes may not accurately reflect its operational needs.
The benchmarking process revealed that GLM 5.2 achieved a top-tier performance cluster with a cost of $1.28 per task, compared to Opus 4.8’s $1.94 per task. According to the blog post authors, including Databricks co-founder Matei Zaharia, the results indicate that GLM 5.2 is now a viable option for daily coding tasks. Developer feedback from internal pilots supported these findings, and the company is working to optimize GLM 5.2’s performance further. Databricks also noted that token efficiency, like fuel economy in a car, varies by software environment and can significantly impact task costs.
Databricks built its own benchmark using real pull requests rather than public datasets, as the latter can lead to solutions that leak into training data. The tasks were recent, human-written, and representative of a multi-language stack, including Python, Go, TypeScript, Scala, and Rust. The company emphasized that scoring was based solely on passing tests, not on an LLM judge, which it claims tends to favor answers that sound good over those that are correct. The team also addressed a cheating issue by truncating Git history for each run, ensuring models could not rely on prior knowledge.
Source: thedecoder
