OpenAI Finds 30% of SWE-Bench Pro Tasks Broken
OpenAI found that about 30% of tasks in SWE-Bench Pro are broken, according to a recent audit of coding benchmarks.
Archive
Browse all published articles.
ARCHIVE
-
-
Anthropic Turns Fable 5 Into Manager Using Sonnet 5
Anthropic is reducing Fable 5's high cost by using it as a manager that delegates tasks to Sonnet 5, achieving 92% performance at 63% cost on SWE-bench Pro.
-
Mistral Launches Robostral Navigate, 8B Model for Robot Navigation
Mistral's Robostral Navigate, an 8B model, achieves 79.4% success rate on R2R-CE benchmark using just one camera.
-
xAI's Grok 4.5 Costs Less Than Fable 5 and GPT 5.5
xAI's Grok 4.5 scores 83.3% on Terminal Bench 2.1, nearly matching GPT 5.5's 83.4%, but trails Fable 5 by one point.
-
Google updates Android AI dev benchmark with Fable 5 and other models
Google has expanded its Android Bench benchmark with eight new models, including Fable 5, to evaluate AI agents in app development.
-
Prime Intellect Raises $130M Series A to Help Enterprises Build AI Agents
Prime Intellect, a startup focused on AI agent development, raised $130 million in Series A funding at a $1 billion valuation, with Radical Ventures leading the round.
-
OpenAI Launches GPT-Live for Real-Time ChatGPT Conversations
OpenAI introduced GPT-Live, a new model allowing ChatGPT to listen and speak simultaneously, improving conversation flow with 75.7% user preference over previous versions.
-
Meta AI Glasses Add Camera Safeguard to Address Privacy Concerns
Meta announced a new camera safety feature for its AI glasses to disable recording if the LED light is tampered with, amid ongoing privacy controversies.
-
General Intuition Seeks to Revolutionize Physical AI with Gaming Data
General Intuition, a $2.3 billion startup backed by Bezos, raised $320 million to develop world models using video game data for physical AI training.
-
AMD Introduces SGLang-ATOM for ROCm-Native LLM Inference
AMD's SGLang-ATOM bridges SGLang serving with ROCm-native execution for AMD Instinct GPUs, offering performance improvements on MI355X and MI308X.
-
AMD Enhances JAX-Triton Support for Gluon and Kernel Launch
AMD improved JAX-Triton to support most Triton features, allowing users to run nearly any Triton or Gluon kernel in JAX with minimal changes.
-
Amazon Bedrock Powers Automated Email Management for Public Sector
Amazon Bedrock enables public sector organizations to automatically sort and prioritize emails, improving response times by 40% in pilot tests.