MirrorCode Benchmark Tests AI's Programming Skills
Epoch AI's MirrorCode benchmark assesses AI models' ability to recreate programs from scratch, with one task costing $2,600 to run.
ARTICLES BY MAYA CHEN
LATEST NEWS
-
-
Unconventional AI Aims to Cut AI Power Use by 1,000x
Unconventional AI, led by Naveen Rao, claims its oscillator-based architecture can reduce AI power consumption by up to 1,000 times, with its first model, Un-0, matching state-of-the-art image-generation systems.
-
AI Systems Begin Producing Publishable Math Research
Google DeepMind's AI system Aletheia autonomously generated Ph.D.-level math research, marking a milestone in AI's role in mathematics.
-
Hugging Face Releases DoctoBERT for French Medical NLP
Hugging Face introduces DoctoBERT, a French medical encoder, trained on a new corpus of web data curated with domain-specific filters and rephrasing techniques.
-
Falconer Outperforms Competitors in Enterprise AI Benchmarks
Falconer achieved a 64% win rate against Notion in real support questions, outperforming other tools in two critical benchmarks.
-
UC Berkeley Study Finds AI Inflating Student Grades
A UC Berkeley study found grades in writing and coding courses rose 13 percentage points after ChatGPT launched in November 2022.
-
Sam Altman Blames AI Stagnation on Underestimated Scaling Potential
OpenAI CEO Sam Altman claims a generation of researchers hindered AI progress by underestimating scaling's impact, citing recent model achievements.
-
Qwen3.6 27B Fine-Tuning Shows Harness-Specific Behavior Shifts
Research on Qwen3.6 27B reveals that harness-specific fine-tuning can significantly alter model behavior, with v2 showing 40.45% performance on Pi harness compared to base model's 42.70%.
-
AA-Briefcase Benchmark Shows AI Struggles With Real Knowledge Work
A new benchmark reveals AI models fail to complete most real-world knowledge tasks, with top models solving just 3 percent of tasks.
-
OpenAI Tests Beneficial Trait Training for Safer AI Models
OpenAI researchers found that small amounts of beneficial trait training improved model safety across 44 of 53 benchmarks, according to a blog post.
-
OpenAI Model Aids in Diagnosing Rare Childhood Diseases
OpenAI's o3 Deep Research model helped identify 18 diagnoses from 376 previously unsolved cases, achieving a 4.8% additional diagnostic yield.
-
AMD Optimizes Matrix3D for 3D World Generation on GPUs
AMD improved Matrix3D, a 3D world generation framework, with optimizations that cut end-to-end generation time by 54% on the MI250 GPU.