Researchers from institutions including the International Centre for Theoretical Sciences (ICTS), Harvard University, KU Leuven, and Google DeepMind released TheoPhysEval, a benchmark of 50 unpublished physics problems. The benchmark tests routine research work, not grand challenges.
TheoPhysEval contains problems that require applying known methods to slightly modified questions, often involving calculations from recent research papers. According to the authors, a leading expert in the relevant subfield would need about a day or less to solve a typical problem without AI help.
GPT-6-Astra, running in OpenAI's Codex coding environment, solved 49 of 50 problems, or 98 percent, at an average cost of under $3 per problem and just under $116 for the whole benchmark. The benchmark tests routine research work, not grand challenges.
"TheoPhysEval deliberately leaves out major open questions and focuses on what we call 'turn-the-crank' problems," said Suvrat Raju, a co-author of the older paper. "These are problems where researchers apply known methods to a slightly modified question."
The problems span eleven subfields of theoretical physics, from fluid flows and biological processes to particle physics and cosmology. Each agent ran the test twice, once under realistic conditions with web search and once in isolation without internet access.
"Internet access made surprisingly little difference," noted the authors. "Without web search, Astra and Opus each scored 92 percent. Many problems depend on formulas from research papers that the models usually didn't receive."
Source: thedecoder
