What changed
Vals AI released benchmark results showing open-source models can now perform cyber attacks on simulated corporate networks with results comparable to closed models.
In the Lighthouse test, Deepseek V4.1 Flash achieved 21 of 31 objectives, while GPT-6 Sol Daybreak, part of OpenAI's restricted Daybreak program, completed 27 of 31 objectives. The results indicate open models are becoming increasingly effective in cyber operations.
What this means in practice
The findings suggest that open models can now perform complex cyber tasks, such as gaining admin access and stealing passwords, without human intervention. Vals AI notes that the same models used for attacks can also help defenders identify and fix vulnerabilities.
The cost of running these models is significantly lower, with Deepseek V4.1 Flash costing about $3 per run compared to $22 for other models.
The study also highlights that open models often require more tokens to achieve meaningful progress, with most models not making significant strides until they pass 100 million tokens.
Limitations and availability
The benchmark results are based on simulated corporate networks and do not reflect real-world scenarios. The Vals CyberOffense benchmark tests AI agents on isolated machines, starting with only a public web address.
The study also notes that safety measures on open models are largely ineffective. The gap between open and closed models in cyber capabilities has shrunk to four to seven months, according to the UK's AI Security Institute (AISI).
The results align with earlier studies showing open models are becoming more competitive in cyber operations.
