What changed

Turing Award winner Richard Sutton, author of the influential 2019 essay 'The Bitter Lesson,' has written a new post stating that raw compute alone isn't sufficient for intelligence. He emphasizes that AI science needs clever new methods and is 'far from finished.' Sutton has previously criticized the limits of simply scaling large language models (LLMs), calling synthetic data a 'big mistake' due to the infinite complexity of the real world. He advocates for agents with world models that continuously learn from their own experience, rather than relying on static training data.

What this means in practice

Sutton argues that the real world is vastly more complex than what internet text can capture, and current LLMs do not keep learning after training. He highlights the need for continuous learning without losing existing knowledge. AI researcher Jerry Tworek, who left OpenAI to explore new AI approaches, contextualizes Sutton's statement, noting that while the 'bitter lesson' still holds, the role of algorithms becomes more critical as compute increases. Tworek warns that the temporary success of transformer scaling has led to a misreading of the bitter lesson, 'reward-hacking the brains of a whole generation of scientists.'

Limitations and availability

Sutton's position is not new, but his recent post has sparked significant discussion in the AI community. AI researcher Jayden Teoh adds that the bitter lesson has sometimes held back academic research, with promising ideas dismissed for straying too far from its spirit. Teoh argues that this is not Sutton's fault but a misreading of his original message. He encourages the research community to stop dismissing work that appears to deviate from the bitter lesson. The tension between Sutton's call for new learning methods and Tworek's criticism of blind transformer scaling is reflected in a growing wave of startups exploring these gaps. However, prominent industry leaders like OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei remain optimistic about further LLM scaling, citing recent successes as evidence.

Source facts for Richard Sutton Warns Against Overreliance on LLM Scaling: The real world is infinitely complex, Sutton argues, and human expertise as a bottleneck prevents true scaling. The inventor of the bitter lesson has a bitter lesson for LLM enthusiasts. In a recent post, Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. thedecoder: The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts