Father Paolo Benanti, the Pope's AI advisor, warned that the world's largest artificial intelligence labs are acting like a cartel, shaping the future of AI without public oversight. He argues that the current narrative of rapid development and superintelligence distracts from the need for democratic regulation and open debate about how to constrain the companies building the technology.
Benanti emphasized that ethics should not be left to corporate boards, stating, "Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square." He called for standards that ensure AI models remain under human control, rather than being dictated by the labs themselves.
In an essay published alongside safety warnings about AI, Anthropic CEO Dario Amodei called for mandatory third-party evaluations of new AI models and a crackdown on distillation, a process where labs train their AI on proprietary outputs. Others have argued for a total ban on recursive self-improvement, where AI develops future iterations of itself.
Benanti is less interested in restricting AI development methods, which he fears could stifle innovation, and more focused on establishing standards to prevent models from slipping beyond human control. "The narrative of the frontier labs is self-interested, to gain or maintain a position in the market," he said. "This has a really clear name: a cartel."
Benanti also criticized the way some in the AI industry frame superintelligence as something divine or omnipotent, comparing it to the reverence typically reserved for a god.
He argued that this framing threatens to skew the discourse about AI, saying, "We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation going on."
The Pope's encyclical on AI, published in May, warned against the technocratic logic of the industry, which presents "efficiency as the ultimate measure of value" and casts humans as "a project to be optimized." Benanti said the focus on existential risk could distract from the encyclical's ambitions, by giving control of AI's direction to a small group of executives and scientists.
Source: wired
