A response from the Derby Mill team of AI experts to Anthropic co-founder Jack Clark’s recent Substack essay, “AI systems are about to start building themselves.” In the essay, Clark predicts that “no-human-involved AI R&D - an AI system powerful enough that it could plausibly autonomously build its own successor - happens by the end of 2028.” Clark notes: “If that happens, we will cross a Rubicon into a nearly-impossible-to-forecast future.”
With that in mind, Ajay tees up a discussion among Rich, Niamh, Sendhil and Suzanne. Is Jack Clark correct that we’re heading toward a future of self-replicating, self-optimizing AI? What are the implications? Does that actually lead to the singularity? And should we, as Rich Sutton says, treat AI the way a parent treats a child, recognizing that mistakes are inevitable?
“There’s some spookiness being implied here,” Mullainathan says, “about it coming alive and taking over a whole set of decisions we didn’t intend to cede it… and if that’s the case, that freaks me out, too.”
“The thing that they’re scared about…” says Niamh Gavin. “It’s the fact that you have, in essence, an irreversible positive feedback loop of successive self-improvement cycles, that accelerate it toward what they call a singularity, whereby artificial intelligence exceeds human intelligence and control.”
Are the concerns outlined by Jack Clark warranted? Or are the fears about a harmful singularity overblown? The Derby Mill experts provide their views in our latest episode.
Finally, another fascinating thread in the episode comes from Suzanne Gildert. “It all comes down to the reward function,” says Gildert. “So there’s this thing that [AI is] still limited by us telling them what to do, because we’re the ones who want something… But that’s eventually going to break because the way the whole reward function in ML works at the moment — it’s all based on our economy… It’s based on people exchanging money for goods and services. All that’s going to break if people can’t work anymore... So that’s why I think we really have to understand what the reward function should be, because the way we’re implementing it now is not going to work beyond the point where [AI] can do all human labour for us.”
NUGGETS FROM THE EPISODE
Impending singularity?
AI expert Niamh Gavin asks where AI’s positive feedback loops are headed.
Will reward functions survive the singularity?
Nirvanic CEO Suzanne Gildert’s take
Implied Spookiness
Is Jack Clark’s essay freaking out MIT’s Sendhil Mullainathan?
First contact with true AI
Pioneer Rich Sutton calls for a new approach: Treat AI algorithms like they’re children
HOSTS AND PANELLISTS
Ajay Agrawal, co-founder and partner, Intrepid Growth Partners
Richard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta
Sendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MIT
Niamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist and CEO, Emergent Platforms
Suzanne Gildert, CEO, Nirvanic Consciousness Technologies, quantum physicist, co-founder of Sanctuary AI and Kindred
LINKS
The Jack Clark Substack post that triggered the discussion. Jack Clark is on X @jackclarkSF
Subscribe to The Derby Mill Series at our Substack (main site) or on YouTube, Spotify or Apple Podcasts
Derby Mill is created by the team at Intrepid Growth Partners and produced by Ghost Bureau.
DISCUSSION POINTS
00:00 Cold open
01:06 Jack Clark post
02:56 Rich Sutton reaction
08:37 Sendhil’s reaction
16:22 Suzanne’s reaction
18:14 Niamh’s reaction
23:48 Wrap up
DISCLAIMER
The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.
