A trillion-dollar clash of ideas is roiling the artificial intelligence community. Today, in a special episode, our host Ajay Agrawal leads Rich Sutton, Sendhil Mullainathan and Niamh Gavin, and special guest Suzanne Gildert, in a fascinating exploration of the issue: Are Large Language Models (LLMs) sufficiently “bitter lesson pilled” to live up to their hype?
“Bitter lesson pilled” is the AI community’s term of art for scaling with the constantly falling cost of compute (e.g., search and learning). The term arises from Rich Sutton’s 2019 essay, The Bitter Lesson.
As he recently told independent journalist Dwarkesh Patel on the Dwarkesh Podcast, Rich Sutton does not believe that LLMs are sufficiently “bitter lesson pilled.” In other words, Rich believes LLMs suffer from a key vulnerability: A limit exists on their ability to improve – and it’s much closer than we’ve been led to believe.
GUESTS AND HOSTS
Ajay Agrawal, co-founder and partner, Intrepid Growth Partners
Richard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta
Sendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MIT
Niamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist, CEO, Emergent Platforms
Suzanne Gildert, founder and CEO, Nirvanic Consciousness Technologies
LINKS
The Dwarkesh Podcast episode featuring Rich Sutton. The computer scientist Andrej Karpathy’s take. Rich’s original Bitter Lesson essay.
Meta machine-learning engineer Chris Hayduk’s tweet about the debate on X, retweeted by Rich and referenced in this episode by Sendhil.
Good description of the train-fly problem that Sendhil mentioned, from Presh Talwalkar.
Derby Mill series website. Derby Mill is created by the team at Intrepid Growth Partners.
Rich Sutton’s home page. Follow Rich on X
Sendhil Mullainathan’s website. Follow Sendhil on X
Be sure to catch every episode of The Derby Mill Series by subscribing on the following platforms:
YouTube // Spotify // Apple Podcasts
DISCUSSION POINTS
00:00 Cold open
00:39 Context for episode
01:39 The bitter Lesson
02:49 Supervised learning
04:30 Challenge of RL
09:49 Discussing a Tweet
13:30 Rich’s opinion on the big lesson
21:28 Tension in the LLM space
23:25 Behaviour and extrapolation
25:27 What is considered AI
26:05 Final remarks
NUGGETS
Why Squirrels Still Outthink Supervised AI (1801)
Derby Mill Series host Ajay Agrawal asks co-host Suzanne Gildert, why can’t AI learn like a squirrel?
Addressing Rich’s Tweet (1802)
MacArthur Genius Award recipient Sendhil Mullainathan responds to a tweet that underscores a key difference between LLMs and humans.
What Happens if LLMs Don’t Pay Off Soon (1803)
The Bitter Lesson says, “look out if you’re putting all your eggs into the basket of human knowledge,” according to Turing Award recipient Richard Sutton.
DISCLAIMER
The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.
