In this unpacked episode, the team further expands its discussion of themes that came up in episode seven, which explored the automation of customer support with artificial intelligence. Our guests in that episode were a duo that is leading efforts in that space: CEO Mike Murchison and chief product and technology officer Mike Gozzo from Ada.
In this episode, Intrepid Growth Partners cofounder and partner Ajay Agrawal leads the discussion with Intrepid Senior Advisors Rich Sutton (Turing Award winner), Sendhil Mullainathan (MacArthur genius grant recipient) and Niamh Gavin (CEO, Emergent Platforms).
In the previous episode, we learned that Ada’s north star is “percent automated resolutions”, or the percentage of customer inquiries that are fully resolved by AI without human intervention. One challenge is that Ada relies on large language models (LLMs) rather than action-based goals, often requiring human agents to step in when confidence is low.
“It’s a mistake to think that [Ada’s AI agents] have goals,” says Sutton. ”What we have instead … is we have [AI agents] mimicking people.”
All of which raises the question of how customer support will evolve as this technology advances towards the limit.
Our team also debates the need for clear, objective measures of AI performance and the challenges of achieving true goal-oriented AI systems.
Our panel of experts:
Ajay Agrawal, co-founder and partner, Intrepid Growth Partners
Richard Sutton, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta
Sendhil Mullainathan, MacArthur Genius grant recipient and professor, MIT
Niamh Gavin, Applied AI scientist, CEO, Emergent Platforms
Sutton, Mullainathan and Gavin are all Intrepid Growth Partners’ senior advisors.
LINKS
Ada website
This episode extends the discussion from Derby Mill episode 07: Customer Support Rich Sutton’s home page. Follow Rich on X
Sendhil Mullainathan’s website. Follow Sendhil on X
Be sure to catch every episode by subscribing on the following platforms:
YouTube // Spotify // Apple Podcasts
DISCUSSION POINTS
00:00 Introduction and opening credit
02:00 Ada refresher
03:46 Clip: Testing harness
06:50 Clip discussion begins
10:15 What are goal-based objectives?
13:30 Is this the year of the agent?
17:40 What makes agents goal-oriented
19:20 Decision-making fundamentals in AI
21:27 Clip: Automating system improvement over time
23:32 Clip discussion begins
30:13 Automating the evaluation process
34:08 What could Ada look like in the limit?
41:03 Closing remarks
NUGGET 01: Vertical vs. Horizontal Competition
Fine-tuning used to be costly and impractical, pushing companies to open-source solutions—only to revert to OpenAI due to complexity. Now, companies like Ada build on top of model providers, offering flexibility while managing AI’s complexity. Niamh discusses the competitiveness between verticalized AI (industry-specific applications) and horizontal AI (broad sector models).
NUGGET 02: The Challenge of Interpretability
Ada's evaluations rely on human judgment. The challenge here is interpretability—determining whether an outcome is truly good without direct human input. Rich Sutton offers potential solutions, including using reinforcement learning with human feedback (RLHF) as a proxy measure trained on high-quality data.
NUGGET 03: Benchmarking vs. Deployment in the Field
Niamh and Sendhil discuss how, despite concerns about hallucinations in AI-powered customer service, CEOs adopt GenAI more for signaling competence than for real effectiveness.
DISCLAIMER
The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.
Share this post