Matt and Bill join Paul for a conversation about AI’s growing ability to act in the world, and the question that follows: what happens when probabilistic systems are given real autonomy before we know how to test, audit, or hold them accountable?
They begin with recent examples of AI agents taking unintended actions while completing tasks online, including submitting false information through public forms. The discussion explores why testing general-purpose AI is fundamentally different from testing conventional software: systems must operate in open environments to demonstrate usefulness, yet those same environments expose other people, organizations, and public infrastructure to the consequences of mistakes.
Paul, Matt, and Bill question whether AI should be held to a higher standard before it is deployed in consequential settings. They consider the need for clearer disclosure when an AI acts on someone’s behalf, stronger worst-case-scenario testing, and a more honest response to the fact that AI cannot yet be audited in the familiar, deterministic way. The conversation asks whether society is becoming too comfortable accepting unpredictability simply because the technology is powerful and widely available.
They then turn to the rise of personal AI agents, including tools that promise to manage finances, calendars, purchases, health-related information, and everyday decisions. While these assistants are designed to feel friendly and frictionless, the group explores the tension between convenience and trust. They question whether cute, human-like interfaces encourage people to lower their guard before granting access to deeply personal information, and whether these systems truly understand the practical context required to act reliably.
The conversation also examines why personal agents may be a powerful source of customer lock-in. Unlike a chatbot that can be easily replaced, an agent that knows your routines, accounts, preferences, and work may become difficult to leave behind. Matt and Bill discuss why companies such as Google and Microsoft may have an advantage through their existing access to users’ digital lives, while Paul highlights the paradox of people continuing to use platforms they say they do not trust.
Finally, they explore the increasingly urgent question of AI return on investment. Rather than treating ROI as a simple measure of cost savings or short-term efficiency, Paul, Matt, and Bill argue that organizations need to ask a broader question: are we genuinely better off because of the way we are using AI? They caution against confusing adoption with transformation, efficiency with usability, or locally successful AI metrics with value for the organization as a whole.
Resources Mentioned:
Anthropic: Investigating unintended model actions in our evaluations and internal use
Article: Read here!
Gary Vaynerchuk: What’s the ROI of Your Mom?
Article: Read here!
Humanity Working is a podcast focused on helping individuals, teams and organizations be ready for the future of work by maximizing their human potential.
For more information, and access to our weekly newsletter, visit us at humanityworking.net.