Description

What happens when AI attempts the same complex work as human experts with 14 years of experience? The answer might reshape our understanding of the economic future.

TL;DR:

• GDP Val tests AI on complex, multimodal tasks requiring handling of CAD designs, spreadsheets, and presentations
• Tasks are created from actual professional work products that take humans an average of 7 hours to complete
• Claude Opus performed best with 47.6% of its deliverables rated as good as or better than human experts
• AI shows potential to make workflows 40% faster and 63% cheaper when paired with human oversight
• 3% of AI failures were classified as "catastrophic," including incorrect medical diagnoses and suggestions of financial fraud
• Simple prompt improvements like asking models to self-check their work significantly reduced formatting errors
• Current models still struggle with ambiguity and tasks requiring tacit knowledge or complex human interaction

GDP Val represents a fundamental shift in how we evaluate artificial intelligence. Rather than abstract academic metrics, this new benchmark from OpenAI measures how well frontier AI models handle real-world economic tasks across nine major sectors worth $3 trillion annually. 

The methodology is ruthlessly practical—AI models must complete complex assignments that typically take human experts seven hours, handling everything from CAD designs to financial spreadsheets while synthesizing information from up to 38 reference documents.

The results are both promising and sobering. Claude Opus led the evaluation with 47.6% of its outputs rated equal to or better than work from professionals at organizations like Apple, Goldman Sachs, and Boeing. When integrated into realistic workflows with human oversight, these models demonstrated potential to make knowledge work 40% faster and 63% cheaper. 

Yet failures remain significant—3% were classified as "catastrophic," including incorrect medical diagnoses and recommendations of financial fraud.

Perhaps most valuable is GDP Val's illumination of where AI currently excels (document formatting, data analysis) and where it falters (following complex instructions, handling ambiguity). 

This economic lens offers businesses and policymakers unprecedented clarity about AI's near-term impact on knowledge work, while highlighting that the highest-value human skills—tacit knowledge, real-time collaboration, and complex communication—remain beyond current AI capabilities. 

How quickly will that gap close? That's the trillion-dollar question worth pondering.

Listen into a audio version of this report created using Google Notebook LM for your listening pleasure.

Link to research: GDPval.pdf 

Support the show

𝗖𝗼𝗻𝘁𝗮𝗰𝘁 my team and I to get business results, not excuses.

☎️ https://calendly.com/kierangilmurray/results-not-excuses
✉️ kieran@gilmurray.co.uk
🌍 www.KieranGilmurray.com
📘 Kieran Gilmurray | LinkedIn
🦉 X / Twitter: https://twitter.com/KieranGilmurray
📽 YouTube: https://www.youtube.com/@KieranGilmurray

📕 Want to learn more about agentic AI then read my new book on Agentic AI and the Future of Work https://tinyurl.com/MyBooksOnAmazonUK