Martin Tingley, Head of Windows Experimentation at Microsoft and former Head of the Experimentation Platform Analysis Team at Netflix, talks about why humans are the bottleneck in experimentation, and how a five-level maturity framework points the way toward self-optimizing software.
Our conversation traces the path from basic hypothesis testing to a frontier where Generative AI creates, evaluates, and refines product variants in a closed loop. We explore the architectural shift required to move from testing single variants to optimizing entire parameter spaces, and how startups are already using AI to generate production-ready landing pages for Fortune 500 companies in hours rather than weeks. Tingley also shares a strategic lens on "experimentation programs," explaining how plotting the distribution of treatment effects across different product areas can serve as a powerful tool for capital allocation and high-level strategy.
LINKS
