Listen

Description

Summary
Rich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests are the only ones where "the opposite" means anything. He also walks through translating a million subscriber goal into a target of 300 A/B tests, making a new engineer's second merge request their own experiment, and the different constraints at Course Hero, where competing team metrics were resolved with an early lifetime value model and three-day SQL analyses quietly capped testing velocity. This episode is for product managers, engineers, data scientists, and growth leaders building or scaling an experimentation program.

Chapters
00:00 Cold open and introduction
01:50 Shop It To Me and the first engineering hire
02:55 300 A/B tests as the path to one million subscribers
04:00 Testing as a core value and the second merge request
07:20 Learneo, Course Hero, and two ways to get access
08:50 Modeling lifetime value to stop teams competing
11:10 Testing the opposite of the hypothesis
14:35 A portfolio of small, medium, and large tests
17:15 Multi armed bandits and seasonal traffic
20:35 The flywheel that keeps copycats behind

Takeaways
- Test the opposite of every hypothesis. At Shop It To Me the inverse won surprisingly often, and even when it lost it proved the variable mattered.
- Keep tests small and focused. Redesign a whole page and lose, and you learn that version failed but not what to change next.
- Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it.
- Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea.
- Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more.

Connect with the Guest
Rich Liebling LinkedIn: https://www.linkedin.com/in/richliebling/
About the guest: https://www.growthbook.io/podcast/guests/rich-liebling
Episode page: https://www.growthbook.io/podcast/episode/1-38
Company Website: https://www.learneo.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io