podcast
details
.com
Print
Share
Look for any podcast host, guest or anyone
Search
Showing episodes and shows of
Growthbook
Shows
The Experimentation Edge
How FLYR helped an airline grow revenue by double digits
SummaryAirline seats are perishable inventory, and small decisions about pricing and product offers can have enormous consequences. Vamsi Kodam, Senior Director of Product Management at FLYR, joins host Ashley Stirrup on The Experimentation Edge to explain how FLYR is helping airlines modernize commerce, personalization, pricing, and offer orchestration.Vamsi revisits a formative "human versus machine" experiment in airline revenue management: total revenue barely changed, but looking across the network revealed where human overrides helped, did nothing, or actively hurt performance. He also shares how FLYR applied multi-armed bandit experimentation to an airline pricing problem...
2026-11-20
20 min
The Experimentation Edge
How to rebuild trust in an A/B testing platform
SummaryMohan Dasannacharya, Ex Albertsons, Shutterfly, eBay, joins host Ashley Stirrup to talk about what it takes to trust the numbers an experimentation platform gives you. Mohan's path runs from physical design to UX at eBay, product and experimentation at Shutterfly, and now the in-house experimentation and analytics platform at Albertsons, which moved off Adobe Target and cut results reporting from about 7 days to overnight. Mohan explains how A/A tests that kept producing winners led Albertsons to split tests into A, A and B for almost 2 years, why a flat add-to-cart test turned o...
2026-11-19
27 min
The Experimentation Edge
From Opinion to Evidence: Evaluating AI Ads at Walmart Connect
SummaryHow do you measure the quality of an AI-generated ad when designers, legal teams, and advertisers may all define “good” differently? Nishanth Kadiyala, Product Leader at HelloFresh and formerly an ad tech product leader at Walmart Connect, joins Ashley Stirrup on The Experimentation Edge to explain how the team shifted from subjective creative preferences to measurable acceptability, compliance, and performance.Nishanth walks through the realities of building creative automation for smaller advertisers: segmenting users, validating AI output against a large evaluation dataset, running A/B tests to earn stakeholder buy-in, and letting real advertiser need...
2026-11-17
32 min
The Experimentation Edge
What Wealthsimple decides to test and what to just ship
SummaryDelina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams.Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and...
2026-11-12
36 min
The Experimentation Edge
How Andy Denning decides which A/B tests are worth running
SummaryAndy Denning, Product Leader, joins host Ashley Stirrup to talk about the judgment that comes before any A/B test. Drawing on his experience with a two-sided jobs marketplace serving job seekers and recruiters across the UK and US, Andy explains why underpowered, badly chosen tests can create false confidence, how experimentation brings organizational honesty, and why his teams start with the riskiest assumption instead of the most comfortable one. He breaks down a US branding test that barely moved applications, how traffic and market shape test design, and why AI makes judgment, not test volume, the...
2026-11-11
32 min
The Experimentation Edge
How GE HealthCare tests for quality leads, not just more clicks
SummaryTom Gores, Director, Product - Digital Experience & E-commerce at GE HealthCare, joins host Ashley Stirrup to share what changes when you take an experimentation program from B2C retail to B2B healthcare. Tom built his first A/B testing program at Kohl's 12 years ago and now runs a centralized program at the center of GE HealthCare's digital ecosystem, where buying cycles can take a year and each qualified lead carries much higher stakes. Tom explains why a Contact Us test that lifted clicks also lifted junk leads, how his team turns one-off p...
2026-11-10
18 min
The Experimentation Edge
How Glassdoor turned a handful of A/B tests a year into dozens a quarter
SummaryJeremy Echols, Director of Digital Analytics at Glassdoor, part of Indeed, joins host Ashley Stirrup to talk about building an experimentation program around people, not sessions. Jeremy came to analytics from an unusual path, as a former pastor with a master of divinity, and he sees math and statistics as a way to do applied psychology. Jeremy explains how Glassdoor went from a slow homegrown system and a handful of tests a year to dozens of tests a quarter, with a center of excellence, office hours, Amplitude templates and a shared data language. H...
2026-11-09
22 min
The Experimentation Edge
How RealPage finds its biggest A/B test wins, one funnel at a time
SummaryAndrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joins host Ashley Stirrup to share how experimentation works inside Loft, RealPage's end-to-end renter app. Andrew has stood up experimentation programs at 3 companies, and today he runs a portfolio of moving services where a winning A/B test ties directly to revenue. Andrew explains how a small, centralized optimization team ships tests in Optimizely faster than engineering release cycles allow, and how he builds a culture of experimentation with funnels, group ideation, the Opportunity Solution Tree and simple test templates. He also b...
2026-11-05
32 min
The Experimentation Edge
Why you're already wrong 75% of the time
SummaryIn this special episode of The Experimentation Edge, host Ashley Stirrup looks back at lessons from more than 50 guests with two people from GrowthBook: co-founder and CEO Graham McNicoll and Head of Customer Success & Strategy Cris Dizon, who ran experimentation programs at The Home Depot, Walmart and Peacock. The starting point is a humbling number: across companies, only about 25% of experiments win, so even your best ideas are wrong most of the time. Graham shares the untested redesign that quietly cut new-user sign-ups by about 20% for six months, and the bug that set a...
2026-11-04
30 min
The Experimentation Edge
How Preply runs close to 1,000 experiments a year
SummarySebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team. Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Pr...
2026-11-03
32 min
The Experimentation Edge
Building a Culture of Experimentation in Sports Analytics with New York Red Bulls
SummaryVamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin.Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized...
2026-10-29
25 min
The Experimentation Edge
Motive's Shiva Manjunath on CRO as a mindset, not a role
SummaryShiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas.The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For...
2026-10-27
38 min
The Experimentation Edge
Engagement with safety: Headspace's guardrails for AI tests
SummaryMary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely. Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor te...
2026-10-20
40 min
The Experimentation Edge
How Walmart Connect learned to align cross-sell with shopper intent
SummaryArv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how months-long ad-free holdouts measure what ads cost in customer trust.Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that...
2026-10-15
19 min
The Experimentation Edge
Why more info lifted VidaXL's account creation nearly 40%
SummarySapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the biggest winner of her career: B2B buyers were abandoning a registration page, a survey showed they could not see what the subscription included, and adding that information to the signup flow lifted account creation 38.7%.Host Ashley Stirrup also hears the story of a localized payment method the whole company expected to win, why it lost to...
2026-10-14
19 min
The Experimentation Edge
The auto researcher: How FanDuel finds a winning model in 300 attempts
SummaryMichael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for roughly $300 in tokens, handing back the winner as readable code. Host Ashley Stirrup gets the three things that surprised FanDuel as it scaled: the agent builds its own features (756 down to 20), the holdout is enforced so it cannot cheat, and it reframes problems when the scoring rules tell it to.The conversation covers...
2026-10-13
40 min
The Experimentation Edge
How Quill turned loyalty points into a 1% conversion lift
SummaryVipul Tyagi, Senior Product Manager at Quill, owns the online experience from add to cart through checkout, with a focus on getting customers to use Quill's loyalty program. Host Ashley Stirrup digs into why a homepage rewards widget drew under 0.5% engagement over 60 days, and how moving the value to the cart with a Lego-inspired points to dollars option lifted conversion by 1% while shrinking the points liability on Quill's books.The conversation covers Vipul's 8 winners out of 11 loyalty initiatives and the BillionToOne advice behind them, how he coaches new PMs on primary and secondary hypotheses and...
2026-10-08
26 min
The Experimentation Edge
What Grubhub tests for big product bets before they ship
SummaryGrubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users.She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews...
2026-10-08
21 min
The Experimentation Edge
How MilliporeSigma lifted add to cart with a smarter search
SummaryDorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell changes country by country, and personas stop being a marketing exercise.She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so...
2026-10-06
20 min
The Experimentation Edge
PayPal's $180 million experimentation win
SummaryGaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700...
2026-09-30
31 min
The Experimentation Edge
ServiceNow's Customer Zero approach to AI experimentation
SummaryServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer.She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should...
2026-09-29
27 min
The Experimentation Edge
Why Twilio ships on signals instead of significance
SummaryWanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not.The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on...
2026-09-24
24 min
The Experimentation Edge
Why GoPro stopped judging A/B tests by win rate
Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test program.Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent...
2026-09-22
26 min
The Experimentation Edge
What Samsung learned bringing B2C rigor to B2B
SummaryWhat changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the data, and why not everything deserves an A/B test. She walks through the add-on experiment that nearly doubled attach sales once her team realized...
2026-09-17
30 min
The Experimentation Edge
Even a loss is a win: Charlie Health's approach to experiments
SummaryOn this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut about 80% of its market for one feature and roughly quadrupled retention, and how premortems that ask "what went wrong?" before launch make dissent safe and surface safeguards that avoid catastrophe. He also shares the two questions he...
2026-09-15
30 min
The Experimentation Edge
Realtor.com on using your AI as a junior data scientist
SummaryWhat do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up.Chapters00:00 Cold open and welcome01:10 From growth hacker...
2026-09-10
20 min
The Experimentation Edge
How Zalando connects every experiment to its North Star
SummaryHow do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery feeds homepage launch when success had no established metric, and how a KPI tree cascades the company North Star down to the controllable inputs teams ship every day. She also looks ahead to LLM-based agents...
2026-09-09
20 min
The Experimentation Edge
Learneo on testing the opposite of every hypothesis
SummaryRich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests are the only ones where "the opposite" means anything. He also walks through translating a million subscriber goal into a target of 300 A/B tests, making a new engineer's second merge...
2026-09-08
21 min
The Experimentation Edge
The four questions Early Warning asks before any A/B test
SummaryWhat separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the four questions to ask before launching any A/B test, why 85 to 90% of tests are supposed to fail, how pre-registration and kill criteria stop p-hacking before it starts, the pitfalls that fake wins (novelty effects, hidden heterogeneity, multiple comp...
2026-09-03
21 min
The Experimentation Edge
How Supercell A/B tests 300 million players without breaking trust
SummaryOn this episode of The Experimentation Edge, Ashley Stirrup talks with Shan Huang, data scientist on the central experimentation team at Supercell, the Helsinki mobile game company behind Clash of Clans, Clash Royale, Brawl Stars, Hay Day, and Boom Beach. Shan explains how a famously decentralized, creative-first company with 300 million monthly active users runs fewer than 100 A/B tests a quarter and why that restraint is deliberate, how Supercell announces experiments to players in advance and promises make-up events to keep testing fair, and how importable AI skills now let anyone at the company analyze their own...
2026-09-01
23 min
The Experimentation Edge
How Clover experiments when billions of dollars flow through daily
SummaryHow do you run an experimentation program when classic A/B testing is off the table? Ben Schein, Director of Product Management at Clover, joins host Ashley Stirrup to explain how a platform serving 300,000+ merchants and processing billions of dollars daily proves every feature through pilots and ground-level testing before rollout, why uncertainty and downside — not feature visibility — decide testing depth, and what his years leading product at Shake Shack taught him about turning a checkout funnel into a brand channel. This episode is for product managers, engineers, and data scientists building experimentation programs where the stakes are...
2026-08-27
30 min
The Experimentation Edge
Why JobLeads says one test won't move you, but 100 will
SummaryWhat happens when the most logical feature you've ever built has zero impact? In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Edd Saunders, product experimentation manager at JobLeads, to unpack the pizza personalization experiment that cut an ordering flow from 22 clicks to 5 and changed nothing. Edd shares the problem mapping framework he uses to move new experimenters from solution space to problem space thinking, how JobLeads grew from 0.3 to 2.8 experiments per month, and why democratizing experimentation across a whole company comes down to habit change rather than education. A...
2026-08-26
26 min
The Experimentation Edge
Why The Aspen Group targets a 25% win rate
SummaryWhat does a winning A/B test mean when your website serves 1,100 dentist owned offices? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arie Polycarpou, Senior Manager of Test and Learn at Aspen Dental, about building experimentation programs at Kohl's, Marriott, Total Wine, and now the largest company in The Aspen Group's healthcare retail portfolio. Arie explains why appointment bookings are the North Star but never the whole story, why he deliberately targets a 25% win rate and expects it to fall as the program matures, and why he applies a haircut to every...
2026-08-20
23 min
The Experimentation Edge
How Principal A/B tests on customers who don't exist
SummaryWhat if you could A/B test on customers who don't exist before spending a single live impression? Erika Dunn, assistant director of data science at Principal Financial Group, joins host Ashley Stirrup, CMO at GrowthBook, to share how she built synthetic digital audiences entirely in house: profiles shaped by real data that rank content by likelihood of engagement, whose first live A/B test selection just beat the control. They also dig into the looping metric her team built in SQL to find where customers get stuck without heat-mapping tools, why testing gets watered down into "...
2026-08-18
31 min
The Experimentation Edge
Why US Bank considers missing even 1% of customers unacceptable
SummaryHow does a major bank scale experimentation when even one percent of customers missing an experience is unacceptable? Vijay Lal, Lead Product Manager for Experimentation at US Bank, joins host Ashley Stirrup, CMO at GrowthBook, to share how his team made their experimentation platform self serve for non technical marketers, how a login widget experiment led to a two second fallback that accounted for every customer, and why metrics should be driven by hypotheses instead of handed down by leadership. They also dig into where AI genuinely saves time in experiment analysis, why a human in the...
2026-08-11
22 min
The Experimentation Edge
Why Farfetch manages by learning rate, not win rate
SummaryLuis Trindade, Principal Product Manager of Experimentation at Farfetch, joins host Ashley Stirrup to explain how one of the world's largest luxury marketplaces built its own experimentation platform and the culture around it. Luis covers the move from a hybrid setup with an external testing vendor to Fabs 2.0, the in house system where a feature toggle is the single entry point for every experiment, why Farfetch manages by learning rate instead of win rate, and the two year Inspire experiment that replaced the world's leading recommendation engine vendor. He also shares how a deliberately shrinking center of...
2026-08-05
40 min
The Experimentation Edge
How Cogniteer Built an Experimentation Engine From Scratch
SummaryIn this episode of The Experimentation Edge, host Ashley Stirrup talks with Fabian Hans, founder and behavioral psychologist at Cogniteer, a consultancy that helps enterprises build in-house experimentation programs and raise both test velocity and win rate. Drawing on fifteen years in conversion rate optimization, Fabian explains why mass-producing the same A/B tests across clients quietly kills learning, why most ecommerce drop-offs are structural rather than your fault, and how matching the interface to how people actually buy, new versus returning, B2C versus B2B, can move conversion far more than another button. It is...
2026-07-23
33 min
The Experimentation Edge
How Fin A/B Tests Millions of Samples in Days
SummaryOn this episode of The Experimentation Edge, host Ashley Stirrup talks with Pedro Tabacof, Principal Machine Learning Scientist at Fin (formerly Intercom), about how one of the most advanced AI customer support agents in the world is built on relentless experimentation. Pedro explains why unit tests don't work on non-deterministic AI, how Fin runs up to two dozen concurrent A/B tests pulling millions of samples in days, and shares two counterintuitive experiments: one where slowing the agent down improved every metric, and one where adding more context made Fin more helpful and more prone to fake...
2026-07-21
45 min
The Experimentation Edge
How Kargo turns losing experiments into competitive edges
SummaryIn this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with James Falzone, Director of Product Management at Kargo, to unpack how a high scale ad tech marketplace turns failure into its biggest advantage. James explains how Kargo connects advertisers to publishers through real time auctions that resolve in milliseconds across up to 10 billion ad requests a day, why experimentation is embedded in the company's culture rather than siloed in a team, and what happened when a winning click optimization model failed completely after being copied to a new customer type. The...
2026-07-14
22 min
The Experimentation Edge
The 'wine effect' and other surprises that reshaped how Box runs e-commerce experiments
SummaryIn this episode of The Experimentation Edge, host Ashley Stirrup talks with Danielle Olean, Director of E-commerce at Box, about what it really takes to build a culture of experimentation inside a B2B company. Drawing on 15 years across B2C and B2B at Wayfair, Drizly, Zoom, and now Box, Danielle explains why experimentation belongs to every product team and not just e-commerce, walks through a pricing page saga of one win and two losses that exposed the limits of simplification, and shares the "wine effect" test that won for a reason no one predicted. It's...
2026-07-09
27 min
The Experimentation Edge
Diligent explains why moving on from an experiment might cost you
SummaryDan Layfield, Director of Product Management at Diligent, joins host Ashley Stirrup on The Experimentation Edge to trace what fifteen years of A/B testing across Codecademy, Uber Eats, and the Fortune 1000 boardroom actually taught him. He breaks down the Codecademy trial-model rebuild that took four months and several rounds to deliver a 35% conversion lift, why moving on from a losing experiment too early is one of a PM's costliest mistakes, how to escape the B2B feature factory with metrics that genuinely ladder up, why retention should ride a product's natural use case instead of fighting...
2026-07-07
21 min
The Experimentation Edge
The metric Stitch Fix says every experimenter should chase
SummaryIn this episode of The Experimentation Edge, GrowthBook CMO Ashley Stirrup sits down with Nick Beyler, data science manager at Stitch Fix, where he leads the decision and insights team and owns the company's internal experimentation platform. Nick shares why the metric he most wants is the one he can't measure yet, a North Star that predicts a client's long-term value from their earliest behaviors, and why the most impactful experiment learnings tend to come from adoption friction rather than product bugs. He makes the case that if you're only testing winners you're not taking enough risks...
2026-07-02
20 min
The Experimentation Edge
What the Expedia Group cannot measure, it cannot ship
SummaryAmir Moghaddam, Director of Software Engineering at Expedia Group, joins host Ashley Stirrup on The Experimentation Edge to make the case that measurement is not a reporting step but a gate: what you cannot measure, you cannot ship. Drawing on nearly four years at DoorDash and his current work leading Expedia's air booking platform, Amir explains why he refuses to label experiments winners or losers, how a "failed" pricing test pushed his team toward full personalization, and why a three sided marketplace forces hard trade-offs between competing metrics. The conversation closes on how the same experimentation discipline...
2026-07-01
28 min
The Experimentation Edge
How Fin went from weeks to hours of analysis using AI
SummaryIn this episode of The Experimentation Edge, host Ashley Stirrup sits down with Raunak Kumar, senior manager of GTM analytics at Fin (formerly Intercom), to unpack how experimentation actually works when the data is messy and the traffic is thin. Drawing on nearly 12 years in marketing analytics across Atlassian, Stripe, and Fin, Raunak explains how AI tools like Claude Code have collapsed analysis from weeks to hours and freed his team to clear its experiment backlog, why declining organic search traffic and a 5x jump in untagged ChatGPT referrals are forcing teams to rethink attribution, and how...
2026-06-30
23 min
The Experimentation Edge
Inside The Home Depot's experimentation at a $25B scale
SummaryWhat does experimentation look like inside a $150 billion retailer? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Kim Ting Li, Senior Manager of Experimentation at The Home Depot, where one centralized team tests every major change to a $25 billion online business. Kim explains how 40 people serve 40–50 business teams, why executives join test readouts and ping analysts directly, how every result since 2020 lives in a searchable library, and why scaling beyond hundreds of experiments per year depends on server-side testing capabilities more than AI. For product, data, and engineering leaders building or scaling experimentation pr...
2026-06-29
11 min
The Experimentation Edge
How Disney picks which experiments to run
SummaryWhat does it look like to kill a multimillion dollar feature before anyone builds it? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Crystal Ammari, a digital product optimization and experimentation strategy leader whose career spans Nike and The Walt Disney Company. Crystal shares the "dry test" that used a single fake button to measure demand for video chat (4 million users, 106 clicks), why she reframes experimentation as savings and gains rather than wins and losses, how a misconfigured tool, not bad methodology, made tests take six months, and how a stuck Disney team...
2026-06-29
33 min
The Experimentation Edge
Ship faster, measure better: experimentation tips from JPMorgan Chase
SummaryHow do you know if the thing you just shipped actually worked? On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Kevin Yang, Executive Director and Head of Experimentation at JPMorgan Chase, who has spent six years building experimentation across Chase's digital platforms. Kevin shares how his team turned experimentation into more than a billion dollars of estimated value, why the losing experiments matter more than the winners, and the simple chart exercise he uses to prove that a million-dollar change is invisible without a control group. He and Ashley...
2026-06-25
26 min
The Experimentation Edge
Twitch on why false negatives kill product ideas
SummaryHow do you make a high-stakes product decision when the safe choice is to never test it at all? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arun Bodapati, director of data science at Twitch, about the discipline behind trustworthy experimentation. Drawing on his experience at Schwab, Uber, and Twitch, Arun explains why false negatives are the most dangerous result a team can produce, what hygiene to nail before you push play, and how Twitch used geo-fenced experiments and causal inference to finally settle a pricing question it had avoided for years. It's...
2026-06-24
28 min
The Experimentation Edge
Squarespace killed its blank template and built something better
SummaryWhat do you do when your big launch increases engagement and tanks conversion? On this episode of The Experimentation Edge, host Ashley Stirrup talks with Lina Blackman, Director of Product Analytics at Squarespace, about the blank template launch that flopped — and how its learnings became Blueprint, Squarespace's AI-guided website builder. Lina explains how her embedded analyst team runs 150–200 experiments a year for 3 million customers, the two questions she asks every time a test loses, why teams only need one or two big wins a quarter, how Squarespace calibrates statistical certainty to business stakes, and where AI belongs (and...
2026-06-23
22 min
The Experimentation Edge
Signet Jeweler's "View All" page made more money by showing less
SummaryCraig Kistler, VP of Experience Design, Personalization, and Experimentation at Signet Jewelers (the parent company of Kay, Jared, Zales, Peoples, and Banter), joins host Ashley Stirrup on The Experimentation Edge to unpack how a hybrid online-and-in-store jewelry retailer runs experimentation at scale. Craig shares the counterintuitive "view all" experiment where his team blocked the product grid, added friction on purpose, and grew revenue; why he optimizes for revenue per visitor instead of conversion rate; and how Signet deliberately traded a high-volume testing program for fewer, higher-value experiments. A practical listen for product managers, designers, and experimentation leaders...
2026-06-17
27 min
The Experimentation Edge
RingCentral's DART framework: The four metrics that actually measure AI agents
SummaryRingCentral's Director of Product Management for AI Products, Mayank Agarwal, joins host Ashley Stirrup to dismantle the metrics most teams use to judge AI agents. Drawing on his background founding an AI-first quantitative trading firm and scaling Groupon's bookable marketplace, Mayank explains why accuracy and thumbs-up/down feedback both mislead, and introduces DART — a four-metric behavioral framework (decay, acceptance, relevance, task completion) ported from how he measured trading strategies. He also breaks down a Groupon flash-discount experiment that backfired and the scarcity pivot that fixed it. Essential listening for product managers, engineers, and data scientists building or me...
2026-06-15
32 min
The Experimentation Edge
The 2% close rate increase that turned Ford Credit's product teams into believers
SummaryOn this edition of The Experimentation Edge, Ashley Stirrup talks with Geoffrey Bell, Experimentation Product Specialist at Ford Credit, about building an experimentation practice inside a captive auto lender. Geoffrey shares the losing test that earned his program credibility, the "experimentation piggy bank" he picked up at Microsoft, and the breakthrough of connecting online experiments to offline dealership receivables. The throughline: a program proves its worth not just by the wins it ships, but by the expensive mistakes it prevents and the revenue it can finally trace. It's for product managers, data scientists, and growth leaders who...
2026-06-02
36 min
misc. LM
2026 AI 에이전트와 MLOps 기반 교육 혁신 가이드
2026년 교육계를 뒤흔들고 있는 '에이전트 도약(Agent Leap)' 현상, 즉 단순한 챗봇을 넘어 자율적으로 복잡한 워크플로우를 수행하는 AI 에이전트의 모든 것을 깊이 있게 파헤칩니다. 초개인화된 맞춤형 튜터링부터 교사의 행정 업무 자동화까지, AI가 어떻게 단순한 도구를 넘어 교육자와 학생의 '든든한 팀원(Teammate)'으로 진화하고 있는지 생생한 실제 사례를 통해 살펴봅니다. 최신 기술 트렌드와 현장 도입 가이드부터 윤리적 과제 해결 방안까지 총망라하여, 미래 교육 혁신을 주도하고자 하는 분들에게 실질적이고 강력한 인사이트를 제공합니다🔗 참고 소스 및 출처2026년 AI 에이전트 가이드 - IBM: 원문 바로가기2026년 최대 트렌드, AI 에이전트가 뭔데?: 원문 바로가기2026년에 AI 에이전트를 만들 때 어떤 도구를 사용해야 하는지 (초보자를 위한) 가이드 - Reddit: 원문 바로가기AI Agents and MLOps for Production-Ready AI - Coursera: 원문 바로가기AI Agents in K-12 Education: 10 Use Cases (2026) | Digiqt Blog: 원문 바로가기AI agent trends 2026 report | Google Cloud: 원문 바로가기AIED 2026 · Seoul · Home: 원문 바로가기Agentic AI in Education: Use Cases, 2026 Trends, Playbook: 원문 바로가기Agentic Workflows: Scale AI Prompts Beyond Cursor—No Code Required - Duolingo Blog: 원문 바로가기Building AI Agents – Fundamentals to Advanced | Coursera: 원문 바로가기Coursera launches five new Professional Certificates and expands AI and human skills learning: 원문 바로가기Duolingo's AI Strategy Fuels 51% User Growth and $1B Revenue - Chief AI Officer: 원문 바로가기Education Agent - VoltAgent Use Case: 원문 바로가기Full-Stack AI and Mastering AI Agents Bootcamp - Coursera: 원문 바로가기General Call for Paper · Call for Paper · AIED 2026: 원문 바로가기Generation AI: News & Updates - ISTE: 원문 바로가기How Khan Academy Optimizes AI Tutoring with Experimentation - GrowthBook Blog:
2026-05-25
23 min
The Experimentation Edge
Atlassian on the talent product turnaround from A/B testing
This episode of The Experimentation Edge explores how A/B testing, feature flags, and user research transformed Atlassian's talent product after it failed with its first users. Andrew Willingham — 11 years at Amazon, now Head of Legal and People Products at Atlassian — shares how product experimentation works when you can't test at scale, why your customer and your user are not the same person, and how the metrics you choose decide which experiments you can even run.SummaryAndrew Willingham, Head of Legal and People Products at Atlassian, spent 11 years at Amazon before joining Atlassian a year ago...
2026-05-13
22 min
The Experimentation Edge
How DoorDash saved millions with one A/B test
This episode of The Experimentation Edge unpacks how DoorDash's experimentation platform runs 12,000+ A/B tests per year across 42 million monthly active users — and now powers merchant-led testing on menu pricing and promotions. Ilya Izrailevsky, Senior Engineering Manager leading the platform, explains how feature flags, marketplace experimentation, and CEO-level experiment reviews built a multi-million-dollar experimentation culture across consumers, dashers, and merchants.SummaryMost companies struggle to scale experimentation beyond engineering teams. DoorDash runs over 12,000 experiments per year across 42 million monthly active users — and now they're enabling restaurant owners to run their own A/B tests on menu pric...
2026-05-12
31 min
The Experimentation Edge
How UPS generated half a billion from 80+ apps with A/B testing
This episode of The Experimentation Edge shows how UPS's A/B testing program drove $500M+ in incremental revenue across 80+ customer-facing applications. Dave Massey — head of the J.E.D.I. team (Journey Experience and Design Innovation) — walks through the first test that proved UX could move revenue, how he defended counterintuitive results to skeptical execs, and how a small experimentation team can override opinion with data at enterprise scale.SummaryDave Massey walked into UPS in 2016 and immediately got pulled into a meeting about AB testing tools. By the end of the day, he owned the plat...
2026-05-11
23 min
The Experimentation Edge
How Experimentation Led to Annual Growth at Fanatics
SummaryMost e-commerce companies test a handful of features each month. Fanatics runs nearly 100 experiments monthly and delivers a big portion of the company's total annual growth through experimentation alone. Medha Umarji, VP of Growth and Experimentation at the multi-billion dollar sports merchandising retailer, explains how she built a program that scales from 10 tests per month to 100—and maintains enough rigor to spot false positives before they become costly decisions.The difference isn't tooling or headcount. It's culture. When your CEO reads Excel spreadsheets for fun and actively wants data to prove him wrong, you stop de...
2026-05-07
27 min
The Experimentation Edge
How Experimentation Leads to 8% Annual Growth Every Year at Fanatics
SummaryMost e-commerce companies test a handful of features each month. Fanatics runs nearly 100 experiments monthly and delivers 8% of the company's total annual growth through experimentation alone. Mehta Umarji, VP of Growth and Experimentation at the multi-billion dollar sports merchandising retailer, explains how she built a program that scales from 10 tests per month to 100—and maintains enough rigor to spot false positives before they become costly decisions.The difference isn't tooling or headcount. It's culture. When your CEO reads Excel spreadsheets for fun and actively wants data to prove him wrong, you stop debating whether to te...
2026-05-07
28 min
The Experimentation Edge
How Chewy Quadrupled Experimentation Velocity by Eliminating Its Bottleneck
SummaryWhen Kinjal Shah joined Chewy three years ago, his centralized experimentation team ran about 75 tests per year. Many had sampling bias. The team acted as a bottleneck. Today, Chewy runs over 300 experiments annually with data quality issues driven to "almost zero percent." The shift required fixing infrastructure first, then changing how teams interpret results.Here's the playbook Kinjal and Stefano Barbier, Associate Director of Data Science, used to scale experimentation at a $12.6 billion business serving 21 million active customers.Timestamps02:17 How Chewy's centralized team became a blocker at 75 tests per year
2026-05-05
32 min
EasyVibeCoding Podcast
@aparnadhinak:Agent Harness 中的 Context 管理 每個 Agent harness 都會遇到相同的限制:Context window 對於模型可能需…
Agent Harness 中的 Context 管理 每個 Agent harness 都會遇到相同的限制:Context window 對於模型可能需要記住的所有內容來說實在太小了。隨著對話進行,讀取的檔案變多、Sub-agent 呼叫次數增加,以及工具輸出的堆積,harness 必須決定哪些內容保留在工作集 (working set) 中、哪些需要壓縮,以及哪些留待稍後再檢索。 過去兩年我們一直在開發 Arize 的產品內 Agent「Alyx」,並遭遇過這個問題的各種版本。我們看過對話長到讓模型遺失任務目標、檔案讀取佔用了 Context window 一半的空間,以及工具執行結果擠壓了實際對話內容。 重要的問題不再只是「什麼該放進 Prompt」,而是 harness 如何隨著時間管理 Context。最優秀的系統不會把 Context window 當作被動的對話紀錄緩衝區,而是主動進行管理:將高價值狀態保持在手邊、按需求分頁讀取資料、建立索引以搜尋所需內容(grep 就是這樣做的),並以能暗示還有哪些內容可存取的方式截斷內容。 Pi、OpenClaw、Claude Code 和 Letta 在這裡做了不同的選擇,但它們正趨向於相似的底層模式。Context 不再只是「塞得進對話紀錄的東西」,而是系統必須主動管理的資源。真正的設計問題在於:這種管理有多少是在 harness 內部完成的,又有多少是預期由模型自行處理的。 賭注:信任模型能自行管理 Context 每一個 Context 管理決策都隱含了對模型行為的假設。關鍵問題在於:harness 應該主動限制 Context 的使用,還是依賴模型自行正確管理預算? 檔案讀取讓這個問題變得具體。當模型需要讀取的檔案超過 Context 容量時,必須有人決定保留什麼。這四個 harness 都支援分頁功能的 offset 和 limit 參數。 Pi (pi-mono) Pi 讀取檔案時有 2,000 行或 50KB 的硬性上限(以先達到者為準),即使模型沒有要求分片也是如此。內容會進行頭部截斷,且工具輸出會附加一個明確的接續提示:[Showing lines 1-2000 of 50000. Use offset=2001 to continue.]。工具說明也強化了這一點:「輸出被截斷為 2000 行或 50KB。對於大檔案請使用 offset/limit。」 Pi 的方法是「harness 優先」:harness 先保護你,再教導模型如何分頁。 OpenClaw OpenClaw 繼承了 Pi 的讀取工具及其 2K 行 / 50KB 的截斷規則。在一般檔案讀取時行為相同。除此之外,它還為引導檔案(bootstrap files,即對話開始時載入的一次性 Context 檔案)增加了額外上限:每個檔案 12,000 字元,總計 60,000 字元。當引導檔案超過預算時,它會採用 75% 頭部 / 25% 尾部的分割方式:你會看到開頭和結尾,中間被切掉。 工具結果有獨立的預算:16,000 字元或 Context window 的 30%(取較小者)。當尾部看起來「很重要」(如錯誤訊息、JSON 結束括號、摘要關鍵字)時,它會切換到「頭部+尾部」模式;否則就只保留開頭。 OpenClaw 的方法是「縱深防禦」:Pi 的截斷作為第一層,然後是引導檔案注入的額外上限,最後再加上工具結果的預算。 Claude Code Claude Code 對檔案讀取應用了兩層防禦。第一道關卡是 256KB 的位元組上限,在檔案開啟前透過 stat 呼叫進行檢查——如果檔案超過此限制,讀取會立即被拒絕,並顯示錯誤訊息引導模型使用 offset/limit 或 grep。第二道關卡在讀取後執行:輸出會針對 25,000 token 的預算進行計算,藉此攔截那些位元組數雖小但 token 密度高的檔案。這兩個限制都可以由 Anthropic 透過 GrowthBook 功能旗標 (feature flags) 遠端調整,無需發布新版本。 即使檔案在上限內,工具預設也只會回傳開頭的 2,000 行,且任何超過 2,000 字元的行都會被截斷。模型必須明確使用 offset 和 limit 參數來請求更多內容。 工具說明是一段完整的多段落 Prompt,解釋了分頁機制、提到大小上限、涵蓋圖片/PDF/筆記本支援,並鼓勵跨多個檔案進行平行讀取。offset 和 limit 參數有各自的說明,告訴模型它們是用於過大的檔案。還有一個條件式指令,會根據功能旗標直接在 Prompt 中顯示 256KB 的上限。 檔案去重 (dedup) 系統也值得一提。如果模型在相同的範圍內重新讀取同一個檔案,且檔案時間沒有改變,Claude Code 會回傳一個存根 (stub) 而非完整內容,避免在 Context 中產生重複的 token。 Claude Code 的方法是「具備遠端調整能力的 harness 優先」:包含預讀位元組閘道、讀後 token 閘道、行數與行長預設值、可操作的錯誤訊息、豐富的工具 Prompt、讀取去重,以及讓 Anthropic 能在伺服器端調整所有設定的功能旗標。 Letta Letta 採取了截然不同的方法。每個上傳的檔案都會被解析、分塊並嵌入 (embedded) 到向量資料庫中,因此 Agent 同時擁有精確搜尋和語意搜尋能力。這賦予它三種檔案工具:openfiles 用於直接檢視(讀取原始文字)、grepfiles 用於精確模式匹配(也是原始文字),以及 semanticsearchfiles 用於針對嵌入片段進行基於意義的檢索。 當檔案在 Agent 的 Context 中「開啟」時,其可見內容會被截斷為每個檔案的字元限制,該限制會隨模型 Context window 的五個等級而擴展:8K Context 為 5,000 字元、32K 為 15,000、128K 為 25,000,200K+ 為 40,000。同時開啟的檔案數量也會隨之擴展,小型模型為 3 個,大型模型最多 15 個,預設為 5 個。當超過限制時,會採用 LRU (最近最少使用) 策略來移除檔案。 Letta 的方法是「記憶優先」:檔案同時存在於原始文字和向量資料庫(嵌入區塊)中,Context window 只顯示受管理的視圖,模型則使用工具來存取更多內容。 真正的工程挑戰:對話修剪 (Session pruning) 隨著對話增長,每個 harness 都必須決定保留什麼、捨棄什麼。這是設計差異最顯著的地方,因為壓縮策略決定了長期運行的 Agent 是能保持連貫性,還是會逐漸退化。 Pi (pi-mono) Pi 使用壓縮:由 LLM 驅動的摘要功能,由 token 閾值觸發。 觸發條件:預估的 Context token 超過 contextWindow - reserveTokens(預設保留:16,384 tokens)。 保留內容:從對話末端往回走,保留最近約 20,000 個 token 的訊息 (keepRecentTokens)。 摘要內容:更舊的內容會傳送給 LLM 進行摘要。 摘要存放位置:成為一則合成的使用者訊息,置於保留的尾部之前。 工具呼叫安全性:絕不會在孤立的工具結果處截斷。會遍歷邊界以確保「工具呼叫/工具結果」對保持完整。 OpenClaw OpenClaw 在 Pi 的壓縮機制之上運行了兩種不同的 Context 管理機制: 觸發條件:歷史紀錄超過 Context window 的 50% (maxHistoryShare,預設 0.5)。 保留內容:將歷史紀錄分割成等量的 token 區塊;丟棄最舊的區塊,其餘部分保留,並修復工具呼叫/結果對。 摘要內容:被丟棄的內容會經過分階段的多輪 LLM 摘要,並包含合併步驟。 摘要存放位置:與 Pi 相同——合成訊息置於保留的尾部之前。 工具呼叫安全性:repairToolUseResultPairing 可修復區塊丟棄後任何孤立的工具結果;splitMessagesByTokenShare 避免在工具呼叫/結果對內部進行切割。 壓縮前刷新:一個靜默的 Agentic 轉向 (turn) 讓 Agent 在歷史紀錄消失前將狀態持久化到記憶檔案中。 第二層:工具結果的非破壞性記憶體內修剪(軟修剪,然後硬清除),快取 TTL 為 5 分鐘,在保護持久化對話的同時為當前請求回收 Context。 Claude Code Claude Code 透過查詢前最佳化 (pre-query optimization) 和 LLM 驅動的壓縮來管理 Context。 觸發條件:預估 token 超過有效 Context window 減去 13,000 token 的緩衝區(對於 200K Context 的模型,壓縮會在約 167K token 時觸發)。 摘要內容:將完整對話傳送給模型,並附帶一個結構化的 9 節 Prompt,涵蓋主要請求、關鍵技術概念、檔案與程式碼、錯誤與修復、問題解決、所有使用者訊息、待辦任務、當前工作以及選用的下一步。 摘要存放位置:成為一則使用者訊息,告知模型該對話是從先前因 C…
2026-04-26
07 min
The Experimentation Edge
Inside Chess.com's Plan to Run 1,000 Experiments in a Single Year
SummaryChess.com ran its first A/B test in 2023. Two years later, the team is on track to run 1,000 experiments in a single year—and they've already shipped 195 in Q1.In this episode, Ashley Stirrup sits down with Nafis Shaikh, Director of Product Management at Chess.com, to get inside the experimentation engine powering one of the world's most beloved gaming products.Nafis brings experience from Zynga and Prodigy and a refreshingly honest take on what changes when a product built on passion suddenly has to serve a 10-million-DAU user base that spans ab...
2026-04-21
31 min
The Experimentation Edge
Ancestry on AI Storytelling, Paywalls, and Decision Quality
SummaryWhat happens when A/B testing stops being a tool and becomes your operating system? Suresh Teckchandani, VP of Product & Technology at Ancestry (formerly PayPal and eBay), shares how the team scaled experimentation from isolated tests to capability-building that drives roadmap and revenue. He details the “growth metering” and paywall experiments that unlocked a 5.3% lift in key engagement and improved conversions—then became platform features. Suresh explains Ancestry’s centralized experimentation platform with self-serve access for PMs and engineers, why “obvious” UX changes can be the riskiest, and how removing friction actually hurt engagement by 20–25% due to user mental mo...
2026-04-14
25 min
The Experimentation Edge
Fyxer's engineering playbook to go from $1M to $35M ARR
SummaryHow do you drive hypergrowth without guessing? Kameron Tanseli, Head of Growth Engineering at Fyxer—an AI assistant for your email—breaks down the experimentation playbook that helped the company scale from $1M to $35M ARR, with sights set on $100–$150M. Kameron explains how startups should think about A/B testing differently: de-risk big bets, not just button colors. He shares a risk-based approach to when to run rigorous tests vs. ship-and-measure, why a 25% win rate is a sign you’re testing ambitiously, and how PLG features should be shipped first, then rapidly iterated to drive usage. You’ll h...
2026-04-02
30 min
The Experimentation Edge
The 5 pixels that cost LinkedIn a million dollars a month
SummaryHow do you build a culture where nothing ships without evidence—and leaders actually act on the data? Makram Mansour, Head of Marketplace at ID.me and former experimentation leader at LinkedIn and Intuit, shares the systems, mindsets, and guardrails behind “experimenting everywhere.” At LinkedIn, he helped support 10,000+ annual experiments with 2,000 weekly platform users, and he explains the hard-earned lessons (like a 5px UI tweak causing a million-dollar ad loss) that led to a “test before release” mandate. At Intuit, he operationalized “fail forward,” partnering with HR to rewrite OKRs so teams are rewarded for learning, not just launching...
2026-04-02
37 min
The Experimentation Edge
Truist on shipping faster and more safely with AI and human-in-the-loop banking
How do you boost developer velocity in a highly regulated industry—without sacrificing safety or customer trust? Charles Williams, Senior Vice President and Software Engineering Director at Truist (formed from the BB&T and SunTrust merger), shares how his team elevates developer experience to ship faster and more reliably. Charles breaks down shifting quality “left” with automation, measuring success with both DORA metrics and developer sentiment, and why human-in-the-loop is non-negotiable for AI in finance. He details Truist’s governance model—steering committees, enterprise architecture, and clear guardrails—to avoid tool sprawl while building a purpose-built AI ecosystem: Microsoft Copilot for...
2026-04-02
27 min
The Experimentation Edge
Shipping Faster, Safely: Truist’s SVP on AI, Developer Experience, and Human-in-the-Loop Banking
SummaryHow do you boost developer velocity in a highly regulated industry—without sacrificing safety or customer trust? Charles Williams, Senior Vice President and Software Engineering Director at Truist (formed from the BB&T and SunTrust merger), shares how his team elevates developer experience to ship faster and more reliably. Charles breaks down shifting quality “left” with automation, measuring success with both DORA metrics and developer sentiment, and why human-in-the-loop is non-negotiable for AI in finance. He details Truist’s governance model—steering committees, enterprise architecture, and clear guardrails—to avoid tool sprawl...
2026-03-18
27 min
The Experimentation Edge
From chatbots to open-world agents at Microsoft: evals, go-live metrics, and copilot velocity
AI is moving so fast that what “good” looked like a few months ago is already outdated.So how do you measure value, ship safely, and scale what works? Marco Casalaina, VP of Products, Core AI and AI Futurist at Microsoft, joins to unpack how his team builds and evaluates next‑gen AI—at hyperspeed.Marco leads the AI Futures team and previously led Azure OpenAI, Azure Cognitive Services, Responsible AI, and AI Studio; before Microsoft, he ran Salesforce Einstein.He explains why enterprise value is best measured by go‑lives and real usage, how...
2026-03-17
28 min
The Experimentation Edge
Upwork on AI-Driven Ops at scale
SummaryWhat do you test rigorously—and what do you ship fast and fix forward—when every change could impact millions?Vinoj Kumar, Vice President of Engineering at Upwork, leads at the intersection of infrastructure and product, where feedback loops are longer and the blast radius is wider.He shares a pragmatic framework for experimentation—blast radius x reversibility—that sets testing rigor, plus how he measures success in product terms: faster search, resilient marketplace trust, and developer velocity. Vinoj explains why “high engagement” can mask low-quality experiences, how his team instrumented an internal NL chatbot w...
2026-03-11
30 min
The Experimentation Edge
How Moxie Pest Control boosted conversions 5% with data and coaching
Think pest control isn’t a digital business? Think again.Raj Mehta, Vice President of Product and Technology at Moxie Pest Control, outlines how he turned a spreadsheet-run operation into a data-driven engine across 9,000+ daily calls.Raj shares how consolidating fragmented systems into a data lake unlocked automation—from shrinking lead routing from 20–25 minutes to under 30 seconds—to deploying AI-powered call intelligence that scores every sales and retention conversation against a playbook. He breaks down a practical roadmap for traditional businesses: build MVPs, pilot in one branch with a trained feedback team, iterate fast, then scale.
2026-03-05
17 min
The Initial
How Fyxer.ai ran 512 Experiments in a year by using AI
we unpack one of the fastest built experimentation engines i've seen!! In this episode, I sit down with Kameron Tanseli (Growth Engineering) and Rosie Hoggmascall (Growth & Product) from Fyxer.ai and we retrace the steps of building a 500+ /year experimentation program. We go deep into:Why their first onboarding experiments actually made things worseThe tooling shift from PostHog to GrowthBook (and why it mattered)How decentralized ownership (no PM, no heavy process) unlocked serious speedWhy every engineer acts like a founderHow Slack transparency builds experimentation cultureUsing AI agents to automate experiment reporting, analysis...
2026-03-03
42 min
The Experimentation Edge
Typeform on how to stop running experiments and start earning them
If 80% of A/B tests fail, how do you de-risk decisions that touch pricing, product, and brand? Aleksandra (Aleks) Bass, Chief Product & Technology Officer at Typeform, shares how her team “earns the right to A/B test” with medical-grade rigor—moving from literature reviews and user tests to simulated trials before exposing changes to customers. She details Typeform’s repositioning from “forms” to an AI engagement platform—and the pricing and packaging bet behind it: a 15% drop in new business count offset by a 32% increase in ASP and a 25% lift in annual attach.Aleks unpacks how Typeform AI acts as a c...
2026-02-24
31 min
Prayerson's Podcast - What to Build | Why It Matters
the ai toolkit every product manager needs in 2026
Listen now:Spotify // Applein this conversation, you’ll learn:* how to build an ai stack that behaves like a small product team, not a folder of tools.* why ai copilots, research agents, qa systems and orchestration layers are now core pm infrastructure.* the eight pillars that matter for pm leverage in 2026.* how orchestration ties everything together into a system that thinks and works with you.where to find prayerson:* x: https://x.com/iamprayerson* linkedin: https://www.linkedin.com/in/pr...
2025-12-08
12 min
Попы и культура
Нейросети feat. Тимур Сейфельмлюков
Неожиданный спешал! Позвали в гости нашего товарища Тимура из подкаста «Завтракаст» и устроили большой разговор про нейросети. Не только про генерацию аниме-картинок, но и про реальные рабочие кейсы: от Big Data и исследований, до автоматизации быта и генерации тех самых реалистичных видео, где моряки чистят китов от ракушекОчень важный момент: это не подкаст-инструкция по использованию AI, а скорее обсуждения темы нейросетей с точечными советами и рассуждениями ведущихВ выпуске все разбито по таймкодам, поэтому не забывайте ими пользоватьсяТелеграм-канал Тимура: https://t.me/radiotimurИнструменты, которые обсуждали в выпуске:- Для работы с текстом: ChatGPT, Claude, Deepseek, Gemini и Grok- Для исследований и Data Science: ChatGPT и GrowthBook- Для работы с изображениями: Midjourney, Flux, ChatGPT 4o, Grok, Recraft, Playground AI, SDXL и Le Chat- Для работы с видео: Sora, Runway, Icon, Luma, Kling, а так же Hunyuan, WAN локально- Для кодинга: Gemini, Cursor (+ связка с Claude) и Copilot- Для генерации голоса и музыки: Suno, ElevenLabs, Riffusion и Udio- Инструмент для контроля локальных нейросетей генерации видео, аудио, изображений: ComfyUIБусти: https://boosty.to/popsandcultureНаш телеграм-канал: https://t.me/popscultureЮтуб: https://www.youtube.com/@popsandcultureОтправить донаты: https://donatty.com/popsandcultureНаш дискорд: https://discord.gg/3GaTuq3qnV
2025-04-09
2h 30
Build vs. Buy
Graham McNicoll, Co-Founder & CEO at Growthbook
analysis in product development, and how GrowthBook addresses these needs. The conversation also highlights the role of open source in GrowthBook's growth and their innovative product WebLens.ai, which leverages AI for A/B testing suggestions.00:00 Introductions01:06 Guest Introduction: Graham McNichol01:33 The Genesis of GrowthBook03:45 Challenges in Building AB Testing Platforms05:10 GrowthBook's Open Source Journey15:36 The Impact of Open Source on Business17:15 Open Source Tools and Paid Features18:33 Community Contributions and SDKs20:00 Build vs Buy: The Decision Process22:41 Targeting...
2024-12-16
37 min
Recordings: LinuxLounge
LL273 Der Einhorn-Flüschterer
Korrektur letzte Sendung "Korrektur: LL272: Conduit unterstützt es nicht, die History zu bearbeiten, sondern die edit history von einzelnen Nachrichten an Clients auszuliefern, in dem sie den /relations-Endpunkt unterstützen." Quelle Neues aus dem Repo Growthbook v2.4.0 erleichtert den Einstieg Lemmy-Automoderator Audiobookshelf whishper 1.1.0 - Transkriptions-Plattform Kavita ist eine OSS Comic/Manga Verwaltung Mission Center verbreitet Taskmanager-Vibes | Webseite des Projekts QEMU 8.1 Highlights Linux From Scratch 12 DXVK 2.3 veröffentlich Ghostfolio soll das Verwalten des eigenen Vermögens erleichtern Beitrag Chaos Communication Camp 2023 - Jonas Newsflash OpenTF kündigt Fork von Terraform Thund...
2023-09-10
1h 19
Recordings: LinuxLounge
LL273 Der Einhorn-Flüschterer
Korrektur letzte Sendung “Korrektur: LL272: Conduit unterstützt es nicht, die History zu bearbeiten, sondern die edit history von einzelnen Nachrichten an Clients auszuliefern, in dem sie den /relations-Endpunkt unterstützen.” Quelle Neues aus dem Repo Growthbook v2.4.0 erleichtert den Einstieg Lemmy-Automoderator Audiobookshelf whishper 1.1.0 - Transkriptions-Plattform Kavita ist eine OSS Comic/Manga Verwaltung Mission Center verbreitet Taskmanager-Vibes | Webseite des Projekts QEMU 8.1 Highlights Linux From Scratch 12 DXVK 2.3 veröffentlich Ghostfolio soll das Verwalten des eigenen Vermögens erleichtern Beitrag Chaos Communication Camp 2023 - Jonas Newsflash OpenTF kündigt Fork von Terraform Thunderbi...
2023-09-10
1h 19
Recordings: LinuxLounge
LL273 Der Einhorn-Flüschterer
Korrektur letzte Sendung "Korrektur: LL272: Conduit unterstützt es nicht, die History zu bearbeiten, sondern die edit history von einzelnen Nachrichten an Clients auszuliefern, in dem sie den /relations-Endpunkt unterstützen." Quelle Neues aus dem Repo Growthbook v2.4.0 erleichtert den Einstieg Lemmy-Automoderator Audiobookshelf whishper 1.1.0 - Transkriptions-Plattform Kavita ist eine OSS Comic/Manga Verwaltung Mission Center verbreitet Taskmanager-Vibes | Webseite des Projekts QEMU 8.1 Highlights Linux From Scratch 12 DXVK 2.3 veröffentlich Ghostfolio soll das Verwalten des eigenen Vermögens erleichtern Beitrag Chaos Communication Camp 2023 - Jonas Newsflash OpenTF kündigt Fork von Terraform Thund...
2023-09-10
1h 19
TheRadio.CC: LinuxLounge
LL273 Der Einhorn-Flüschterer
Korrektur letzte Sendung "Korrektur: LL272: Conduit unterstützt es nicht, die History zu bearbeiten, sondern die edit history von einzelnen Nachrichten an Clients auszuliefern, in dem sie den /relations-Endpunkt unterstützen." Quelle Neues aus dem Repo Growthbook v2.4.0 erleichtert den Einstieg Lemmy-Automoderator Audiobookshelf whishper 1.1.0 - Transkriptions-Plattform Kavita ist eine OSS Comic/Manga Verwaltung Mission Center verbreitet Taskmanager-Vibes | Webseite des Projekts QEMU 8.1 Highlights Linux From Scratch 12 DXVK 2.3 veröffentlich Ghostfolio soll das Verwalten des eigenen Vermögens erleichtern Beitrag Chaos Communication Camp 2023 - Jonas Newsflash OpenTF kündigt Fork von Terraform Thund...
2023-09-10
1h 19
TheRadio.CC: LinuxLounge
LL270 TIL: Toystory-Namen für Debian
Neues aus dem Repo Debian 12 | Pläne: Debian 13 IceWM 3.4 scanservjs 2.27.0 - Webinterface für SANE-Scanner Intels Open Image Denoise v2.0.0 veröffentlicht NextCloud Tables für Android 1.0.3 Orb 0.21 stable Growthbook 2.2.0 Podman Desktop 1.1.0 | Blog: Lokales Kubernetes Cluster mit Podman und Kind Organic Maps 2023.06.04-13 Coreboot 4.20.1 Update unterstützt weitere Mainboards Newsflash PineTab 2 wird ausgeliefert FFmpeg fügt Unterstützung für animierte JPEG-XL hinzu Google's Patches verbesser Performance bei Over-Committed Hypervisor PuzzleFS als neues Container-Dateisystem für den Kernel czkawka 6.0.0 Verbesserungen am Virt-IO-GPU Treiber für Zink JDK 21 Feature freeze Nitrokey Bericht: Vorwurf an Qualcomm persönliche Daten zu le...
2023-06-11
1h 23
Recordings: LinuxLounge
LL270 TIL: Toystory-Namen für Debian
Neues aus dem Repo Debian 12 | Pläne: Debian 13 IceWM 3.4 scanservjs 2.27.0 - Webinterface für SANE-Scanner Intels Open Image Denoise v2.0.0 veröffentlicht NextCloud Tables für Android 1.0.3 Orb 0.21 stable Growthbook 2.2.0 Podman Desktop 1.1.0 | Blog: Lokales Kubernetes Cluster mit Podman und Kind Organic Maps 2023.06.04-13 Coreboot 4.20.1 Update unterstützt weitere Mainboards Newsflash PineTab 2 wird ausgeliefert FFmpeg fügt Unterstützung für animierte JPEG-XL hinzu Google's Patches verbesser Performance bei Over-Committed Hypervisor PuzzleFS als neues Container-Dateisystem für den Kernel czkawka 6.0.0 Verbesserungen am Virt-IO-GPU Treiber für Zink JDK 21 Feature freeze Nitrokey Bericht: Vorwurf an Qualcomm persönliche Daten zu le...
2023-06-11
1h 23
Recordings: LinuxLounge
LL270 TIL: Toystory-Namen für Debian
Neues aus dem Repo Debian 12 | Pläne: Debian 13 IceWM 3.4 scanservjs 2.27.0 - Webinterface für SANE-Scanner Intels Open Image Denoise v2.0.0 veröffentlicht NextCloud Tables für Android 1.0.3 Orb 0.21 stable Growthbook 2.2.0 Podman Desktop 1.1.0 | Blog: Lokales Kubernetes Cluster mit Podman und Kind Organic Maps 2023.06.04-13 Coreboot 4.20.1 Update unterstützt weitere Mainboards Newsflash PineTab 2 wird ausgeliefert FFmpeg fügt Unterstützung für animierte JPEG-XL hinzu Google's Patches verbesser Performance bei Over-Committed Hypervisor PuzzleFS als neues Container-Dateisystem für den Kernel czkawka 6.0.0 Verbesserungen am Virt-IO-GPU Treiber für Zink JDK 21 Feature freeze Nitrokey Bericht: Vorwurf an Qualcomm persönliche Daten zu le...
2023-06-11
1h 23
Open Source Startup Podcast
E66: Open Source Feature Flagging & Experimentation with GrowthBook
Graham McNicoll & Jeremy Dorn are Co-founders of GrowthBook, the open source feature flagging and experimentation platform out of YC. Their feature flagging and A/B testing project, also called growthbook, has 4K stars on GitHub. In this episode, we discuss the importance of having an iterative culture, having a dynamic view on positioning, and the importance of content and market education.
2022-12-05
34 min
The Business of Boutique Fitness
{Best of} Retreats With Real Revenue: Building an Amazing Adventure That Wows Your Students and Your Bank Account. Pt. 2
In this episode you'll learn: • A recap on identifying a retreat versus an event • What is the marketing cycle if planning an event {and why you should give your clients the opportunity to purchase the next one} • What is an affiliate and the importance of having a strong affiliate network • How to provide a content calendar to you affiliates • Strategies for tracking your sales • What is the marketing cycle if planning a retreat {and some tips and tricks when using video in it} • And more! LINKS:
2019-01-29
35 min
The Business of Boutique Fitness
{Best of} Retreats With Real Revenue: Building an Amazing Adventure That Wows Your Students and Your Bank Account
In this episode you'll learn: • How to set yourself up for profits when planning a retreat or an event • Tips to help you in the process of choosing the right venue • How to handle lodging {and what are the various options available} • What to consider in terms of food, beverage and other additional services {and how to negotiate prices for them} • The two biggest numbers you need to calculate for a successful retreat • What is the difference between an event and a retreat • How to calculate prices and add profit...
2019-01-22
36 min
The Business of Boutique Fitness
{Best of}: Meet Jodh Dyson and Savitree Doppelt, the Co-Founders of Urban Yoga Chicago and Learn How They've Transformed Their Studio {and Teacher Trainings} Into a Profitable Powerhouse That's Always Serving Clients
Things you will learn in this episode: • How Jodh and Savitree met and built Urban Yoga Chicago in the most organic of ways • Why they introduced kundalini yoga to the public • The strategy they used in starting their teacher training without having a brick and mortar • What challenges they went through that ultimately made them feel like they had to start over • There are no rules when building your business • Culture and community start from the top down • What is intentional intake and how it changed Urban...
2019-01-17
54 min
The Business of Boutique Fitness
{Best of} The Four Stages Every Studio Must Pass Through: The Freedom Stage, or Stepping Back to Allow the Empire You've Built to Grow {yes, even when you're not there}
In this episode you'll learn: • The biggest warning I have for you in this stage • Stick to what your clients truly need and want in this stage • Why you need a day-to-day Operations Manager • What activity you can do here that will grow your clientele quickly • The downfall that can come with mass diversification • Set your business up for sale - even though it's not for sale! Get the Growth Book now! http://www.lisekuecker.com/growthbook Sign up for the video series mentioned at http://www.th...
2019-01-15
31 min
The Business of Boutique Fitness
{Best of} The Four Stages Every Studio Must Pass Through: The Grow Stage, or Leveraging Team and Offerings to Maximize Your Profits
Things you will earn in this episode: • What your role needs to be in this stage and why you have to build just beyond yourself • What I did to shift my focus in this stage to becoming a CEO • How to investigate why things go well and why things don't {and then build a system around it to propel you forward} • What I looked for first when hiring someone • How to build the most powerful type of marketing • The differences between a sales manager, marketing and operations manager, and...
2019-01-08
37 min
The Business of Boutique Fitness
{Best of} The Four Stages Every Studio Must Pass Through: The Build Stage, or Creating Consistent Clients, Profits, and Revenue You Can Rely On, All Year Long
In this episode you'll learn: • Why you need to define the difference between urgent and important when managing your tasks in this phase • Why it is important to categorize your tasks using a matrix and identify areas that can be delegated • The top two major parts of your business that people tend to leave off the matrix completely • You need to build in consistent reoccurring revenue – with 90% of the products you sell doing this • What are sales benchmarks and what you should be reaching for • Content is king. What typ...
2019-01-01
41 min
Growth Now Movement with Justin Schenck
Scale or Fail with Allison Maslan - Ep. 149
Allison Maslan is the CEO of Allison Maslan International, a Global Business Mentoring Company. She was recently named "One of the Top Women Entrepreneurs Who Inspire" by Self Made Magazine. No. 1 Best Selling Author of the book, Blast Off! The Surefire Success Plan to Launch Your Dreams Into Reality, Allison has built ten successful companies from the ground up starting as a single mom on her own. Her client list has included Ben & Jerry's, Charlotte Russe, Supercuts and Allstate. Allison has guided countless business owners to grow high net-worth companies, multiply their income and create more passion and freedom on a daily...
2018-10-30
37 min
The Business of Boutique Fitness
Meet Tali and Tammar, the founders of Off Road DC and Learn How They've Built a Foundation of Fun
In this episode you'll learn: How Tammar and Tali met and set out to create "The Cheers of Fitness Studios" {while both holding full-time jobs} The factors they believe have contributed to their amazing success What their customers experience when walking through their doors and how they are able to continually infuse creative fun into the business The biggest lessons they learned from starting the business and then expanding to a second location Why they felt the need to change to a membership model What approach they take i...
2018-10-04
52 min
The Business of Boutique Fitness
Meet Studio Grow, Our New Brand, and See Just Why It Was Time To Make This Shift
In this episode you'll learn: What led me to starting a small consulting business Why we decided that my business needed a shift How our business intentions resulted in the naming of our business Why it took a year to launch new branding and website How I'm really look forwarding to meeting my staff in the upcoming weeks! LINKS: http://www.lisekuecker.com/growthbook
2018-10-02
16 min
غربتلي
غربتلي #2 مع آندرو عزمي... من صعيد مصر إلى وادي السيليكون
الحلقة الثانية من بودكاست غربتلي مع آندرو عزمي، مُتخصّص في استسراع النمو والتسويق الإلكتروني حساب آندرو على تويتر twitter.com/AndrewAzmi موقع آندرو عزمي www.andrewazmi.com/blog/ قناة آندرو على يوتيوب www.youtube.com/channel/UCdJIBY9raORdIb0ieEoTuJA كتاب للتسويق الإلكتروني من آندرو www.andrewazmi.com/blog/growthbook/ سناب شات آندرو www.snapchat.com/add/andrewazmi
2018-09-28
40 min
The Business of Boutique Fitness
Meet Brooke Bowersock, the Founder of Align Studios} and See How She's Bridging the Gap Between Pilates and Other Movement Modalities
In this episode you'll learn: How Brooke built her very first studio Why she supports her trainers on their path to find their own unique set of skills The reason Brooke made the decision to sell her thriving business and how she planned that exit strategy What made her decide to branch out from just Pilates The benefits of hiring instructors that will build community What Brooke says is the hardest part of building a business LINKS: http://www.lisekuecker.com/growthbook http://www.alignaustin.co...
2018-09-27
34 min
The Business of Boutique Fitness
Niches build Riches: 3 ways to use niches to dominate your market
In this episode you'll learn: What narrowing down to a very specific market will do for your business Why many of us often resist doing this Some trends and niches we notice that are building extremely successful businesses Ways to market to your niche and the power of Google AdWords LINKS: http://www.lisekuecker.com/growthbook http://www.lisekuecker.com/uncategorized/episode160/
2018-09-25
25 min
The Business of Boutique Fitness
Meet Jacob Andreae and Learn How He's Built A Business Around 7 Core Clients {and 6 Core Movements}
In this episode you'll learn: How Jacob got started on his amazing journey in fitness and business and his transition out of a teaching career to get there The way he incorporates goal-based challenges to get results like no one else for his clients What he's realized about his clients' motivation and how he's seen his clientele change over time The significant role of practice and repetition in building systems and a successful business Why you need to not only love what you do, but look at how you're doing it What to focus on if...
2018-09-20
1h 00
The Business of Boutique Fitness
The Four Stages Every Studio Must Pass Through: The Freedom Stage, or Stepping Back to Allow the Empire You've Built to Grow {yes, even when you're not there}
In this episode you'll learn: The biggest warning I have for you in this stage Stick to what your clients truly need and want in this stage Why you need a day-to-day Operations Manager What activity you can do here that will grow your clientele quickly The downfall that can come with mass diversification Set your business up for sale - even though it's not for sale! Get the Growth Book now! http://www.lisekuecker.com/growthbook Sign up for the video series mentioned at http://www.th...
2018-04-10
32 min
The Business of Boutique Fitness
Meet Jodh Dyson and Savitree Doppelt, the Co-Founders of Urban Yoga Chicago and Learn How They've Transformed Their Studio {and Teacher Trainings} Into a Profitable Powerhouse That's Always Serving Clients
Things you will learn in this episode: How Jodh and Savitree met and built Urban Yoga Chicago in the most organic of ways Why they introduced kundalini yoga to the public The strategy they used in starting their teacher training without having a brick and mortar What challenges they went through that ultimately made them feel like they had to start over There are no rules when building your business Culture and community start from the top down What is intentional intake and how it changed Urban Yoga Chicago Get the...
2018-04-05
55 min
The Business of Boutique Fitness
The Four Stages Every Studio Must Pass Through: The Grow Stage, or Leveraging Team and Offerings to Maximize Your Profits
In this episode you'll learn: What your role needs to be in this stage and why you have to build beyond yourself What I did to shift my focus in this stage to becoming a CEO How investigating why things go well and why things don't, then building a system around it, will propel you forward Why I hired for attitude That way to build the most powerful marketing out there How I filled time blocks, services and opportunities outside of what I can do The d...
2018-04-03
37 min
The Business of Boutique Fitness
Meet Freda Nunn, a Pilates Studio Owner, who's Built a Full-Time Income Source as She Prepares for Retirement from the Federal Government
In this episode you'll learn: How Freda went from working for the federal government to owing and running her very own Pilates studio The steady path to growth she has taken How Freda stays focused and manages to balance both a full-time job and a full-time studio When your purpose is rooted in other people, that is your driving force… The struggles Freda had with increasing her prices and how she was able to overcome this How she advertises her studio and what has worked best for her Get the...
2018-03-29
40 min
The Business of Boutique Fitness
The Four Stages Every Studio Must Pass Through: The Build Stage, or Creating Consistent Clients, Profits, and Revenue You Can Rely On, All Year Long
In this episode you'll learn: In this episode you'll learn: • Why you need to define the difference between urgent and important when managing your tasks in this phase • Why it is important to categorize your tasks using a matrix and identify areas that can be delegated • The top two major parts of your business that people tend to leave off the matrix completely • You need to build in consistent reoccurring revenue – with 90% of the products you sell doing this • What are sales benchmarks and what you should...
2018-03-27
41 min