Stefan Pernek & I just recently finished a podcast with Nils Stotz. It was our first episode for The Hotfix Podcast and I would not have thought that I would be learning so much.
Framework for Types of Failures: Basic, Complex, and Intelligent
When being asked about his failure story, Nils didn't jump right into a story but shared a framework that shows that failure isn't one dimensional. There's different types of failures. One is unavoidable, one is fatal and one is even desired:
* Basic Failures: The “oops” moments—avoidable errors due to oversight or missed details. Nils emphasized using checklists and rituals (inspired by fields like medicine) to keep everyone aligned and minimize these small but costly mistakes.
* Complex Failures: These are harder to foresee and often arise from cross-functional misalignment. Regular retros and open communication can help teams recognize and address misalignments before they become major issues.
* Intelligent Failures: The valuable failures. These are planned experiments meant to teach us something about our users, products, or markets. Nils shared how these “intelligent failures” have saved resources by identifying what doesn’t work early.
Pro Tip: This is a great framework in case you're being asked "Tell me about a time when you failed in a job interview" 💡.
Experimentation as a Product Mindset
Nils generally talks a lot about experimentation. I loved his definition of experimentation:
Experimentation is any structured approach to gathering evidence for decision-making.
Experimentation is not only about running A/B tests, but in his view, experimentation is about systematically collecting information to support decisions rather than solely relying on intuition or observational data.
Itamar Gilad's Confidence Meter
The Confidence Meter (borrowed from Itamar Gilad) is a great tool to evaluate different methods of experimentation against each other..
Knowing When to Stop Experimenting
Stefan Pernek mentioned a time where he and his team weren't 100% sure when to stop an experiment.
For this Nils shared an incredibly helpful framework. He suggests to classify all experiments either under "optimization" or "innovation". Whenever we experiment only in the favour of optimising our product we should also stop it faster. Whenever something is in favour of innovating, the risk of failure is higher, when also the potential return could be higher (if successful). So whenever we experiment in favour of innovation it's ok to keep experiments running for longer.
A known framework for balancing these two types is also Small bets vs. Big Swings by Jim Collins:
* Small Bets: These are low-risk, incremental experiments or optimizations aimed at refining existing features or processes. They usually involve smaller investments of time and resources but can yield reliable, if modest, improvements. Small bets are often used to test assumptions, validate ideas, or optimize minor elements in the product, like design tweaks or UX flows. They’re practical and help teams make continuous progress without jeopardizing stability.
* Big Swings: In contrast, big swings are high-risk, high-reward initiatives that focus on transformative changes or bold innovations. These are substantial commitments that could lead to significant breakthroughs or new product directions. Big swings might involve launching a new product line, entering a new market, or radically rethinking a core feature. While they carry higher risk, the potential payoff is also much greater, offering the chance to redefine value for the customer or even reshape the market.S
Stefan Pernek created a simple, but powerful visualisation for this in the past:
When running experimentation work it's important to know for which of these two types of features we're currently experimenting for.
Actionable Tips on Communicating with Leadership
As a last item we talked about a common PM dilemma:
A leader asks us to build something. We don’t get to development right away, though.We want to do what’s right and do some additional validation. Not getting to work right away, can come across as being-hard-to-work-with or, even worse, as unreliable. At the same time it's our job as PMs to avoid building something no one wants or needs.
Nils’s advice is to always anchor the conversation in strategic priorities. Rather than pushing back emotionally. How to get your message as a PM across matters. One very practical tip is simply announcing what you're going to do before you do it. E.g. saying "I need to push back here a little...". This clearly states your intention.
Stefan pointed out that leaders are often under pressure to hit their numbers—whether that’s user acquisition, retention, revenue, or other key metrics. Because of this, they may sometimes push for quick wins or immediate execution on certain initiatives to meet short-term goals. Stefan noted that if a product manager can demonstrate how their proposed approach (like taking time for validation or experimentation) could help achieve those same targets more effectively, leaders are likely to listen.
**📺 📻 Listen or watch the full episode here: https://pal.bio/the-hotfix-podcast.**
Subscribe to miss future episodes. We still have epic ones coming up! 👊
