Justin Reock is Deputy CTO at DX, the developer intelligence platform built on the research behind DORA, SPACE, and the DevEx framework. An engineer, speaker, and writer with more than 25 years in software, Justin spends much of his time with engineering leaders worldwide, helping them measure developer experience, turn that data into continuous improvement action, and figure out what their AI spend is actually buying.
Episode Description
Everyone wants to know whether AI is paying off. Justin Reock has more data on that question than almost anyone and claims ROI may not be here yet. Token spend is up roughly 50x year over year, but velocity is climbing in a straight line rather than a curve, and CFOs have started showing up in engineering meetings. He argues that’s a strong signal the current hype cycle is about to get a reality check.
Nerd/Noir has been an early adopter and partner of DX, so Justin and Dave start with the origin story: how DX grew from a PhD-grade perceptual survey into the Core 4, why qualitative signals from engineers turn out to be more reliable than system metrics in almost every divergence, and why you should never wait for your data to be “ready” before you start measuring. From there, the conversation turns to the hard part: change management. Justin lays out the two beliefs an organization has to adopt before any of this data does anything: developer experience is a systems problem, not a people problem, and it is the leading indicator of productivity.
The back half is about what endures. Physics and cognitive psychology have not changed. The theory of constraints is still Justin’s north star. Psychological safety remains the strongest predictor of a productive team, and it is under more threat now than in years. Justin also previews what DX is building next: the CAFE(S) framework for context engineering, and PDLC Insights that bring the DXI’s rigor to product managers and designers. Because once the code is instant, the bottleneck moves somewhere else.
Links & Resources
Guest Links
* DX
DX Research & Resources
* State of AI Impact in Engineering: Q3 Report
* The DX Core 4 and the Core 4 guide
* Developer Experience Index (DXI)
* CAFE(S): Your Agent Is Only as Good as Its Context in ACM Queue, and the DX whitepaper
* AI Effectiveness report docs
Frameworks & Research Referenced
* DevEx: What Actually Drives Productivity by Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler
* The SPACE of Developer Productivity by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler
* Google’s Project Aristotle on psychological safety
* Theory of Constraints and the five focusing steps
* Likert scale and “top box” scoring
* Big Ball of Mud, the classic Foote and Yoder paper
* LinkedIn’s Developer Productivity and Happiness Framework
Books Mentioned
* Peopleware: Productive Projects and Teams by Tom DeMarco and Timothy Lister
* The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt and Jeff Cox
* It’s Not Luck by Eliyahu M. Goldratt
* The Infinite Game by Simon Sinek
People Referenced
* Abi Noda, CEO and co-founder of DX
* Dr. Margaret-Anne Storey, University of Victoria, co-author of SPACE and DevEx
* Dr. Nicole Forsgren, co-author of DORA, SPACE, and DevEx
* Brian Houck, distinguished scientist at DX, formerly Microsoft developer productivity research, lead author of CAFE(S)
* Eirini Kalliamvakou, GitHub, co-author of CAFE(S)
* Max Kanat-Alexander, Capital One, co-author of CAFE(S), formerly LinkedIn’s Developer Productivity and Happiness framework
Also Mentioned
* Agile Slovenia, where Justin gave the closing keynote, “Building 10x Organizations in the Era of AI,” on October 21, 2026
* Honeycomb, creator of an observability platform and source of Justin’s “drowning in a sea of useless metrics” sticker
Topics Discussed
From Research to Platform: How DX Grew
DX started with research, not a product. Abi Noda’s joint work with Dr. Margaret-Anne Storey and Dr. Nicole Forsgren produced the DevEx framework, which, alongside DORA and SPACE, meant three metric frameworks were competing for attention. The Core 4 collapsed them into one, and the product grew outward from there.
Key points:
* The CEO is “allergic to scope creep,” which Justin credits for stable growth without sprawl
* The surveys were engineered at a PhD level and get unusual participation: a recent client ran 23,000 engineers at 98% participation
* Evolution went from survey, then system metrics normalized into the Core 4, then benchmarks, then AI impact measurement
Making the Case for Perceptual Data
Leaders tend to value tool telemetry over sentiment. Justin’s counter argument: in almost every case where a qualitative signal and a system metric diverge, the system metric is the one that is broken.
Key points:
* Engineers are great observers of their environment when asked psychologically safe questions about the system rather than about themselves
* Per Deming, 90-95% of productivity output comes from the system, not the worker
* The DXI is a weighted composite scored on a top-box Likert scale; every one-point improvement returns roughly 10 hours per engineer per year
* Nobody will ever score 100 on their Developer Experience Index; entropy always wins somewhere
System Telemetry Made Useful
Hooking up 15 or 20 tools and piling the output on a dashboard produces fragmented, diluted, and unactionable data. The fix is a normalization layer that maps everything back to a small set of meaningful metrics, and a culture that actually uses them.
Key points:
* None of the data matters without an audience committed to continuous improvement; it is an infinite game, not a project
* Don’t fall into the “data readiness trap.” Your data will never be ready. Start now and let qualitative signals guide the cleanup
* You need both: perception tells you builds are too slow; system metrics tell you whether that means two hours or ten minutes
From Data to Action: The Core 4 as a Contract
Dave raises the failure mode of leaders using the data for apples-to-apples team comparisons. Justin reframes the whole thing as change management, which starts with adopting two beliefs and then breaking fragmentation.
Key points:
* Belief one: developer experience is a systems problem, not a people problem. That makes leadership accountable for delivering it
* Belief two: developer experience is the leading indicator of productivity
* Peopleware found about a 20% productivity spread inside a single organization but an 11x spread between organizations. A 10x developer in a 1x organization will never 10x
* A single framework gives frontline managers autonomy to improve their teams however they see fit while keeping the organization aligned on one definition of productivity
* Look for bright spots, not low scores. Get the leaders of high-performing teams together and ask what is replicable
* This data should never be weaponized. Individual performance is an HR conversation, not a DX one
The Core 4, Defined
Justin walks through the four measurements: speed (PR or merge throughput, or DX’s TrueThroughput, which weights PRs by complexity with a small ML model), effectiveness (the DXI), quality (change failure rate or defect ratio), and impact (innovation ratio, the share of time on new features versus keeping the lights on).
Key points:
* The four dimensions are atomic: you cannot derive speed from impact or quality from effectiveness, which is why Justin doubts there will ever be a “Core 5”
* The counterbalancing comes straight from SPACE, what Storey calls “a constellation of metrics in tension with one another”
* None of these are perfect metrics, but they are good proxies for change over time
What Leaders Are Worried About Right Now
Fresh off a travel binge through India, Singapore, Sydney, Western Europe, and the UK, Justin reports that despite the fastest hype cycle of his career, most leaders are still asking first-principles questions about how their developers are doing.
Key points:
* What is good for humans is good for agents: fast CI, automated test gates, and low friction help an agent far more than they help a person. The AI bill is finally funding platform work that should have happened years ago
* Engineering managers are shipping about six times as much code as a year ago, a restoration of the player-coach model
* Spend is up roughly 50x year over year, but the innovation ratio isn't keeping pace. Velocity gains are linear, not exponential
* You know you have hit the next phase of the hype cycle when the CFO starts showing up in meetings
* The response has not been “use less AI.” It has been intelligent model selection and intent gating, or, as Justin reluctantly concedes, “tokenomics”
Measuring Agent Experience and AI ROI
DX now looks at AI through three lenses: platform readiness via scorecards, AI effectiveness via a local daemon that re-inferences completed agent sessions and lets the agent grade the human, and token economics cross-referenced by model and use case.
Key points:
* Justin uses his own DX instance like a Fitbit for his agentic coding sessions. The harshest feedback so far: his scope “rambled back and forth with no clear direction”
* Prompt engineering is not dead. Representing a complex problem as a constraint graph instead of a wall of text uses far fewer tokens and produces more accurate output
* Subscription pricing is subsidized. Everyone will eventually pay what this actually costs, so build the habits now
* The AI metrics tell you who is using what and how often. The core productivity metrics tell you whether any of it is working
* Strap a rocket to a ball of mud, and you do not get a faster ball of mud
What Endures
Physics and cognitive psychology haven't changed, and they are still the fundamental units of productivity theory. What has changed is the amplitude: change failure rate used to swing half a point between organizations and now swings three or four.
Key points:
* The theory of constraints remains Justin’s north star. Go beyond The Goal; every leader should be able to articulate cost, throughput, and inventory, and how to optimize for throughput
* Project Aristotle showed psychological safety overwhelmingly predicts team performance, and it is under threat right now from job displacement fear and from non-engineering leaders who misunderstand what the technology can do
* Deep work pays off even more when paired with an agent, as long as engineers are allowed to execute calmly
* We are still in the early adopter phase. The early majority has not arrived, and practices will keep changing
What’s Next at DX
Two roadmap areas: better signals for AI and context engineering, and better insight into what is holding up product managers and designers.
Key points:
* CAFE(S) stands for Clarity, Actionability, Fidelity, Efficiency, and Security. It aims to do for context engineering what SPACE did for developer productivity, and benchmarks are coming to the platform
* PDLC Insights brings the DXI’s survey rigor (driver research, cognitive interviews, quant validation) to PMs and designers, with system metrics from tools like Figma to follow
* “The code is instant” means engineers should never have been blamed for slow software delivery. Now the 200-day idea-to-PRD cycle is visible, and that is where the friction lives
* When the backlog empties, ideation has to get strategic and scientific. You cannot just fill it to fill it
Hard Boiled Software is hosted by Dave Laribee.
