If you run an experimentation program, get ready for a shocking statistic: 88% of A/B tests fail to achieve a statistically significant win over the original experience.
That's not an off-hand estimate - it's a figure published by companies like Optimizely that are titans in the experimentation space. Other leading practitioners like Airbnb, Booking.com, and Amazon have shared similar "success" rates as low as 10-15%.
For anyone new to experimentation, that 88% failure rate may seem abysmal. How can an effective program have such a high rate of fruitless tests? Shouldn't we be aiming for a much higher hit rate?
As counterintuitive as it may seem, that 85-90% failure rate is actually the high watermark that top experimentation programs strive for. A success rate much higher than that is a red flag that your program may be playing it too safe and leaving massive returns on the table.
The Danger of Playing It Too Safe
So what's wrong with a high "success" rate? As Keith Swiderski explains, "The number one reason why your winning test percentage may be higher [than 10-15%] is that you're playing it too safe - running a lot of safe tests that you think will win to boost up your winning percentage."
It feels good to rack up wins - you make your boss happy, you hit your goals, you get that bonus. But there's a massive opportunity cost to only running low-risk tests designed for easy victories.
By only making incremental, surface-level changes, you're leaving potentially exponential upside on the table. The tests that could truly transform your product's experience and massively increase key metrics never see the light of day. You leave the biggest wins on the table in pursuit of a high stated "win" rate.
That may keep you employed, but it's not a recipe for driving breakthrough results and helping your company get ahead of the competition. As Keith notes, "It's like any good investment strategy...you want to mix in safe bets with some smart long shots because that's the only way you will outperform the benchmark."
Recalibrating for Massive Wins
So if a high stated win rate isn't the goal, what should you optimize for instead? Keith suggests metrics that directly tie back to your company's core objectives:
"A winning KPI needs to ladder up to the company's business goals and you need to show that the thing you're doing is helping the company accomplish its goals. Winning test percentage may or may not do that."
For most businesses, that means measuring the incremental revenue or revenue lift generated by your experimentation program. You could look at total incremental revenue from all your test wins, revenue delivered per test (like a batting average), or the overall percentage revenue lift your program drove.
By focusing on financial impact rather than simplistic win rates, you realign your program to be judged on real, measurable value creation. That gives you the air cover to run bolder tests - even if only 10-15% of them "win", those few breakthrough winners could generate exponential value that dwarfs what you'd get from playing it safe.
The Value of "Failed" Tests
An 88% test "failure" rate doesn't mean you wasted time and money on fruitless experiments. Those "failed" tests still generated learnings about what did and didn't resonate with your customers. They showed you which bold bets missed the mark so you could course-correct.
In this light, celebrating "wins" too much can become counterproductive. As Keith highlights, it can incentivize you to start gaming the system by only running safe tests you think will boost that figure. True innovation comes from taking risks - even if that means waving goodbye to an unsustainably high "success" rate.
Following the Trailblazers' Lead
Of course, this aggressive testing mindset is easier said than done. Your leadership may balk at pumping resources into a program that will "fail" over 85% of the time. They may question why you're celebrating things that technically "lost" against the original experience.
This is where it's vital to understand that you're simply following in the footsteps of the world's most innovative, boundary-pushing companies. Amazon. Booking.com. Airbnb. Microsoft. Meta. Google. They've all embraced that 85-90% failed test rate as the price of admission for driving exponential growth and market dominance.
As Keith states, that incredible 85%+ failure rate "goes doubly for experimentation. Experimentation in the scientific process helped create some of the most amazing products or life saving medicines that help us live our lives today. And I can assure you that those inventions didn't happen because somebody played it safe."
The world's greatest innovations and most transformative products didn't come from making low-risk, incremental iterations. They came from bold bets, smart long shots, and persevering through countless "failures" to finally achieve greatness.
For any company that wants to be a true product leader and innovator, that 88% failure rate is the expectation, not the exception. Embrace it. Optimize for it. And don't let a simplistic "wins" mentality blind you to exponential value creation opportunities.
So keep swinging for the fences in your experimentation program. You'll strike out far more often than not - but when you finally crush one of those moonshots, it could catapult your product light years ahead of the competition that kept playing it safe.
Comments