As marketing professionals, you're no stranger to the exhilarating dance of A/B testing. You've celebrated victories — the soaring conversion rates, the triumphant headlines. And you've mourned defeats — the flatlining metrics, the dashed hopes. But what about those murky, in-between moments? The tests that neither shout "Eureka!" nor whisper "Back to the drawing board." The ones that leave us suspended in uncertainty. Today, let's explore the five stages of grief that accompany these often-forgotten experiments.
1. Denial
Picture this: Your latest test has been running for weeks. It's like a marathon runner stuck in a perpetual sprint. You cling to optimism. "There's still time," you tell yourself. "Statistical significance is just around the corner." Denial wraps you in a comforting cocoon. You scrutinize the test setup, check the tags, tweak the parameters. Maybe if you squint hard enough, the results will magically align with your expectations. But deep down, you know—the sands of reality have shifted.
Think about the development impact of waiting. If you keep that test running and just not getting any results that are telling you whether it’s a good idea or a bad idea, how will that impact your next Sprint? How will that impact your next release and how far will that push out other possibly winning ideas?
2. Anger
The numbers mock you. The inconclusive graphs taunt your patience. Anger flares—a primal response to emotional discomfort. You're not vulnerable; you're furious. "Why did we launch this test?" you demand. "Whose idea was it? Wow, we've wasted so much time." But pause. Breathe. You didn't fail; you explored. Every inconclusive test is a breadcrumb—a lesson in disguise. Consider the value of knowing what doesn't move the needle. Those areas where you can unleash creativity without fear. Conversations with designers become liberating: "Go wild; this part isn't critical."
3. Bargaining
The fog of uncertainty thickens. You’re no longer in denial, but you’re not ready to surrender either. Bargaining sets in. “Maybe we can relaunch the test with a slight tweak,” you propose. “Perhaps our statistical significance requirements are too high. Maybe we can be confident with being less confident.”
If the potential impact of this test idea was truly substantial for the business, consider relaunching it with a slightly different experience. Tweak it, iterate, and ground your decisions in data—not wishful thinking. Remember, we don’t live our lives by statistical significance levels and p-values. If someone tells us that a decision provides more than a 50% chance of success in real life, we probably would make that decision. So, do we have to make every business decision based solely on rigid statistical rules? Perhaps not.
Consider this: If your company wasn’t testing at all, they might have made a decision one way or the other without the safety net of experimentation. Maybe good enough is better than perfect. Maybe perfect is getting in your way of making a decision that’s probably correct. These are important reflections. If you have learnings from this test, if you arrive at an informed decision, you may be able to leap right into the fifth stage—acceptance
4. Depression
The fog thickens further. You’ve moved beyond anger; now you’re adrift in the abyss of doubt. “Maybe we should stop testing,” echoes in your mind. “Maybe we have just hit a wall. Maybe it just doesn’t make sense to do it anymore.” The weight of indecision presses down. But pause. Take a breath. You didn’t fail; you explored. Every inconclusive test is a breadcrumb—a lesson in disguise. It’s time to reexamine our roadmap. Let’s dissect every assumption we’ve made. Pull it apart, scrutinize it, and figure out why we’re wrong.
But remember, you’re not alone in this journey. Gather your team. Seek input. Hold a retrospective. What went well in this test? What did we learn, if anything? And most importantly, what would we have done differently if we ran it again? As a group, come to a consensus.
5. Acceptance
The final stage beckons—the phoenix rising from the ashes. You've shared findings, dissected failures, and learned. Acceptance isn't resignation; it's wisdom. Turn off the test; let the data rest. Analyze, segment, and glean insights. Consider the development impact of waiting. Then, as the bargaining phase fades, make an informed call. Maybe it's not perfect, but good enough. Share your learnings across the organization. Remember, credibility lies in transparency. And as we emerge from the five stages, we emerge wiser.
At ZIlker Trail, we've weathered the storms of experimentation. Let's chart new courses together!
Comments