Image by Author
A/B testing has become the backbone of product development at tech companies. When done right, it’s how you figure out what actually works. But even companies with world-class data teams get it wrong. Sometimes they build the wrong thing. Sometimes they break their own tests. And sometimes they run tests they never should have run in the first place.
Here are five instructive failures from companies like Etsy, Duolingo, and Facebook. Each one teaches a different lesson about what can go wrong and how to avoid making the same mistakes yourself.
When Your Product Fails the Test
1. Etsy’s Infinite Scroll Disaster
Etsy spent months building infinite scroll for search results, expecting users to view more items and buy more. The logic seemed sound: remove friction, keep users scrolling, increase engagement. Instead, the test showed the opposite: users clicked fewer results, favorited fewer items, and stopped using search altogether.
Here’s what happened. Traditional pagination gives users clear stopping points and a sense of progress. Infinite scroll removed those landmarks. Users got lost, couldn’t find items they’d seen before, and felt overwhelmed by the endless stream. The hypothesis that more items equals more engagement turned out to be backwards.
The lesson: A “failed” test that kills a bad idea is actually a success. Etsy’s mistake wasn’t the test. It was building a massive feature without validating the core assumptions first. They could have run a quick prototype test or surveyed users before investing months of engineering time. The A/B test did exactly what it should. It just delivered bad news.
2. Duolingo’s “You Signed Up” Badge
When Duolingo first tested achievement badges, they gave users a badge that simply said “You signed up”. It had zero effect on engagement. The post-mortem revealed the obvious: signing up isn’t an achievement, and one badge doesn’t create the psychology of collecting. The team learned that gamification isn’t a feature you can sprinkle on. It’s a system that requires careful design. A single badge fails the same way a single note isn’t a song.
The lesson: “Minimum viable” doesn’t mean “barely functional.” The test correctly identified that their implementation was too weak to matter. Sometimes you need to build enough of the system to actually test the concept. This isn’t about ignoring MVP principles. It’s about understanding what constitutes a viable test.
When Your Test Methodology Fails
3. Heap’s Early Stopping Problem
Heap ran an A/B test on landing page headlines and saw an early winner. They shipped it. Later, they discovered the result was just random noise. When the test ran longer, there was no real difference.
What happened? They kept peeking at the results and stopped as soon as they saw statistical significance. This is like flipping a coin until you get heads, then declaring the coin is biased toward heads. When you peek repeatedly, you’re giving yourself multiple chances to find a false positive. Heap calculated that this approach inflated their false positive rate to over 60%. Most of their “wins” were just random variation.
The lesson: Statistical significance is only valid when you follow the rules. Early stopping destroys your results. You need to set your sample size upfront and stick to it. If you must peek, use sequential testing methods that account for multiple looks. Otherwise, you’re just fooling yourself with statistics.
4. SumAll’s Optimizely Illusion
At SumAll, Optimizely experiments showed massive “wins”: 60% lift here, 15% there. Almost no losers. The team celebrated. Leadership loved the results. But there was a problem: overall user acquisition stayed completely flat.
After a painful investigation, they found their testing process was fundamentally broken. They were committing all the classic sins: early stopping, underpowered experiments, testing too many variations without adjusting for multiple comparisons. The tool wasn’t lying. They were just misusing it. Every “win” was a false positive, and they’d built an entire growth strategy on statistical noise.
The lesson: When all your tests are “winners” but your business metrics aren’t moving, your testing process is broken. This is a useful diagnostic. If you’re seeing consistent wins without business impact, stop and audit your methodology before you keep going. Real wins should show up in your top-line metrics eventually.
When Your Test Succeeds But Everything Else Fails
5. Facebook’s Emotional Manipulation Experiment
In 2012, Facebook manipulated the News Feed for 689,003 users to test emotional contagion. They adjusted what users saw to be more positive or more negative, then measured whether it affected the emotional content of users’ own posts. The experiment worked. They found statistically significant effects showing that emotions spread through social networks.
From a scientific standpoint, this was a successful test. Clean methodology, large sample size, publishable results. But when the PNAS paper came out in 2014, the backlash was immediate and intense. Users felt violated. Regulators got involved. The core issue: Facebook never got informed consent. They’d treated their users as lab rats without telling them.
The lesson: A technically valid test can still be catastrophically wrong if it ignores ethics, user trust, or stakeholder concerns. Not every valid experiment is worth running. You need to consider the second-order effects like regulatory risk, PR disaster, and user trust erosion that go beyond the immediate test results.
What These Failures Teach Us
A/B test “failures” aren’t all the same. Sometimes the test works perfectly and kills a bad product idea. That’s a win for your process. Sometimes the test itself is broken by statistical errors that invalidate your results. And sometimes the test succeeds scientifically but fails morally or strategically.
The key is recognizing which type of failure you’re dealing with. If users hate your feature, that’s product feedback. Thank the test and move on. If your statistics are broken, fix your methodology before you ship anything else. And if you’re running tests that could damage user trust or violate ethical boundaries, stop and reconsider whether the data is worth the cost.
The companies that learn from these failures don’t just avoid repeating them. They build better testing cultures. They validate assumptions early, respect statistical principles, and consider the broader implications of their experiments. That’s how you turn testing from a source of expensive mistakes into your most reliable guide for product decisions.
