An investigation of $p$-hacking in e-commerce A/B testing

I designed and executed a study of 2,270 e-commerce A/B tests run by 242 firms to ask a simple but high-stakes question: Do practitioners p-hack? Answering it effectively required developing a new statistical tool—the asymmetric caliper test—which has Pareto-improving characteristics for investigating this question compared to off-the-shelf techniques. I used Python to implement the new test and fit a beta-uniform mixture model using expectation-maximization techniques. The final result is a large-scale peer-reviewed study that demonstrates $p$-hacking may be less common in real-world retail experimentation than previous evidence suggests.

No discontinuity at the platform's default 95% significance threshold across target-metric $p$-values—or across 16k+ $p$-values on eight dashboard metrics.

The asymmetric caliper test introduced in this paper controls false positives better than a standard caliper test and has significantly higher power than off-the-shelf state-of-the-art techniques like rddensity for the problem at hand.

Discussion in the paper highlights how organizational incentives and implementation experience may matter more for sound statistical tests, as opposed to statistical expertise alone.