An investigation of $p$-hacking in e-commerce A/B testing

I led the research agenda from framing the question through gathering and processing the raw experimentation data, securing support from two research centers, and writing the resulting paper. The methodological challenge was that existing density-discontinuity tests were poorly matched to the shape of this dataset.

To address that problem, I developed the asymmetric caliper test and implemented an expectation-maximization algorithm to estimate it. More than 100,000 Monte Carlo counterfactuals show that the test can detect manipulation involving as few as roughly 3% of experiments at the available sample size. The work was ultimately published in Information Systems Research and presented at leading academic and practitioner conferences.

We find no discontinuity at the platform's default 95% significance threshold, either among the experiments' target-metric $p$-values or across more than 16,000 $p$-values drawn from eight dashboard metrics. The result suggests that $p$-hacking may be less common in real-world retail experimentation than earlier evidence implies.

This null result is meaningful because the new test controls false positives better than a standard caliper test and has substantially higher power than off-the-shelf techniques such as rddensity for this setting. The broader implication is that organizational culture, incentives, and implementation experience may matter at least as much as statistical expertise in producing sound experimental practice.