An empirical meta-analysis of e-commerce A/B testing strategies

I originated the research idea and analysis plan, then worked directly with the CRO vendor's data-science team to establish secure access to its raw data. I wrote the SQL pipelines that turned event-level PostgreSQL tables into an analysis-ready dataset spanning more than 500 million sessions and thousands of experiments.

The empirical analysis uses inverse-variance weighted least squares with Monte Carlo variance propagation to account for noise in the underlying effect-size estimates. I developed the work into a full-length research paper and presented it at four academic conferences, including MIT's Conference on Digital Experimentation and INFORMS CIST.

Experiment returns are highly concentrated: 20% of tests generate 81% of aggregate conversion uplift. For e-commerce teams, this makes a strong case for running enough experiments to create multiple shots on goal rather than expecting every individual test to produce a meaningful win.

The analysis also shows that experiment type and funnel location cannot be considered independently. List-price promotions become less effective as shoppers move deeper into the funnel, while shipping incentives become more powerful when they appear on cart or checkout pages. In other words, both what a retailer offers and where it surfaces the offer shape the resulting purchase behavior.