An empirical meta-analysis of e-commerce A/B testing strategies

Big Data (500M+) PostgreSQL Econometrics Meta-analysis Conference Awards
Project Overview:
I spearheaded and executed a large-scale meta-analysis of 2,732 A/B tests run by 252 U.S. e-commerce firms, assembling 500M+ anonymized visitor sessions directly from a CRO platform’s raw PostgreSQL data lake. The result: a full research paper using large scale cross-firm meta-analysis that reveals how experiment characteristics and funnel positioning jointly drive e-commerce conversion outcomes.
Paper Thumbnail
Read the Research Paper
35-page detailed analysis with methodology, findings, and managerial implications
Co-authored with Kartik Hosanagar
PDF document →
Key Contributions & Technical Execution
🎯 Research Leadership
Originated the research idea & analysis plan and embedded with the CRO vendor’s data-science team to negotiate secure data access.
🔧 Data Engineering
Wrote SQL pipelines to query raw event tables, extracted dataset totaling over 500M+ sessions across 2,900+ tests, and transformed into table suitable for econometric meta-analysis.
📊 Statistical Analysis
Implemented inverse-variance weighted least squares (WLS) coupled with Monte Carlo variance propagation to handle noisy effect-size estimates.
📢 Research Communication
Wrote full-length research paper and presented at four academic conferences, including MIT and INFORMS CIST (Best Student Paper Award).
Key Research Findings
Power-law Distribution of Returns
20% of experiments drive 81% of aggregate conversion uplift, underscoring the value of getting many shots on goal when A/B testing in e-commerce CRO.
Experiment Type × Funnel Location Interaction
Cross-firm, large-scale evidence on the importance of experiment type and on-site funnel location in shaping conversion outcomes in A/B testing.
Second-order Price-Partitioning Effects
List-price promos fade as shoppers move deeper in the funnel, while shipping incentives surge when surfaced on cart or checkout pages. This demonstrates that what you offer and where you surface it jointly drive purchase behavior.
Recognition & Impact
📚 Conference Presentations
  • Wharton Innovation Doctoral Symposium – Philadelphia, PA
  • MIT Conference on Digital Experimentation (CODE@MIT) – Cambridge, MA
  • Purdue Conference on Data Science for Business – Lafayette, IN
  • INFORMS Conference on Information Systems & Technology – Phoenix, AZ
🏆 Best Student Paper Award
INFORMS Conference on Information Systems & Technology, November 2018
Top recognition in Information Systems