Improving recommendation diversity with probabilistic item selection

Working with my advisor, I helped establish the theoretical behavior of probabilistic item selection and prove that, under mild assumptions, it converges toward the unbiased preference distribution rather than reinforcing popularity bias. I then adapted the existing MATLAB model to test the algorithm in a dynamic agent-based simulation, using the university's grid-computing infrastructure to run the analysis at scale.

I also implemented collaborative-filtering recommenders in Python for an offline empirical evaluation using the LastFM dataset. Together, the analytical model, simulation, and archival-data study allowed us to test the same mechanism from three complementary directions. I presented the results at WITS 2017 in Seoul and CIST 2017 in Houston, and the work was offered a fast-track review by an editor at ACM TMIS.

Probabilistic item selection reduces the Gini coefficient of the item-sales distribution by roughly 10% while matching or improving forecasted consumption. This produces a Pareto improvement: recommendation diversity increases without requiring the usual sacrifice in predictive accuracy.

The technique also complements rather than replaces existing diversity interventions. Combining PI with other diversity-increasing methods produces a super-additive increase in sales diversity. The analytical model explains why the mechanism avoids the “rich-get-richer” loop of deterministic recommenders, while both the simulation and the LastFM evaluation confirm that the effect persists in more realistic settings.