Choosing a recommender by evaluating the evaluation

The question was which recommendation approach to use. The answer turned out to be that the question could not be settled with the evaluation we had, and that finding out was more valuable than picking one.

What was done

The reasoning came first, in What collaborative filtering assumes about people: user-user and item-item filtering compute nearly the same thing while assuming different things about where taste lives. Those are structural claims, and structural claims are cheap, so the next step was to run them.

Reading a recommender notebook in the open applies a popularity baseline and both similarity models to a real play-count dataset, tuning each one. The notebook is on that page in full — code, output and all — because a comparison whose numbers cannot be checked is an assertion.

What it showed

The popularity baseline was harder to beat than expected states the finding: both models improved on the baseline, and the margin was small enough that this evaluation cannot confidently rank them. Tuning mattered more than the choice of model.

Why the metric was the problem

Play counts in this dataset concentrate heavily. When most of the signal is concentration, a model that reproduces the concentration captures most of the available accuracy — so an accuracy metric will score all three approaches similarly regardless of how differently they behave for an individual user.

That is not a flaw in either model. It is a mismatch between what the metric rewards and what a recommender is for, which is surfacing something a person would not have found on their own.

Recommendation

Do not choose a model yet. Build the evaluation that can tell them apart first: measure coverage of the long tail, and measure improvement on users with short histories separately from the aggregate. If neither similarity model beats the baseline on those, the honest answer is that this catalogue does not need a recommender.

That ordering costs a week and prevents shipping a model whose value cannot be demonstrated. Evaluation describes the general form of the argument.