A higher offline AUC naturally creates the expectation of an online gain. The relationship is rarely that simple.
Offline evaluation asks whether a model fits labels under a historical dataset and a chosen definition. The online system faces users, content, and traffic allocation that are changed by the model itself. The missing link is the process that generated the data.
Inspect the data before the metric
A credible comparison should answer several questions:
- Were training and validation data split by time?
- Can the same user or item leak across the split?
- Are negatives natural non-clicks or sampled examples?
- Is the label window complete?
- Does the dataset only contain items selected by the previous policy?
The final question matters because logs describe what the old system was willing to show. They say much less about the new model in unexplored regions.
Do not ask one metric to do every job
AUC measures global ranking quality but may be insensitive to the first few positions. NDCG, MRR, and Precision@K are closer to limited display slots. Calibration error asks whether predicted probabilities can safely drive downstream decisions.
| Model role | Signals worth watching |
|---|---|
| Candidate filtering | Recall@K, coverage, latency |
| Final ranking | NDCG, segment AUC, top-position changes |
| CTR prediction | LogLoss, calibration, segment stability |
| Multi-objective ranking | Per-objective gain, constraints, Pareto movement |
Segments are closer to the real problem
An average gain may come from a small group of highly active users while new users or long-tail creators regress. Useful slices include tenure, activity, content type, freshness, traffic source, and serving cost.
A model that works only for some segments is not necessarily a failure. It may be evidence for a segmented policy instead of one universal ranker.
Put offline evaluation in the right place
Offline results are excellent release gates and efficient filters for bad ideas. They do not replace online experiments. A safer path is offline regression, guarded low-traffic testing, and gradual expansion.
The long-term asset is not one metric increase. It is the accumulated relationship between offline signals and online outcomes.
DISCUSSION