Offline metrics

How this question breaks down

1. 1ClassificationCore18 minA classifier's score becomes a decision only at a threshold. Precision and recall describe one threshold; ROC AUC and average precision describe all of them; and which one you report depends on how rare the positive class is and what a mistake costs.
2. 2RankingCore17 minA ranked list is judged by where the good items land. Recall@k asks whether they made the cut, which is the retrieval stage's job; NDCG, MRR and MAP ask how high they sit, which is the ranker's. The discount, the cut-off and the way you average over queries all change the verdict.
3. 3CalibrationCore19 minA calibrated model's 0.2 means "happens 20% of the time". Ranking metrics can't see calibration, but anything that adds, multiplies or thresholds probabilities depends on it. Measure it with a reliability diagram and a calibration ratio, and fix it with a small monotone map fitted on held-out data.

Work through the parts in order and you’ve covered the full design. Each part is one topic.