Label noise.

Every Loftmarket listing gets a category from its seller, chosen from a menu while they upload photos. That choice is free and plentiful, so it became the label for the category classifier, which suggests a category to new sellers and routes listings into browse pages. Nobody checks it. Most sellers pick correctly, but a hurried seller files ankle boots under Sneakers, a seller chasing views files a wallet under Handbags because more buyers browse there, and a few pick the first option on the menu.

Think of whatever produced the labels as a model of its own, with an error pattern you never chose. A useful first question is not how many labels are wrong but what the errors depend on, because the three answers call for different fixes.

Seller choice is only one entry point. Vendor labelers get tired late in a shift. A guideline changes between versions and old labels are never revisited. Proxy labels built from behaviour are wrong in their own ways (see implicit labels). A label model built from rules has its own error rate (see weak supervision). And some noise is not noise at all but a bug in the pipeline that builds the table, which the in-practice section covers.

A large enough network can learn anything, including nonsense. Zhang and colleagues replaced every CIFAR-10 label with a random one and standard image networks still reached zero training error; weight decay and dropout did not stop them. So given enough epochs, the category classifier will memorise that listing #88213, an obvious boot, is Sneakers.

The order matters, though. Arpit and colleagues showed that networks pick up the patterns shared by many consistent examples first and memorise the odd ones out later. A wrongly labeled boot contradicts thousands of correctly labeled boots, so for a while the model keeps predicting Boots for it and its training loss stays high. Two practical tools follow: stop training on a clean validation set before memorisation sets in, and treat a stubbornly high per-example loss as a hint that the label, not the model, is wrong. Co-teaching (Han et al.) builds on the second: two networks each pick the lowest-loss examples in a batch and train the other one on them, so neither feeds on its own mistakes.

When Northcutt, Athalye and Mueller put confident-learning flags from ten benchmark test sets in front of five crowd workers each, 51% of the flagged items were confirmed as errors on average. The rest were correct labels the model found hard, or items where both labels applied. Deleting every flagged Loftmarket listing would therefore throw away roughly half good, hard examples: suede sneakers that look like boots are exactly what the model needs to see.

So flagged listings go to the relabel queue, where a reviewer looks at the photo and picks a category, or "neither", before seeing anything else. Only after that answer are the seller's category and the model's guess shown, so the reviewer can take a second look at a disagreement. The mechanics of that queue (guidelines, repeated labels, agreement) belong to human labeling, which gives the same rule for model pre-labels: a reviewer who sees the model's guess first tends to anchor on it and pass the model's blind spots straight back into the labels.

Noise in the test set does not just lower every score by the same amount. Northcutt, Athalye and Mueller estimate at least 3.3% wrong test labels on average across ten widely used benchmarks, and at least 6% (2,916 images) in the ImageNet validation set. Removing those items left model rankings unchanged, but on the items that could be corrected, judging against the corrected labels reorders models: on ImageNet, ResNet-18 overtakes ResNet-50 once the share of originally mislabeled test items rises about 6 points; on CIFAR-10, VGG-11 overtakes VGG-19 at about 5 points. Larger models had learned to agree with the systematic errors.

Loftmarket's rule follows from that: the 2,000-listing test set is labeled by two category experts with a third settling disagreements, and the cleaning loop never edits it. If the loop were allowed to fix test labels using the model's own opinion, the test set would drift toward agreeing with the model and every score would look better than it is. How the scores themselves are defined belongs to classification metrics.