Human labeling.

Loftmarket is a fictional second-hand marketplace for phones, sneakers, handbags and furniture. Its search ranker learns from clicks and purchases, because a click really does say something about relevance. Its prohibited-item classifier cannot. A hunting knife, a recalled baby crib or a fake designer bag gets clicked as eagerly as anything else, often more. The question the model must answer is 'does this listing break our policy?', and no amount of user behaviour answers it.

So someone has to read the policy, look at a listing and decide. The policy lists 14 prohibited categories. Loftmarket's trust-and-safety team has a few policy experts who know it cold; they cost about $60 an hour and there are not many of them. A labeling vendor supplies dozens of trained people at about $15 an hour who are quick but make more mistakes. The whole job is spending expert hours only where they change the answer.

Three levers control cost and quality: who labels an item, how many people label it, and which items get labeled at all. The rest of this topic pulls each one.

Snow, O'Connor, Jurafsky and Ng (EMNLP 2008) paid Mechanical Turk workers to redo five language-annotation tasks that experts had already labeled. On the affect-recognition task (seven emotion and valence ratings), about four non-expert labels per item, averaged together, matched one expert's agreement with the other experts. Some subtasks needed only two, and one (fear) never got there. At 2008 prices that meant at least 875 expert-equivalent labels per US dollar. Treat the prices as history, and the ratio as a result for that kind of task: simple judgments with clear instructions. A policy call about whether a replica prop gun counts as a weapon is not that kind of task, which is why Loftmarket keeps experts for the hard cases.

The arithmetic for three labelers at p = 0.7: the majority is right if all three are right (0.7³ = 0.343) or exactly two are (3 × 0.7² × 0.3 = 0.441), so 0.784 in total. The same sum at p = 0.4 gives 0.352, lower than one label on its own. Voting amplifies whatever side of 50% the labelers sit on, so it rescues decent labelers and sinks bad ones.

Plain majority vote also treats a careful labeler and a careless one the same. Dawid and Skene (1979) fixed that: estimate a confusion matrix for every labeler (how often each true class gets each answer) together with the likely true label of every item, alternating the two steps by expectation maximization until they settle. A labeler who calls every replica 'counterfeit' then counts for less on replicas, and one who is reliable on weapons keeps full weight there. The same idea, applied to heuristic rules instead of people, is the label model in the weak-supervision topic.

Gold questions measure labelers directly instead of inferring it. Five per cent of every queue is listings the experts have already decided, indistinguishable from the rest. Loftmarket's rule: a labeler whose rolling gold accuracy drops under 85% is paused and retrained, and their labels since the last good check are sent for another look.

Each Monday the classifier scores every unlabeled listing. The picker chooses 2,000 of them, the vendor team labels them during the week, the aggregator turns judgments into labels, and the classifier is retrained on the larger set. Next week's picks come from the new model, so each batch goes after whatever confuses the model now rather than what confused it a month ago. Settles (2010) calls this pool-based active learning: the learner sees a large unlabeled pool and asks for labels on the items it expects to learn the most from.

It can work well. Settles' survey shows a text classifier separating baseball from hockey posts that reaches 81% accuracy after 30 uncertainty-chosen labels, against 73% after 30 random ones (logistic regression, 10-fold cross-validation), and the active curve stays above the random one for all of the first 100 labels. The survey is equally clear about the failure cases.

• Outliers look uncertain. A listing in Welsh, or a blurry photo with no text, sits near 0.5 because the model has never seen anything like it, not because it is on the policy boundary. Labeling a thousand of them teaches little.
• Picking the top 2,000 by uncertainty from one model snapshot tends to pick near-duplicates, which is why batches need a diversity step; a naive batch can do worse than random.
• Labels chosen this way are a biased sample of the marketplace. Measuring accuracy on them tells you how the model does on its hardest cases, not on what buyers see. Loftmarket keeps 2,000 uniformly random listings labeled by experts, never trains on them, and measures the model there.
• The labelers are not a perfect oracle. The items the model finds hardest are often the ones people find hardest too, so active batches have lower agreement and need more repeat labels. Mixing about 20% random items into each weekly batch keeps the training set anchored to the real distribution.

Human labels are the right answer when the question is a judgment and you can afford a few thousand of them a week. They are the wrong first move when you need labels on day one or ten times more than the budget allows; then heuristics and a label model get you started, as the weak-supervision topic shows, with a small hand-labeled set kept for checking them. When a label source already exists but is known to be unreliable, such as the category sellers pick for their own listings, the job is finding and fixing the wrong ones, which is the label-noise topic. Human labeling is what both of those still lean on for their test sets. And before paying for any of it, Zinkevich's Rules of Machine Learning (#27) suggests the cheapest step: have human raters measure how often the unwanted behaviour happens, so you know the problem is big enough to label for.