Weak supervision.

Loftmarket's luxury handbags category goes live with 2 million listings copied in from sellers' other categories. Somewhere between 3% and 8% of them are fakes, the policy team guesses, and nobody has marked which. Hand-labeling enough of them to train a classifier would take weeks, and fakes are rare, so a random batch of 20,000 would hold only about 1,000 counterfeits. The prohibited-item classifier from human labeling does list counterfeits among its 14 categories, but its training labels hold few luxury bags, and telling a good fake from the real thing takes a closer look at stitching and serials than its 20-second judgments allow.

What the policy team does have is knowledge. They know that sellers of fakes use certain words, price far below resale value, and often open an account the week they list. They know that a verified receipt and a long clean selling record point the other way. And the brand partners send takedown notices with photos of listings already proven fake. Each of these hunches becomes a labeling function (LF): a few lines of code that look at one listing and vote counterfeit, vote authentic, or abstain because the rule has nothing to say.

No single LF is good enough to ship. Some fire on a sliver of listings, some are wrong a third of the time, and two of them can vote opposite ways on the same bag. Weak supervision is the machinery that decides, without any ground truth, which LF to trust how much, and then hands a clean-enough training set to an ordinary classifier.

Applying the LFs is a batch job: run six functions over 2 million listings and store a sparse matrix of votes. The label model never sees a listing's photos or text, only that matrix. Its job is to answer two questions with no ground truth at all: how accurate is each LF, and given a row of votes, how likely is this listing to be fake? The end model is a normal classifier that learns from those probabilities and the listing's full features.

The first question sounds impossible, and it is the heart of the method. The trick is that LFs which are each right more often than chance agree with each other more than chance would predict, and how much each pair agrees depends on how accurate both members are. Given enough overlap, the agreement rates pin the accuracies down. The idea is older than weak supervision: Dawid and Skene (1979) estimated each human rater's error rates from the raters' votes alone, with no answer key. A label model does the same for programs instead of people (the human labeling topic covers the original).

The label model can only label rows where some LF fired: 820,000 of the 2 million listings. The end model is trained on those rows, but it learns from everything in them: the photos, the wording, the seller's history. So it picks up signals no rule mentions, such as a counterfeiter who writes "rep" or "UA quality" instead of "replica", or stitching that looks like the fakes in the takedown matches, and it scores the 59% of listings where every rule abstained.

The end model uses a noise-aware loss: its training target is the probability (0.32, 0.45, 0.97), not a hard 0 or 1, so confident rows count for more. In the Snorkel evaluation (Ratner et al. 2017), end models trained this way averaged 132% better than models trained by distant supervision, an earlier heuristic baseline, and came within 3.60% on average of models trained on large hand-labeled sets.

The label model is not always worth it. The same paper notes that when LFs rarely overlap, there is almost nothing to weigh, and when many LFs vote on every row, an unweighted majority is already accurate; the learned weights help most in between. Measure majority vote on the dev set as a baseline before trusting the fancier answer.

Coverage is where an LF fires, overlaps where it fires alongside another LF, and conflicts where they disagree (all as a share of the 2 million listings, illustrative). An LF with low precision is not automatically bad: new_seller_brand at 0.45 is wrong more often than right, but a listing it flags is still 9 times as likely to be fake as an average one, so it earns a real weight. What an LF must not be is wrong in a systematic way the model cannot see, like price_low firing on every broken bag.

So LFs live in a repository, are reviewed like any change, carry a version, and are tested on the dev set every time they change. The dev set is 600 listings the policy experts labeled: 300 at random and 50 drawn from wherever each LF fires, so every LF has enough examples to measure. Labeling a random 600 alone would give takedown_hash about 9 examples, too few to say anything.

LFs run offline, over history, so they can read signals that do not exist when a listing goes live: a chargeback filed six weeks after the sale, a moderator's free-text note, a brand's takedown archive the company may not expose to a serving path. The model that ships reads only what is there at listing time: photos, text, price and seller features. At Google, Snorkel DryBell used exactly this split, turning internal resources that could not be served into servable classifiers, with an average 52% gain from those resources and quality comparable to classifiers trained on tens of thousands of hand labels (Bach et al. 2019).

At Loftmarket the next LFs to write are chargeback_fake (buyer refunded with reason "not authentic") and brand_confirmed (brand partner confirms a report). Both arrive weeks late, which is fine for labeling last month's listings and useless for scoring today's.