Predicting engagement.

Why five numbers and not one score? Each action means something different to Stoop. A comment starts a conversation the author sees; a share carries the post to people who don't follow its author; a hide is the viewer telling you that you got it wrong. The value model gives each action its own weight, and it can only do that if the engagement model keeps them apart and gets each level right. A model that ranks well but says 8% where the truth is 4% doubles that action's pull in the blend. Facebook's News Feed works the same way: it predicts several kinds of engagement for each story and scores each eligible story independently and in parallel (Meta Engineering 2021).

The visibility rule matters more than it looks. Without it, a post that loaded below the fold and was never seen becomes a negative for every head, and the model learns that posts in slot 20 are unappealing when they were simply unseen. With it, the log only holds posts the viewer had a real chance to act on.

Labels arrive late. A comment can come hours after the post was seen, so the join waits 24 hours before it writes a row, and each row is written once, after the window closes. The price is that the model serving today has learned from impressions up to about two days ago. Split offline data by time as well: train on days 1–14 and test on day 15, the way the model will actually be used. A random split lets the model peek at the same posts' later reactions.

A post an hour old has no history, only counters, so those counters are streamed and read in seconds (see feature-pipelines-and-stores/batch-vs-streaming). Two rules keep them honest. First, log the counter values the model saw at serving time into the impression row, and train on those logged values. Counters recomputed later include reactions that happened after the impression, so training would see information serving never has (feature-pipelines-and-stores/training-serving-skew). Second, smooth early rates toward a prior, so a handful of impressions can't masquerade as a trend. The method, and how strong to make the prior, is in feature-engineering/feature-types; the feed-specific choice is the prior itself. Stoop uses the author's own like rate over the last 28 days rather than a site-wide rate, so a neighbour whose posts usually draw a few likes is not judged against a page that draws hundreds, and the post's own count takes over within a few hundred impressions.

One feature here is easy to overlook: which source proposed the post. It lets the model learn, for example, that a post from the interest source needs a stronger match to earn a like than a friend's post does. It also means a change to the source quotas shifts this feature's mix, so retrain after quotas move rather than before.

Why not just feed position in as another feature? At serving there is no position yet: the order is what you are computing. The simple baseline is to train with position as a feature and serve every candidate the same fixed default (see data-collection-and-labeling/implicit-labels); it works, but the heads are free to entangle position with everything else. A separate shallow tower refines it by keeping the position effect in its own additive term, crossed with device and trained with dropout; YouTube's version and its live comparison against the plain-feature baseline are covered in video-recommendation/ranking (Zhao et al. 2019). Training uses it, so the heads are not credited with engagement the slot caused; serving leaves it out, so every candidate is judged as if shown in the same place. Dropping position on a slice of training rows keeps the heads from leaning on it. One caution: because the old ranker put better posts higher, the tower also absorbs some relevance, which is why the shuffled sessions are needed to check it.

Normalized entropy (NE, defined in offline-metrics/calibration and worked for click prediction in ad-click-prediction/ctr-model) divides a head's log loss by the loss of always predicting its base rate (He et al. 2014). That division is why Stoop reads NE rather than raw log loss: the five heads have base rates from 0.15% to 50%, and a hide head at 0.15% has a tiny log loss even when it has learned nothing. Each head is compared with the live model's NE on the same next-day data.

Freshness is part of the readout too. Every day of data a model has not seen costs some NE, and how fast that cost grows sets the retraining schedule (monitoring-and-continual-learning/retraining draws the curve and cites He et al.'s daily-versus-weekly result). For Stoop the comment head ages fastest, because what neighbours talk about changes with local news, so it sets the pace: retrain daily. With the 24 h settle window, a daily model always serves on data one to two days old, and a candidate is compared with the live model on the same held-out day, never with a model of a different age.

Report every head on slices, not just the average: new viewers (under 14 days), viewers with fewer than 20 friends, video against text, and each source tag. A model that is better on average but worse on out-of-network posts quietly tilts the feed toward friends without anyone having decided that. And offline gains don't always survive contact with real traffic (Zhao et al. 2019 describe this gap), so every candidate that passes these checks still goes through an A/B test.