Feature pipelines and stores

How this question breaks down

1. 1Batch vs streamingCore21 minSome features can be hours old, some must be seconds old, and some only exist inside the request. Batch jobs, stream processors and request-time code each make one kind cheaply; the design question is which feature goes where, and how to avoid writing it twice.
2. 2Feature storeCore19 minA feature store is a registry of feature definitions plus two stores, an offline one with every past value for building training sets and an online one with the latest value per key for millisecond reads, so several models share one definition of a feature instead of each writing their own.
3. 3SkewCore21 minA model that scores well offline and does nothing online has often been trained on features that serving never produces, computed by different code, joined from the future, or sampled differently. Log what serving saw, build training rows as of the moment of each prediction, and measure the gap every day.

Work through the parts in order and you’ve covered the full design. Each part is one topic.