NLP and vision models

How this question breaks down

1. 1TextCore22 minA text encoder splits text into subword tokens, lets every token look at every other through self-attention, and pools the result into one vector per text, so the design questions are which tokens, how long, which pooling, and whether query and document meet before or after encoding.
2. 2ImagesCore20 minAn image encoder turns pixels into a vector, either by sliding small shared filters over the image (CNNs) or by cutting it into patches and running a transformer over them (ViT); the designer picks the backbone, the input resolution, the layer to read, and how much compute each of millions of photos may cost.
3. 3MultimodalCore18 minMultimodal models combine text and images by merging separate encoders' outputs, by training two encoders into one shared space so a photo and a sentence can be compared directly, or by feeding image tokens to a language model; the choice decides what you can do (search across modalities, tag with no labels, describe a photo) and what each request costs.

Work through the parts in order and you’ve covered the full design. Each part is one topic.