Transfer learning and compression

How this question breaks down

1. 1Fine-tuningCore21 minFine-tuning starts from a model someone else pre-trained on far more data and changes only as much of it as your labels can support: a new head, the top layers, every weight at a small learning rate, or millions of low-rank adapter weights beside a frozen base.
2. 2DistillationCore17 minDistillation trains a small student to copy a big teacher's full output distribution, not just its top answer, so the student learns which wrong answers are nearly right and gets much closer to the teacher than the same small model trained on labels alone.
3. 3QuantizationCore22 minQuantization stores and computes weights (and often activations) in 8 or 4 bits instead of 32 or 16, mapping each real value to an integer through a scale and zero-point; pruning removes weights outright. Both shrink memory and bandwidth, but only formats the hardware can run fast turn into speed, and the accuracy they cost hides in outliers and rare cases.

Work through the parts in order and you’ve covered the full design. Each part is one topic.