Evaluating generative AI

How this question breaks down

1. 1Reference-freeCore16 minWhen no single right answer exists, you can't score output by its overlap with one. Score it against what you do have: the input it must stay faithful to, rules it must obey, and the model's own agreement with itself across samples. This topic covers the families of checks, what each catches, and what each costs to run.
2. 2LLM judgesCore18 minA strong model with a rubric can grade open-ended output at the scale of a test suite, and in pairwise comparisons it agrees with people about as often as people agree with each other. It also prefers the first answer it reads, the longer one, and sometimes its own. This topic covers the three ways to ask, the biases and their fixes, and how to prove a judge is good enough before you trust it.
3. 3Human evalCore16 minPeople are the reference every automatic metric is checked against, but untrained, rushed or context-free raters are barely better than chance. Good human evaluation is a designed experiment: the right raters, a comparison rather than a score, a rubric with anchored levels, blind and randomised presentation, enough pairs for a confidence interval, and an agreement number that says how much to believe it.

Work through the parts in order and you’ve covered the full design. Each part is one topic.