Measuring systems whose outputs are not deterministic.
Using it for real work.
How do you evaluate a retriever without labels?
Everyone agrees retrieval should be measured, and almost nobody has a labelled set to measure against. Twenty questions written by hand beat two thousand generated ones — a synthetic question is written from the passage it is meant to find, and shares its vocabulary in a way a real one never does.
4 min