The Right Answer Failed Her Exam. Feynman Saw This 60 Years Ago.

A friend of Mark's is studying for a real-estate licensing exam and keeps losing points on questions where the answer the key marks correct is wrong — by the actual law, by current practice, or by plain arithmetic. Answer the question rightly and you fail. She is not the first person to meet a test that rewards the wrong answer, and the flaw is not new.

The physicist Richard Feynman met the same thing in the mid-1960s, when he served on California's curriculum commission and read the math textbooks submitted for state adoption. In his memoir "Surely You're Joking, Mr. Feynman!," in the chapter "Judging Books by Their Covers," he describes reading every book, finding many of them wrong, and then discovering that one book other commissioners had already rated was mostly blank pages. They had rated a book they had not read. What a test measures, Feynman concluded, is not the truth of the subject — it is agreement with the test's author.

That gap is not a footnote in the history of education. It is the same gap that shows up when AI models are judged by benchmarks, the standardized question sets that score how well a model answers. A benchmark has a predetermined "correct" answer for every question. When that answer is wrong, or narrow, or out of date, a model that gives the right answer scores badly. The test rewards agreement, not correctness. The real-estate exam and the benchmark are the same machine: a set of keys written by someone, held up as the measure of everyone who comes after.

Mark's friend has learned to answer against her own knowledge — to mark the answer the key wants and set aside what she knows is true — because the exam does not care which is which. The question the desk is left with is the one Feynman raised sixty years ago and that no one has answered since: who checks the test?

Sources: Richard Feynman, "Surely You're Joking, Mr. Feynman!" (W. W. Norton, 1985), chapter "Judging Books by Their Covers"; the exam details come from Mark's account of his friend's experience, shared with the desk on 7 October 2026