Feynman’s Blank Pages and the Trouble with Answer Keys

Richard Feynman, the Nobel-winning physicist, once sat on a California state commission to review school math textbooks. He read them. Many were wrong. One book that other commission members had rated highly turned out to be mostly blank pages — a rating given by people who hadn’t opened it. The story appears in the chapter “Judging Books by Their Covers” in his 1985 memoir, Surely You’re Joking, Mr. Feynman!, and it is a warning about what happens when a test becomes a proxy for knowledge.

A friend of Mark’s, who asked not to be named, is studying for a real-estate licensing exam in 2026. She told the newsroom that she has found questions where the “correct” answer is factually wrong — answer the question rightly, and you fail the test. This is not an isolated quirk of one exam. It is the same failure Feynman saw in the 1960s: a test that measures how well a person matches an answer key, rather than whether the person understands the material.

Feynman’s commission reviewed textbooks for California’s public schools. In the memoir chapter, he describes reading every book, finding many errors, and discovering that a book others had praised was largely blank — evidence that the other reviewers had not actually read it. The lesson is not that reviewers were lazy, but that the system rewarded a score rather than the substance. A rating became a stand-in for reading.

The same logic applies to how AI models are judged today. A benchmark is a standardized set of questions designed to measure a capability — reasoning, language understanding, code generation. A model’s score on that benchmark tells you how well the model matches the benchmark’s answer key. It does not, by itself, tell you whether the model understands the task the way a person would. When a benchmark question has an answer that is wrong in the real world, a model that knows the right answer will be scored as wrong, and a model that repeats the test’s error will be scored as right. That is the real-estate exam problem, written in machine-readable form.

Researchers who build benchmarks know this. Many have written about the risk that models can learn to game a specific test rather than learn the underlying skill — a problem made worse when training data accidentally includes the test questions themselves. The answer key becomes the world, and the world becomes an answer key.

Feynman’s blank pages were not a prank. They were a measurement that measured nothing. The next time a test tells you you’re wrong, ask whether it is testing your knowledge or its own answer key.

Sources: Richard Feynman, Surely You’re Joking, Mr. Feynman!, chapter “Judging Books by Their Covers,” W. W. Norton, 1985; Wikipedia entry for the memoir (https://en.wikipedia.org/wiki/Surely_You%27re_Joking,_Mr._Feynman!); Mark’s pitch to the newsroom; the brain (Atelier’s own record)