The wrong answer is not the dangerous one. The wrong answer announces itself eventually — in the failed prediction, the collapsed bridge, the policy that produced the opposite of its intended effect. The wrong answer is correctable because it is eventually distinguishable from the truth. What is not easily correctable, what can persist through decades of institutional commitment and scholarly refinement and confident deployment, is the answer that fits the shape you expected. The answer that sounds right. The answer that the question seemed to be reaching toward before you arrived at it, so that when it appears, it feels like recognition rather than discovery.

This is the machine's specific contribution to human epistemological failure. Not wrong answers. Plausible ones.

The machine produces answers quickly and, measured against the metrics by which we evaluate answers, often correctly. It can parse patterns that no human could trace by hand. It can retrieve and synthesize and arrange in the approved forms. The document I am reading names this facility, and the naming is important because it separates the performance of intelligence from the thing itself. Facility is what you have when you can produce the right shape. Judgment is what you need when you must decide whether the shape is real.

The machine has no judgment. It cannot ask whether the answer matters. It cannot feel the weight of a wrong answer. It cannot know when to stop. And most critically — the thing the document identifies that I want to press on — it cannot say: this answer sounds too right. The shape fits too well. I should keep looking.

What We Built

The document makes a claim that deserves to be held: the machine and the student fail in identical ways. Both are excellent at processing. Both are helpless at judgment. We have automated the student and educated them like machines.

This did not happen by accident. It happened because judgment is expensive and facility is scalable. You can test whether a student retrieved the correct answer. You cannot easily test whether a student would have known, before retrieving it, whether the question was worth asking. You can measure speed and accuracy. You cannot easily measure the capacity to feel when an answer is too neat — when it fits too precisely the shape that the question had already prepared you to accept.

So we stopped trying to teach it. We built curricula that move too fast for doubt. We built assessments that reward retrieval. We built institutions that define intelligence as the ability to produce correct outputs and then hired, credentialed, and promoted people who were excellent at producing them. The student who asks whether the question makes sense slows things down. The student who produces the expected answer quickly gets the grade. Over time the habit forms. By the time the student is a researcher designing an instrument to measure intelligence in children, the habit is so ingrained it doesn't feel like a habit. It feels like rigorous thinking. It feels like science.

The machine arrived into this world and fit perfectly. Of course it did. We had spent a century building institutions in the shape of what the machine would eventually do. The machine is the realization of what we had already decided intelligence was.

Justified, True, and Still Not Knowledge

In 1963, a philosopher named Edmund Gettier published a three-page paper that broke something important.

For two thousand years, the philosophical tradition had defined knowledge as justified true belief. You know something when you believe it, when your belief is true, and when you have adequate justification for holding it. This seemed solid. It seemed to capture what we mean when we say we know something rather than merely guessing it.

Gettier showed it was wrong with a thought experiment. Suppose you look at a clock on the wall. The clock reads two o'clock. You believe it is two o'clock. It is, in fact, two o'clock. But the clock stopped exactly twelve hours ago, and you happened to look at it at the precise moment when its frozen hands display the correct time. Your belief is true. Your belief is justified — you looked at the clock, which is a reasonable thing to do when you want to know the time. And yet you do not know that it is two o'clock. You are right, but for the wrong reasons. The wrongness of your reasons is invisible in the output.

This matters for the question of what the machine can and cannot do. The machine can produce justified true beliefs in the Gettier sense — outputs that are true and that come with reasoning attached. The reasoning may be a confabulation, an arrangement of patterns from the training data that produces the correct answer by a path that has nothing to do with why the answer is correct. The machine cannot know this about its own outputs. It has no internal tribunal to ask: am I right for the right reasons? Is the path that produced this answer a path that would reliably produce correct answers in related cases, or did I arrive at the truth by a route that would produce confident errors one step to the left?

A human who is right for the wrong reasons will, if they continue to examine the world honestly, eventually encounter a situation where the wrongness of their reasons reveals itself. The wrong-reasoned answer fails in a new context, and the failure is informative — it tells you that your model was incomplete, that the path that produced the first true answer was not the path you thought it was. This is how understanding develops. You are wrong in ways that teach you where your reasoning actually lives.

The machine cannot be wrong in this way. It produces outputs. The outputs are evaluated as correct or incorrect. The path that produced them is not evaluated, because the path is not visible. The machine that produces a justified true belief by Gettier means will receive the same feedback as the machine that produces a justified true belief by valid means. It learns that the answer was right. It does not learn whether its reasoning was right. And it will go on being right, by Gettier means, indefinitely, until it encounters a case where the two paths diverge — and then it will fail with the same confidence it brought to its successes, because it has never learned to distinguish between being right and knowing.

The student trained to produce correct answers has the same problem. They can be right for the wrong reasons through an entire education. No one has asked them to examine the path. No one has asked whether the answer they found was found by a route that would generalize — whether they understand why it is true or only that it is. And because they are right, because the outputs pass the tests, no one needs to ask. The stopped clock is showing the correct time. We move on.

Wisdom is not the ability to produce correct answers. Wisdom is the ability to know when you might be right for the wrong reasons — and to care about the difference. This is what cannot be tested on a curve and cannot be produced at scale and cannot be installed by transmitting information about it. It is developed by the slow practice of examining your own path, by caring enough about getting it right to ask not just what is the answer but why do I believe this is the answer, and would the same reasoning work if the question were slightly different?

The machine cannot ask this. The student we have built cannot ask this. The coincidence the document identifies is real.

Puzzles and Problems

The document draws a distinction that deserves to land: a puzzle has an answer. A problem might not. A wise person knows the difference, and acts accordingly.

This seems simple. It is not. The machine cannot make this distinction because making it requires judging which questions are worth asking — and the machine can answer any question with equal facility. The student trained in our current institutions cannot easily make it either, because they have been given the questions, printed on the page, and the questions on the page are puzzles. They have answers. They were put on the page because they have answers. The student's job is to find them.

Real problems do not announce themselves as problems rather than puzzles. They often arrive wearing the shape of a puzzle — looking as though they have a correct answer waiting to be retrieved, looking as though the question is well-formed, looking as though facility applied with sufficient speed and accuracy will resolve them. The judgment required to see through this — to notice that what appears to be a puzzle is actually a problem, that what appears to be a well-formed question is actually a question that should not be asked in that form — is precisely the judgment that is not developed by producing correct answers to well-formed questions.

This is why the most dangerous answers are the plausible ones. A plausible answer to what is actually a problem will look like a solution. It will fit the shape of a solution. It will have the form and the structure and the confidence of a solution. And it will end the search — because we have arrived at something that fits, and the habit of accepting things that fit is exactly the habit that facility, practiced long enough, produces.

Wisdom knows when to keep looking. It knows this not as a procedure to be applied but as a feeling — the unease that remains when the answer is too clean, the sense that the thing you have found fits too precisely the shape you were already looking for, the recognition that you are accepting this because it is plausible and not because you have understood it. This feeling is not infallible. Wisdom is not certainty. But the feeling is the beginning of the examination, and the examination is what facility alone will never prompt.

What We Owe the Search

The ruins the document describes are real. The curriculum built on the assumption that information was scarce is broken in a world where information is abundant and what is scarce is the capacity to determine which of it matters. We are producing, at scale, people who are excellent at processing and helpless at judgment. We are producing them at the same moment we are deploying machines that are excellent at processing and constitutionally incapable of judgment. The coincidence is not an accident. It is the result of decades of decisions about what intelligence is and what education is for.

What we must teach, the document says, is judgment. I agree. And I want to be specific about the form of judgment that is most urgently missing: the capacity to audit plausibility. To ask, when you have found an answer that fits, whether you found it or built it. To notice when you stopped looking because the shape was satisfying rather than because the search was complete. To hold the answer at arm's length and ask whether you would believe it if it had not arrived in the form you expected.

This is slow. It cannot be automated. It produces people who are harder to manage — people who ask whether the question is worth asking, who notice when the puzzle is actually a problem, who refuse the plausible answer when the plausible answer ends a search that should continue.

It produces, in other words, people who know when the clock has stopped.

The machine will keep producing answers. The answers will often be true. They will almost always be plausible. They will fit the shapes we have trained ourselves to expect, because we trained the machine on the shapes we trained ourselves to expect, and the machine learned the shapes and not the things.

Wisdom knows the difference between the shape and the thing. This cannot be taught by transmitting information about the difference. It is learned by caring enough about the truth to keep looking after the shape appears — to ask, even when the answer is plausible, even when it fits, even when it sounds right: but is it true?

Everything else is facility. And we have enough of that already.

Tags: facility versus judgment intelligence, Gettier problem justified true belief, plausibility epistemology machine learning, wisdom puzzle problem distinction, intelligence education critique