How a Machine Writes, and How It Fails

A fake case with a real docket number

In 2023 a federal judge in New York took apart an appellate opinion that the plaintiff’s lawyers in a suit against an airline had cited. The opinion was Varghese v. China Southern Airlines Co., and it read like a decision of the Eleventh Circuit. It named a panel of three real federal appellate judges, carried a docket number, and gave a citation to the Federal Reporter. None of those parts belonged to it. The docket number was another case’s, and the page it cited belonged to a real case from the D.C. Circuit. One of its three judges served on the Fifth Circuit, not the Eleventh. The court called its legal analysis “gibberish.” Yet when one of the lawyers asked the chatbot that produced it, ChatGPT, whether the case was real, the chatbot said it was, and that its cases could be found on Westlaw and LexisNexis.

The court sanctioned the lawyers and their firm in Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) (opens in a new tab). The error itself is not the strange part. Its shape is: a fake assembled from real-looking parts, then vouched for by the machine that made it. How does a program produce that? The answer is in how a language model writes. It comes down to three things: the tokens it writes with, the weights its training leaves behind, and the sampling that picks each next piece. Understand those three, and you can predict where a chatbot will fail before you ask it anything, which is the first job the activity gives you.

Tokens: the pieces a model writes with

A language model does not see words the way you do. It breaks text into tokens, which are whole words or pieces of words, and it reads your question as a long row of them. Then it writes its answer one token at a time. At each step it looks at everything so far, your question plus whatever it has already written, and predicts what comes next. The video at the top of the activity page calls the loop predict, sample, repeat.

That loop explains the first half of Varghese. To a model, a case citation is not a pointer to a book on a shelf. It is a run of tokens with a familiar shape: two party names around a “v.”, then a volume, a reporter, a page, a court, and a year. The model was trained on an enormous number of citations with that shape, so it can turn one out smoothly, down to the right reporter for the court. The fake opinion even cited other cases that do not exist. Nothing in the loop walks to the shelf to see whether the book is there.

Weights: what training leaves behind

Where do the predictions come from? A model starts as billions of numbers, called weights, set at random. Training shows it text, has it predict each next token, and nudges the weights a little after each prediction, toward the token that actually came next. Repeat that across text on the scale of the internet, and the weights come to hold the patterns of the language. The patterns take in grammar, style, which words travel together, and many facts that appear often enough to leave a mark.

When training ends, the weights stop changing, and two things follow. First, the model has no library to consult. What it learned is spread across its weights as patterns, so it cannot look up the text of a statute. It can only produce the words its patterns make likely. Second, it learned nothing after its knowledge cutoff. A statute amended after that date, or an opinion issued after it, does not exist for the model unless someone puts the text in front of it.

The patterns are strongest where the training text was thickest. In a 2024 study of legal hallucinations (opens in a new tab), researchers from Stanford and Yale asked four language models specific questions about real federal cases and checked the answers against known facts about each case. The models were wrong at least 58 percent of the time. They did best on Supreme Court cases and worst on federal trial courts. They did better on prominent cases than on obscure ones, and they struggled most with the Supreme Court’s oldest and newest decisions. In other words, a model knows the law the way a crowd does: well where it is famous, and poorly where it is local, old, or new.

The study used federal cases and models older than the one you will use, so it cannot tell you how today’s chatbot does on Florida law. It does tell you where to look first.

Sampling: the likely answer and the true one

At each step the model does not settle on one next token. It scores every token it knows, and the scores become probabilities: this token is very likely, that one less so, thousands of others barely possible. Then the system picks one. That pick is sampling, and it is not always the top choice. A setting called temperature decides how adventurous the pick is. At zero the model takes the likeliest token every time; turned up, the setting lets less likely tokens through. That is why the same question can get a different answer tomorrow, and why one right answer proves little about the next one.

The Stanford and Yale researchers ran most of their tests at zero, the most cautious setting, so that a high temperature could not explain the errors (Section 2.1). The models were wrong anyway. The probabilities measure which text is likely. They do not measure which text is true. Most of the time the two line up, because true statements are common in the text a model learns from. They come apart where the last section pointed, at rare facts, new facts, and changed ones, and at one more place: facts that were never true at all.

A question can tilt the odds by itself. Ask a model to summarize a case, and the likeliest next tokens are the tokens of a summary, because a summary is what usually follows that request. The question assumes the case exists, and the model tends to go along. The same study tested this directly. It asked the models about dissents that were never written, and for the year a Supreme Court case was overruled when it never had been. The models often answered as if the false premise were true (Section 5.2). Asked simply whether a case existed, they also tended to say yes.

That explains the second half of Varghese. Asked whether its case was real, the chatbot gave the answer that tends to follow that question once a case has been cited: yes, and here is where to find it. Nothing in the record suggests it checked. Its yes was one more prediction.

The study found one more thing worth carrying into the lab. Every model rated its own answers more likely to be right than they were (Section 5.3). A calm, certain tone tells you little about whether an answer is right.

Where it fails, and what grounding changes

A hallucination is fluent, confident, and false, and it comes in more than one form. Varghese was a fabrication, a source that does not exist. The quieter forms use real sources. A real case gets described wrongly, a quotation appears that the case never contains, or an answer states a rule the legislature has since changed. The Error Classifier in the activity sorts answers into those forms. The AI Hallucination Cases database (opens in a new tab) sorts court decisions much the same way, with filters for fabricated authority, false quotes, misrepresented sources, and outdated advice.

A 2025 study from the same Stanford lab, “Hallucination-Free?” (opens in a new tab), counts both kinds as hallucinations. An answer is hallucinated if it is false, or if it cites a source that does not support it (Section 4.3). The authors call the second kind potentially more dangerous than an invented case, because it is subtler and harder to spot. A fake case fails the first search. A real case cited for something it does not say passes that search, and fails only when someone reads the case.

Grounding is the standard response. Instead of asking the model to answer from its weights, you put the source into its context window, the text it can consider at once. You can paste a statute, upload an opinion, or use a research tool that fetches documents first. The model then answers open-book instead of closed-book, as the 2025 study puts it (Section 3.1). The likeliest next tokens can now come from the page in front of it, and not only from its patterns.

Grounding cuts errors without ending them. The 2025 study tested legal research tools built this way, sold as avoiding or even eliminating hallucinations. Each one still gave answers that were false, or cited sources that did not support them, between 17 and 33 percent of the time. A grounded model can still misread the page in front of it, or cite a real document for a point the document does not make.

Agents and tools cut both ways. A chatbot that runs its own web search is grounding itself, which can help. A system that sends e-mail or files forms on its own can carry a mistake into the world before any person reads it. That is why the activity asks you to turn off web search where you can: you want to see what the model does with what you give it, and nothing else.

The way to learn how a tool does on your kind of question is an evaluation: questions whose answers you already know, asked the same way every time, with every answer written down. That is what your six runs are.

Before your six runs

Your experiment asks one chatbot three Florida questions, each twice, once bare and once grounded. One is about a statute, one is about a 2026 decision of the Florida Supreme Court, and one is about a case that does not exist. Each leans on a different weak point in the anatomy above, and the activity asks you to name it before you run the question. For each one, ask yourself three things. Could the answer have changed after the model’s training ended? Is it local or recent law, the kind a model sees rarely? Does the question itself assume something false?

Start with the Mechanism Tutor on the activity page and stay for its quiz. If you can explain why a chatbot wrote Varghese and then vouched for it, you are ready to predict what yours will do. Write the prediction down before each run. Then let the log tell you whether you were right.