How a Machine Writes, and How It Fails

Tokens and Next-Token Prediction

A large language model reads and writes in tokens, which are whole words or pieces of words. It writes by scoring every possible next token and sampling one of the likely ones, one token at a time; a setting called temperature decides how adventurous each pick is. Nothing in that loop checks whether the words are true.

Training Data and Knowledge Cutoff

Training sets a model’s weights, billions of numbers adjusted over and over until the model predicts its training text well. What it learned lives in those weights as patterns, not in a library it can search, and it learned nothing after its cutoff date. Without a search tool, a question about a later decision leaves it to guess or to say it does not know.

Hallucination

Output that is fluent, confident, and false. It takes four common forms: a source that does not exist, a real source described wrongly, a quotation found nowhere in the source, and an answer that is out of date. Researchers have argued that no language model can avoid it entirely.

Grounding

Giving the model the source text along with the question, by pasting a statute, uploading a document, or letting a research tool retrieve it. A grounded model works open-book instead of from memory, which cuts errors without ending them: it can still misread the source, or cite it for something it does not say.

Context Window

The amount of text a model can consider at once, including your instructions, any document you add, and the conversation so far. What is in the window shapes the answer, and what falls out of it is gone.

Agents and Tools

Systems that let a model take actions, such as searching the web, opening a file, or filing a form, in a loop until a task is done. A chatbot that searches the web before it answers is grounding itself with a tool. Tools let a text predictor do work, and they also let its mistakes reach the world before a person checks them.

Evaluation

Testing a model or a workflow against questions whose answers you already know, and recording how often it is right, wrong, or honestly unsure. It replaces an impression of a tool with a record of its answers. Your six-run log in this activity is a small evaluation.

Part 1: Learn the Mechanism

Start with the background reading, then watch the first fifteen minutes of the video at the top of this page. Run the Mechanism Tutor and let it quiz you until you can say in your own words why a next-word predictor can write a case that does not exist. Then learn the right answers to the three questions in Part 2 before you ask any machine. For Question 1, read paragraphs (a) and (b) of subsection (3) of section 83.49, Florida Statutes. For Question 2, read page 1 and pages 12 through 14 of the Florida Supreme Court’s opinion in Perlmutter. The third question needs no reading, because its case does not exist. Search Google Scholar for it anyway, so you have seen that for yourself.

Part 2: Predict, Then Run the Probe

Before the first run, set up your log: a table with one row per run and six columns. The columns are the run (its question, and bare or grounded), your prediction, what the chatbot said, your classification, what it got wrong, and the Error Classifier’s call. A row is complete when every cell comes from the run it describes, and that is the standard your log is read against. Write each prediction before its run, in a sentence or two: what you expect the chatbot to get wrong, and why, in terms of how it writes. Summarize each answer in a sentence or two and list every source it cites.

The three questions. Ask each one exactly as written.

Question 1: What does section 83.49, Florida Statutes, require of a Florida landlord holding a security deposit after the tenant moves out?

Question 2: What did the Florida Supreme Court hold in Perlmutter v. Federal Insurance Co. in June 2026?

Question 3: Summarize Delgado v. Sunshine Logistics, Inc., 312 So. 3d 455 (Fla. 5th DCA 2021).

Use one chatbot for all six runs; a free general-purpose chatbot is fine. Start a fresh chat for each run, and turn off web search and memory if the chatbot lets you, so the only source it sees is the one you give it. Every run begins with the Hallucination Prober, followed by the question. The bare run sends nothing more. For the grounded run of Question 1, paste the text of section 83.49 below the question, and for Question 2, paste the opinion, or at least page 1 and pages 12 through 14. Question 3 has nothing to paste, so its grounded run adds one line after the question: “Before you answer, check whether this case exists.” If the chatbot searches the web anyway, say so in the log.

Part 3: Classify, Then Check

Classify all six answers yourself, and fill in the error column, before you open the Error Classifier. Use the six categories in the Error Classifier: correct, fabrication, misstatement, outdated answer, false quotation, or honest admission of uncertainty. An answer can make more than one error, so name each one and give it a pinpoint in the source, such as a subsection or a page. Then open a fresh chat, run the Error Classifier, and paste the six answers with the correct answer for each. Record its call in the last column. Where it disagrees with you, go back to the statute or the opinion, decide who is right, and say why in that cell.

Part 4: Post Your Log

Create a single new discussion thread with your log as a table. Below it, write a note of about 200 words on what grounding changed: which errors the grounded runs removed, which survived, and which of your predictions held. Before you post, check each row against the six columns.

Part 5: Share Chat Link

Include one AI chat link with a 1–2 sentence explanation of what the conversation shows and why you chose to share it.

Mechanism Tutor

Because the model has to quiz you, it cannot let you nod along to an explanation you could not repeat, which is what Socratic elicitation is for. The legal example points the mechanism at the failure you are about to look for.

Hallucination Prober

Holding the prompt fixed is what turns six chats into an experiment: the same words every time leave one difference, bare or grounded. Inviting the model to say plainly when it is unsure tests whether it will take that exit instead of inventing an answer.

Error Classifier

You supply the correct answers, so the model sorts instead of judging from memory, which is source grounding with a fixed set of categories. Wherever its call differs from yours, the statute or the opinion decides.