What entitled you to say that?

What the demonstration needs to show

A computer-use agent takes actions in a browser or desktop application: opening a menu, comparing results, editing a document, saving a file. One way to produce training data for that work is to have a person demonstrate it. The resulting trajectory is a sequence of observations and actions, sometimes accompanied by a written rationale for each decision.

In this work, people carried out tasks in a controlled desktop environment and wrote those rationales in the first person. I taught the concepts and coached people whose submissions kept returning with quality errors. We checked the finished output and whether each choice made sense from the information available at the time.

Consider a closed menu. The task asks for a particular colour. The person plans to open the menu and select it before checking whether it is offered.

The ten habits below cover what a checkable record needs: observations made before they are used, choices tied to the request, claims matched to evidence, and checks on what is finally handed over. Each gives a reviewer a place to start and a coach something to ask about.

A decision, in order

I taught each decision in order: observe, reason, act, then confirm. Confirmation had to follow the action.

Observation should identify what the decision depends on. Choosing a product may require its name, relevant attributes and price. Opening a filter may only require its label and location. The reviewer needs enough to follow the choice. An inventory of every icon would bury it.

Reasoning connects that evidence to the request. Confirmation checks a claim against the resulting state. “I clicked Save” describes an action. “The exported file opens and contains the requested table” supports a claim about its result. Those statements need different evidence.

What counts as available information?

I often asked people to imagine working from one screen frame at a time, to notice knowledge they were carrying into the task. This was a teaching constraint. Computer-use systems can receive screenshots, structured interface information and earlier observations or actions; the combination depends on the system. OSWorld describes several such configurations.

For the examples here, the task prompt supplies the requirements and the visible interface supplies evidence about its current state. Earlier observations can support a remembered fact if they are retained in the record, but they do not automatically prove that a changing value is still current. General knowledge can suggest where to look. Something found in a separate browser session cannot silently become an observation in this one.

A model may also see the screenshot itself. Omitting a description leaves out which evidence mattered to the decision, but does not necessarily hide that evidence from the model.

Ten habits behind a record you can check

Each habit below comes with the slip it prevents. One sentence can make two slips at once. “I’ll select black,” written before the menu is open, skips the look and also states something nobody has checked, so fixing it takes both habits.

01. Look before you act

The slip it prevents: acting before looking

A person can take an action without recording the screen evidence it depends on. Filters are a useful example because opening one and choosing a value feel like a single movement to someone who already knows the interface.

The pause leaves room for the option to be missing and the next action to change. A short description of relevant options is enough.

An observation can be incomplete without containing a falsehood. If the screenshot shows black but the rationale skips over seeing it, the missing link is the observation. If the record claims black is offered before any available evidence establishes that, there is an unsupported assertion too.

02. Give the reason for the choice

The slip it prevents: choosing without a reason

Sometimes the record shows what was seen but not why it was chosen. “I like this image” gives a preference where the task needs a selection criterion. The useful explanation connects visible qualities to the request: the subject is clear, the image has enough resolution, and there is space for the title.

“The second image leaves clear space on the left for the heading. The first puts the subject across that area, so I’ll inspect the second at full size.”

Design tasks still involve judgement. Even when the user invites creative interpretation, the rationale should explain how the choice answers the brief. An unexplained preference gives a learner little to go on when the brief changes.

03. Confirm what you claim

The slip it prevents: not checking what it copied

“The file saved correctly” sounds conclusive until someone asks which file, or what “correctly” means. Confirmation needs an identifiable object and evidence appropriate to the claim.

Seeing summary.txt in the output folder confirms its presence there. Opening it and reading the requested passage checks its contents. Neither check should be described as more than it establishes.

“Copy the third paragraph” is ambiguous if the layout changes or several blocks could be counted. Identify the passage by its opening words or subject. For a quotation or exact field value, compare the copied text with the source; for a summary, check that it preserves the meaning.

04. Check the whole before handing it over

The slip it prevents: skipping the final check

A person checks the pieces as they go, then assumes the assembled output must also be correct. But adding an image can move a source link to another page. A later edit can undo an earlier requirement. The final check revisits the completed work against the whole request.

I used a cake analogy in training: before a cake leaves the kitchen, check it against the order. Following the recipe does not establish that the size, decoration and dietary requirements are right for this customer. In a document task, the equivalent is to inspect the actual output, including the parts currently below the fold.

“The document contains the item name, price and image. I still need to check that the source link survived on the second page, then inspect the exported copy.”

Name any outstanding requirement and check it in the result. Saving a draft before checking is fine; this habit is about not submitting the final output unchecked.

05. Reason in the order it happened

The slip it prevents: explaining with hindsight

Doing a task fluently makes it easy to perform several actions and explain them afterwards. By then, the person knows which action worked. That knowledge can enter a sentence presented as if it had been written before the result was known.

Check both where the rationale sits in the sequence and whether it uses information that only became available later. Writing afterwards risks hindsight, though a careful reconstruction can preserve what was known at each step.

In a live demonstration, explaining the intended action before taking it helped keep that boundary visible.

06. Say only what you can point to

The slip it prevents: making up facts

“Making up facts” suggests an invented product or a nonexistent source. The cases that occupied my attention were often much smaller: an option assumed to exist, a remembered price written as current, or a value copied incorrectly.

What these share is an assertion that the available evidence does not support. The person may know the answer from elsewhere, or may simply have mistyped it. Those are different causes, even when the resulting record contains the same wrong value.

When an AI model does this, it is called hallucination, and the same range applies. An invented citation is the famous case. The common one is a small detail, stated with the same confidence as everything around it.

For review, first establish the mismatch. Do not infer dishonesty or a particular misunderstanding from the label. For coaching, the cause matters: a transcription check, a fresh observation and a conversation about prior research solve different problems.

The rule I gave people was to point to the screen evidence for a claim. Beyond that teaching setting, I would ask people to point to the permitted source: the prompt, the current observation or the recorded history. A prediction can be useful, provided it is written as a prediction and checked before it is treated as a fact.

07. Say why you keep looking

The slip it prevents: searching in silence

Scrolling past content is a decision about what is worth inspecting. A record of repeated scrolls may show movement while leaving out why the person kept looking.

“Nothing useful, continuing” does not help much. “This section covers battery life; I’m looking for the dimensions, so I’ll continue down the page” identifies what was rejected and why. That is useful even when the search has not succeeded.

Narrating every small scroll can also bury the work in filler. I found the meaningful distinction was whether new, relevant content had appeared. Repeated movement over one large image can be grouped under the collection rules; reaching a new section calls for another observation. The direction and purpose should remain clear.

The reviewer should be able to follow the search and see why a potentially useful section was passed over.

08. Match the evidence to the claim

The slip it prevents: showing proof of something else

A screenshot earns its place by supporting a claim. A filtered results page may show which constraints were applied. An item’s detail page may support its name and specifications. A home page usually supports neither.

The same capture can be useful for one purpose and irrelevant for another. An output-folder screenshot supports a claim about file location; it does not establish what is inside the file. Evidence should be named for the claim it supports so that someone else can make that distinction.

Negative claims need particular care. An empty result set shows that this search, with these filters, returned nothing at that time. It cannot establish that the item does not exist anywhere. Record the search boundary and report the limitation at the same scale.

When a request cannot be met, an alternative may be worth proposing. Substituting it in the final delivery is a separate decision and must be permitted by the task. Evidence that the original request failed is not permission to change it.

09. Open what you hand over

The slip it prevents: not checking the delivery

The work can be correct in the editor and wrong in the file the user receives. A save may use a different folder. An export may omit content. An image download may produce a preview or the wrong file.

Delivery therefore needs its own observation: locate the intended file, check its name and format, and open the saved or exported version to inspect the content that matters. If the handover is a link, check that it points to the intended artefact and, where the workflow allows it, that the recipient can access it.

“I saved it” supports very little by itself. The final record should make clear what is being handed over and which checks were actually performed. The depth of inspection should match the task; reopening a file is not proof of every property it might have.

10. Make the change repeatable

The slip it prevents: doing steps nobody can repeat

This slip covers gestures such as dragging a resize handle until an image “looked right”. The difficulty is that the target and success criterion can disappear into the gesture.

An explicit setting is often easier to explain and inspect: keep the aspect ratio locked, set the width to 5 inches, then check how it sits beside the text. If that leaves too little space for the caption, revise the width and state why. Trial and error is useful data when the trial has a purpose and its result is evaluated.

Dragging itself is not inherently unreproducible. A system can record coordinates or a path; OSWorld, for example, includes a parameterised drag action. The question is whether the demonstration captures enough of the intended change to interpret and check it. A precisely recorded gesture can still have a poorly explained purpose, just as a typed value can be arbitrary.

When the answer arrives too early

The ten habits name what is missing from a record. Two patterns underneath them explained why it went missing. I was the one who caught them in people’s work, and I taught people the harm they could do. Both involved knowledge the person brought to the task, and either could surface under several habits at once.

Rehearsed discovery

Some people found a solution in their own browser, then performed it in the controlled environment. The recording showed a smooth route with none of the exploration that led to it.

A person who already knows the answer can write a convincing route to it. The uncertain search, the misleading page and the decision to try somewhere else may all vanish. That matters when the data is meant to demonstrate how to find an answer in an unfamiliar setting.

The finished record cannot always establish whether this happened. An unsupported detail can be a clue; a direct route is not proof. The task may supply a URL, or the person may legitimately know a familiar site. I needed to ask how they approached the work before deciding what the correction should be.

An expert route can be useful data. Rehearsal becomes a problem when the collection is meant to show discovery from a defined starting state, but the demonstration depends on knowledge acquired outside it.

Implausible planning

The same problem appeared in plans. A person would name the exact category, filter and control they intended to use before the interface had been observed. The plan read smoothly because the uncertainty had already been removed.

A request may specify a font size or filename, and a familiar application may have known controls. The problem is treating an unobserved, task-specific condition as guaranteed.

A plan can be precise about the goal and conditional about the route. This mattered because I had asked people for more detail elsewhere, and some took that as a call for more certainty.

Reading an error without guessing its cause

I would review one decision with its observation and history, rationale, action and result. For a final check or delivery error, I would inspect the finished artefact and handover. A sentence alone cannot show whether the evidence supports it.

A useful review note names the claim, points to the missing or contradictory evidence, and states what would resolve it. If two habits apply, record both and explain the relationship. Keep a suspected cause, such as rehearsal, separate from an observed defect, such as naming a hidden option.

When reviewers disagree, first establish what the record shows. “There is no observation supporting this price” gives them something specific to resolve before discussing the label.

How I coached the reasoning

The error label told me where to start looking. It did not tell me what the person believed. One person might be treating the rationale as a summary. Another might have solved the task beforehand. Someone else might understand the process but compress two decisions into one because they felt obvious.

I listened to how they answered questions. With their work open, I would ask what they thought the next step should be. If they said they would open a dropdown and select an option, I could ask how they knew the option existed. That brought the missing observation into the conversation without giving them a sentence to copy.

Then I gave them a similar scenario and asked how they would tackle it. Their approach to the changed situation showed me what they understood and where to focus next.

In this fictional exercise, each response calls for a different follow-up question.

Showing the screen frame beside the rationale helped expose assumptions. A sentence that had felt obvious became harder to justify without the knowledge in the person’s head. We could then work through what evidence the next decision needed.

Some people explained themselves more openly one to one. For a shared issue, I kept the group small enough to follow each person’s reasoning. They then returned to their work and corrected it; understanding in the session had to carry through to the task.

Many people corrected the recurring issue after one session; some needed another conversation.

Teach the distinction before live work

“Insufficient rationale” gives someone little to act on. More specific feedback tells them whether to look again, explain a choice, identify an object or inspect the completed output. A curriculum can expose those gaps through different tasks:

If I were setting this up again, I would assess these concepts before live work, building on my onboarding sequence: descriptions of screens and actions, then plans, then questions to check understanding. With limited training time, I would target the distinctions each person had missed.

What a demonstration teaches

A demonstration teaches whatever it contains, including its shortcuts. A record that names an option before the menu opens pairs an unsupported claim with success, and nothing in it separates a checked fact from a lucky guess. A rehearsed route leaves out the wrong turns, and with them any example of recovering when a first attempt fails. The ten habits make those gaps visible in the data people produce, while they can still be repaired.

What I’d want to test

This account comes from coaching. These are the questions that would turn it into evidence:

• Given the same records, would independent reviewers name the same habits?
• Does an agent trained on records that keep these habits behave differently from one trained on records that skip them, on the same tasks?
• How much of the change after a coaching session came from the coaching, rather than from practice and other feedback?

The same question, turned around

The question I asked the people I coached works just as well in the other direction. When an AI reports that the file is saved, the search came up empty or the task is done, what entitles it to say so?

The AI’s own explanation is a weak answer: Turpin and colleagues showed that a model’s written reasoning can leave out what influenced it. The same holds for the “thinking” that reasoning models show before they answer: in tests by Chen and colleagues, models that used a hint mostly did not mention it. A good answer points to evidence you can check yourself, such as the saved file or the searches it ran.

Someone who asks the question, and knows what a good answer looks like, can hand AI work that matters and check what comes back.

When I read a record now, I still come back to the moment just before the action. If the explanation only makes sense after I know what happened next, there is more work to do on the record.

Notes and sources

This account draws on the training I designed and delivered from 2022 to 2025. It has no accompanying dataset or measured error frequencies. Every example, interface and check on this page is invented for it, and the sources below describe other systems. The notes in the margin give the terms AI researchers use for each slip.