Reviewing what an agent wrote

An agent read a two-step rule in its first minute: drop every candidate scoring below a cutoff, then keep the top five. More than twenty minutes and nearly four hundred tool calls later it drafted that rule as “keep the top five”, and later rewrote the section without opening the file again. The run is from Context Aging, an essay I wrote with Susnato Dhar; the essay changes the details and keeps the sequence. A sentence like that reads like any other. Catching it means opening the file, and a page that cites no lines gives a reviewer nowhere to start.

That is the reviewer’s position with everything an agent writes. The run that wrote it is over, and next week the agent will write the document again. This prototype is about the review, done inside the document, and its sections follow the order a reviewer works in. The content is invented; the ideas grew out of review tools I have worked on.

You read it and never click

In a pre-registered study of 308 people using an AI assistant, 189 never opened a source, though the sources still reduced reliance on wrong answers and answers given after a click were more often right (Kim et al., 2025). In another experiment, self-reported trust in answers rose when citations were present, even random ones (Ding et al., 2025). A reviewer who does not click judges each sentence by the sentence alone, which is how the one in the opening would have passed.

So the state of a sentence has to be readable without a click. Every citation here says which of five things happened to it: checked against its lines, cited but never checked, flagged, in conflict, or stale because the lines it rests on changed. A flag spells out the disagreement, as in “The sentence says two days; line 37 says three.”, and a conflict shows both sources. One state is missing: a sentence with no citation has nothing to carry a state. The audit below found only 51.5% of sentences fully supported; every sentence in this invented document has something to open. Each count above the document steps through the sentences it counts: “2 open issues” opens the two sentences. When fact-checkers attached warnings to some false headlines, people rated the untagged false headlines as more accurate, and tagging the verified headlines as well removed the effect (Pennycook et al., 2020). A page that flags only its errors invites the same reading, so the dull sentences carry a state too.

You check a sentence, and the source changes next week

A citation says where a sentence came from, not whether the source supports it: in a 2023 audit of generative search engines, only 74.5% of citations supported the sentence they were attached to (Liu, Zhang and Liang, 2023). A reference document is read for months, and its lines change in that time.

A check is bound to the exact lines it read. When those lines change, only the sentences citing them lose their check; everything else keeps its state. Open citation 1 and press “Change this file”: its sentence becomes “needs a re-check” and nothing else moves. GitHub unmarks a viewed file if it changes, and Swimm keeps copied code snippets in sync and flags the document when a change affects them. This adds the same expiry for a sentence of prose.

You approve a section you only half read

Approving makes a reviewer responsible for a section as written, and it often follows a partial read, which the prototype shows instead of hiding. Each approval keeps a receipt, such as “Approved by you, 2 of 3 sources opened”, where opened means clicked through to the lines. Approving a section with an open issue or a lapsed check says so before anything is published, and if a source behind an approved section changes first, the approval lapses.

GitHub records the diff an approval covered and can dismiss the approval as stale when new commits change it, and Reviewable records which files each reviewer marked as reviewed at which revision. Gerrit goes further: it can mark a file reviewed as soon as it is viewed. Writing what was opened into the approval, where the next reader sees it, is the part I have not seen elsewhere, and the part reviewers may like least: it shows how little was opened.

Then the agent rewrites over your fix

When a reviewer rewrites a sentence, the sentence becomes theirs. Generated code “should never be edited by hand, otherwise you can’t safely regenerate it”, so the Generation Gap pattern keeps the handwritten part in a class the generator never touches (Fowler, after Vlissides), and EMF moved the rule inside the file, leaving any method without its @generated tag untouched (EMF docs). Shared editors have the other half, a change a commenter may suggest and an editor accepts (Google Docs). This prototype joins the two with the roles reversed: the agent suggests, and a sentence is protected because a person wrote it.

In the prototype, Dana has corrected when a day’s totals become final. Request changes on “Daily totals” and regenerate: the agent notices that the nightly rollup now starts later, flags Dana’s sentence and pins a proposed change under it. Dana’s sentence stays as written until someone approves the change. Without that rule every regeneration puts earlier corrections at risk, and reviewers learn not to correct.

Open questions

I have not tested this with reviewers. The measure I would use is planted errors: put known mistakes into a document and count how many each reviewer catches before publishing. The receipt gives a second measure, sources opened per approval.

The checker has a blind spot the opening shows. That sentence agreed with its lines and left a step out, and a check that compares a sentence with its source can say they disagree, not that the sentence says less. It would have carried a tick here. The only thing on the page that bears on it is the receipt, which says whether anyone opened the lines.

The proposed sentence in the Retries section is the rule that When knowledge reaches the agent carries into an agent’s session.