Completion Is a Proof, Not a Status

A command in a remote coding-agent sandbox wrote one unbroken 4 MiB line. The HTTP stream returned no output and no terminal event. The producer was still alive, blocked on its pipe, and the request had already cleaned up the ordinary handle used to cancel it.

To the caller, the tool call was over. To the machine, the computation had not ended.

Retrying at that moment could run the same logical command twice. Releasing the sandbox could discard files it had changed. Showing a green done badge would convert real uncertainty into a stated fact.

This class of bug matters because coding agents do more than call short APIs. They compile projects, run test suites, install packages, start servers, and write artifacts. Each action crosses several systems that fail independently: the model, the workflow, the HTTP request, the process owner, the log store, and the workspace store.

What follows is the contract Vivek Sharma and I arrived at, the probes we used to qualify it, and the parts we still cannot prove.

None of the parts is new. The thesis is the end-to-end argument: in Saltzer, Reed and Clark’s careful file transfer, the transfer is declared committed only after the receiver reads the file back from disk and the checksums match. The rules are settled practice: idempotency keys, cursor-based log replay, systemd’s TERM-then-KILL across a control group, Kubernetes’ Unknown pod phase and at-most-one StatefulSets, and durable execution, which Vivek Chandela and I have written about before. What is easy to skip in an agent runtime is the join at the end of a run.

One status hides four clocks

Many agent products compress the end of a run into a single state: running, succeeded, failed, lost. At least four clocks settle independently.

1. Transport: did the request or stream return?
2. Process: can the command, or any descendant it started, still execute?
3. Output: have stdout and stderr been drained to a known final byte?
4. Workspace: are the resulting files present in a verified, restorable checkpoint?

Their order is not fixed. A request can vanish while the command continues. A parent can exit while a child holds a pipe open. Output can drain before an asynchronous upload finishes. The model can emit a final response anywhere on that timeline, and that response is an intent to close, not proof that closing is safe.

Replace the status with receipts

The useful abstraction is a small set of evidence receipts, each produced by the component that has the authority to make that claim.

• The session runtime owns the process tree and the byte log. Its receipt proves the process cannot continue, names the final output cursor, and records the last workspace mutation associated with the session.
• The workspace service owns durable publication. Its receipt names a committed mutation watermark, an immutable checkpoint manifest, and the runtime generation permitted to write it.
• The close protocol freezes admission, settles every admitted session, verifies that the checkpoint covers their terminal mutations, and only then publishes success and releases compute.

This makes status a projection over durable facts rather than a value someone writes. If the process receipt is present but the checkpoint receipt is missing, the task is waiting for one named authority to settle one named obligation, and a retry can re-run the join without re-running the command.

Five rules for agent runtimes

1. Identity must outlive transport

A logical command needs a stable invocation key and an immutable command digest. If admission is retried, the runtime should return the existing session rather than launch a replacement. An HTTP request ID, a UI call ID, or a stream connection is not that identity: each can disappear while the process survives.

2. Logs are byte records, not lines

Line readers either cap line length or buffer without bound, and program output is not always text. Drain fixed-size byte chunks into an append-only log with a monotonic cursor; decode and truncate later, for the model or the UI. The control path should retain the bytes and let a new reader resume after its last acknowledged cursor.

3. The model is not the process supervisor

Timeouts, signal escalation, output drain, and recovery must continue when there is no reader and no model turn in flight. Polling a command from the prompt burns context and makes runtime safety depend on whether the model remembers to keep asking. The model decides what to run; the runtime owns how it ends.

4. Unknown is a real state

If a process owner disappears, a missing session record is not proof that its child died. Freeze new workspace admission while the runtime inspects a reachable owner, terminates the owned process tree, or destroys the exact dedicated sandbox. Until one of those produces positive evidence, do not relaunch and do not manufacture a terminal state.

Death evidence must come from a boundary the runtime owns and can observe empty: a killed cgroup that reports no live processes, a subreaper that has reaped every descendant, or a destroyed dedicated sandbox. A process group is not enough: in our probes, a descendant that started its own session survived the group’s termination and held the output pipe open, so the session correctly recorded an incomplete drain. A subreaper-based owner later contained double-fork and setsid descendants.

5. Close is a join, not a callback

Store the model’s final response as pending. Seal the set of admitted sessions. Settle every session: wait for required commands, and stop or hand off long-running services such as dev servers. Record a final filesystem scan, so the boundary also covers writes a file watcher has not yet reported. Advance durable publication through that boundary. Create the checkpoint manifest last, and read back that exact version. Fence the runtime generation. Then, and only then, publish the outcome and release the runtime.

complete(task) =
  admission_is_sealed
  AND every_session_has(death_proof, final_output_cursor, terminal_mutation)
  AND checkpoint_covers(final_scan_boundary)   -- >= every terminal_mutation
  AND checkpoint_manifest_reads_back_exactly
  AND checkpoint_carries_runtime_fence

What the probes found

We isolated the runtime from the model and ran specified synthetic commands on one live managed sandbox: first against the stream-shaped shell path we were replacing, then against a minimal session owner installed in the same sandbox to test the contract. Each cell is a separate probe, so parameters can differ across a row. These are exact observations from qualification runs, not aggregate success rates, and they cover rules 1 to 3.

The 4 MiB case was not ordinary truncation. A line reader with a 64 KiB default limit (Python’s asyncio StreamReader) failed on the line; the child blocked once the pipe filled; the stream cleaned up; and the runtime lost its ordinary control handle in the process. Byte-oriented drain fixed the size assumption, and durable session identity kept control reachable after the reader disappeared. These were two different bugs with one symptom.

The quiet timeout failed for the opposite reason. Elapsed time was only checked when output arrived, so silence prevented the timeout loop from running at all: the deadline was unreachable in exactly the case it existed for. Moving the clock into an independent process owner made it independent of both output and readers.

Rule 4 was tested separately. In a live run we killed the session owner and its child; the runtime found the owner missing in durable state, destroyed the exact dedicated sandbox, recorded that destruction as the death proof, and only then concluded the session. Rule 5 has so far passed contract tests against a real database, not fault injection at every close boundary.

A review checklist for any agent platform

You do not need this architecture to run the test. Ask the team operating any long-running agent:

• What stable key prevents a retried tool call from launching a duplicate command?
• Who owns the process after the request, worker, or model turn disappears?
• Can a new reader replay output from an acknowledged byte cursor?
• What proves the entire descendant process tree can no longer execute?
• What filesystem mutation must the final checkpoint cover?
• Is the checkpoint manifest immutable, and verified by an exact-version read-back?
• Does an unknown process outcome block relaunch, or get relabelled as failure?
• Can the platform explain which receipt allowed it to show success?

If the answers reduce to the stream ended, the exit code was zero, or the model said it was done, the platform has a status. It does not yet have a completion contract.

What this does not prove

Evidence-carrying completion is a safety contract under explicit assumptions: the authority store is not corrupt, exact-version object reads can be trusted, and the sandbox provider either reports process state correctly or can attest that a dedicated runtime was destroyed. It is not formal verification, not a promise of continuous availability, and not exactly-once execution: after positive death evidence, a retry may run the command again on top of whatever the first attempt left behind. If an isolated runtime cannot be inspected or destroyed, close may remain blocked; that is the intended behaviour, and it is still an outage. Bytes lost before reaching any durable surface cannot be reconstructed. Effects outside the managed workspace — an API call, a database write, an email — need their own transaction or compensation mechanism, and nothing here provides one.

The evidence is also narrower than a finished benchmark. The traces above are qualification records rather than a public corpus. High-concurrency remote stress is not yet qualified, and several provider-specific recovery paths remain unmeasured. We have not published a latency distribution from process exit through verified task success. Those are the next experiments, not numbers to infer from the probes.

The larger shift

Recent systems work is making environment state, recovery, and external effects explicit. Agent libOS goes furthest: it finalises each provider effect as committed, failed or unknown, and never treats a timeout or process death as evidence that the effect did not run. Crab and AgentRewind coordinate recovery with environment state; Cordon stages tool effects behind a semantic transaction. None of them, as far as we found, joins process death, output drained to a final byte, and a checkpoint covering the last write before deciding whether a run may be finalised at all.