Skip to content

When a stage is done

The exit criterion of every stage between a request and a reliable flow, the false exits that look like progress, the stage of the flow each experiment measures, and when to stop the whole loop or stop and ask.

Work on a flow moves through stages: contract, simplest flow, cases, a first look, error analysis, fixing the specification, hypotheses, Explore, Confirm, applying the finding. Each stage ends on a criterion you can check, not on a feeling. The common false exits are a clean aqven check taken for a working flow, one run taken for proof, and a signal on working cases taken for a finding. Every experiment also measures one part of the flow, and the job that part’s output does downstream decides the metric. The loop stops on one of five conditions, and it stops to ask whenever a decision belongs to a person.

StageDone whenNot done just because
Contractpurpose, the input contract (the fields and input kinds that exist at run time, with typical sizes and lengths), output, a measurable “done” (which check, which number), a budget per run and per series, and a latency are agreed; dataset columns production lacks are not inputs; the models and providers are the ones the owner chosethe request sounds clear
Simplest flowthe fewest steps that express the contract; aqven check is clean; the prompt preview of every llm node is read in full; one run on a real input passed, and its output reads well in the run viewaqven check is clean
Casesa dataset with truth and tags exists; negative controls sit next to the positives; the count per compared tag value and half is knownthe dataset file loads
First looka look over working cases ran with cheap deterministic checks, and every failing case row was readthe series finished
Error analysisthe owner was offered the first traces and wrote notes, or declined and confirmed the modes the agent wrote from them; every failure has its first failing node and a failure mode the owner agreed to; new failing traces stop adding modesthe agent grouped the failures itself
Fix the specificationthe failures the prompt never asked about are fixed in the prompt or the type, and the first look ran againthe prompt was edited
Hypotheseseach remaining failure mode has an experiment written before any number, and it passed the validity gateexperiment.yaml passes aqven check
Exploreseries on working cases, one change between series, until the change is done and the question is frozena signal looks good
Confirmone series on held-out cases, announced first in one line: the claim, why now, what each verdict means; verdict.text quotedthe series ended
Applythe flow changed, the cases the change fixes are kept with a regression tag, a look over them is clean, the decision is written downthe winning variant exists
Decidea new round on fresh cases, or a report with the findings, decisions, spend and remaining risksthe experiments allowed so far ran out

These look like progress and aren’t:

  • A clean check. aqven check proves the wiring and a simulated run. It says nothing about what a real model does with a real input.
  • One run. One run proves one case. Two runs, one that extracts every line item of an invoice and one that misses half of them, prove nothing about line items in general.
  • A signal. A series on working cases gives numbers to steer by, never a finding. A good number there is a reason to confirm, not a result.
  • A provisional verdict. A verdict read while the series is still running can flip before it ends.
  • A taxonomy without traces. Failure modes named before anyone read a trace test failures the flow may not have.

“Stage” also means the part of the flow you are building now: one step, a few steps, or the whole flow. How many parts a flow has, and of which kind, is a design choice: one step on the whole input, steps in sequence, or readings in parallel are options to compare with an experiment, not a default shape. Whatever the shape, the metric of a part comes from what its output feeds and which error costs more there:

What the part’s output feedsThe costly errorMain metric
a later step that can drop a wrong item but never recover a missed onea missrecall on labelled positives, completeness
a person or a system that acts on every item it getsa false alarmprecision, agreement with labels
a later step that must tell cases aparta field that carries nothingin a 10-case pilot each field varies across the classes the next step must tell apart; then the next step’s accuracy with it
a caller bound by time or pricea slow or costly passlatency_p95_ms, cost_of_pass, with quality as a guardrail
the caller of the flowmissing the contract’s “done”the contract’s check and number

Write the current part, what its output feeds and its “good” at the top of the project’s research journal, and check it before every held-out series. A threshold for the whole flow, put on one part, can fail a part that did its job: judge a part by the errors it passes on that nothing downstream fixes. Each part you add earns its place by its own experiment against the flow without it.

An intermediate output, such as the facts a reader pulls from an invoice or the turns it marks in a call transcript, is done when the step after it can use it. A field that takes one value on more than 80% of the pilot cases carries nothing, and a field the next step needs to separate two confusable answers must be there.

Stop and report when one of these holds:

  1. Done. Every “done” criterion of the contract is confirmed on held-out cases, and the regression look is clean.
  2. Saturated. A new round of exploration on fresh cases finds no failure mode seen at least twice, and the known modes are closed by a finding or a structural guard.
  3. Flat. Two rounds in a row improved neither quality, cost_of_pass nor p95 without making another one worse.
  4. Out of data. The launch plan recommends more cases than you can get. The answer stays “unclear”, and a structural guard goes in instead.
  5. Out of budget. The spend of the rounds reached the limit agreed with the owner.

Stop before any of those, and ask the owner of the flow, when:

  • traces wait for their notes: the first ones, offered to the owner, or failures that fit no known mode;
  • a step would go past a scope limit the owner set (“keep it simple”, “only this input”, “not yet”), or it needs a new data source, a bulk download or research nobody asked for;
  • a series waits for approval of its spend;
  • variants trade quality against cost, and the choice is a product decision;
  • “done” turned out to be unmeasurable or contradictory;
  • a failure mode needs human labels (tone, usefulness) that no truth by construction can give;
  • the owner asks to skip a stage: name its cost in one line, and let them choose.

Working on your own under an instruction like “go on, don’t ask”, keep to cheap steps inside the task. At the number of experiments the owner allowed, stop, report and update the journal.

The contract of a flow, written before the simplest flow, in the experiment.md of the first look:

MARKDOWN
## Contract
- Purpose: flag customer messages that ask for a refund, for the refunds queue.
- Input: the message text and the order id, if any; both reach the flow in production.
The customer's order history does not, so it is not an input.
- Output: asks_refund (yes or no) and the sentence that asks, if any.
- Done: flag_matches_label above 0.90 on messages that ask, and above 0.95 on messages
that don't, confirmed on held-out cases.
- Budget: $0.002 per run; the project spend cap per series.
- Latency: under 3 s per message.
- Now: building the detector; the queue routing comes later.
  • Check each stage’s exit criterion before you move on, and name which one is not met yet.
  • Treat a clean check, one run, a signal and a provisional verdict as steps, never as results.
  • Write the input contract with the rest of the contract; a column only your data has is not an input.
  • Name the part of the flow you are building, what its output feeds and its metric before you pick a threshold.
  • Offer a new part or another shape as an option with the experiment that would decide it, not as the next step.
  • Stop on one of the five conditions, and stop to ask when a decision belongs to a person.