Skip to content

How to run experiments and series as an agent

Write an experiment file, start a series with series_start, wait for its verdict with series_get, read its outputs with series_outputs, and stop one with series_cancel.

You’re connected to a project over MCP and need a number you can act on: whether a failure stays rare enough, whether a change or another agent is better, whether a cheaper agent is not worse. The question lives in an experiment file, experiments/<experiment_id>/experiment.yaml. A series runs that experiment live: every selected case, for every variant, repeats times. Every attempt is an ordinary run with its own trace, and every attempt calls the models and costs money. Five tools cover it: series_start, series_get and series_cancel run and follow a series, and series_outputs and series_list read what series produced. How an agent takes a task to a reliable flow is the order to use them in, round after round.

  • Write the experiment first, then run aqven_check. There is no MCP tool that creates an experiment: it is a file, and you write it like any other, one file at a time, or a person starts it with New experiment in Studio (How to write an experiment). Give it a failure_mode and a falsifiable description, and one factor in varies: the variants set only its values. aqven_check validates the subject and its range, the factor and every variant’s values (agents, prompt files, alternative nodes, flows), each variant assembled and compiled like a flow, the dataset and its tag filter, the checks and the fields they read, the metric names and validated_by. A series refuses to start on a project with errors, or when a case can’t run for a variant. It fails with NOT_RUNNABLE and one problem per case and variant, before a single model call.
  • series_start takes exactly one of experiment_id or look (below), plus on (dev or holdout, dev by default) and, optionally, cases, repeats (at most 20), cap_usd and a client_op_id. It returns at once with the series summary and the launch plan: attempts, the expected interval half-width, the recommended number of cases and the cap. It has no price: what a series costs depends on the models and is known only from its attempts as they finish. Sending the same client_op_id again returns the same series instead of starting a second one.
  • The status after series_start is running or awaiting_approval. A series starts by itself, under the project spend cap (research.spend_cap_usd in aqven.yaml, $1.00 by default; a local override on the project server wins on that computer) or the cap_usd you pass. Only a cap_usd above the project cap waits for a person before anything runs. While it runs, a series whose spend reaches 90% of its cap starts no new attempts, lets the running ones finish and waits in awaiting_approval with pause.reason spend_near_cap and pause.spent_usd. There is no MCP tool for approving: tell the user what it spent, and they continue it with a higher cap in Studio or with POST /api/series/{series_id}/approve. There is no plan-only tool either: the launch plan is information. A person who wants it first reads it in Studio’s Launch panel, or from POST /api/experiments/{experiment_id}/launch-plan.
  • series_get takes a series_id and wait_seconds (0 to 50). With wait_seconds above 0 it holds the answer until the series is done, cancelled, failed, awaiting_approval or waiting_human, or until the time runs out, and then returns the current snapshot. Call it again until the status settles. The snapshot carries progress, spend against the cap, the per-variant matrix with 95% intervals, and the verdict. eta tells when an active series should finish, from the attempts finished per minute over its last 5 minutes of running time: state is estimating until 3 attempts finished after the first one, running with attempts_per_minute, remaining_seconds and finish_at, and paused while it awaits approval or a person. It is null once the series ends. Pass the time left on to the user as an estimate. spend.unpriced_attempts counts the attempts that ran on a model without a known price: when it is above 0, spend.usd is a lower bound, so say so. A series is failed only when every attempt hit an infrastructure error, and then error names the first one.
  • Read the summary first. series_get with view: "summary" answers in a few KB however many attempts ran: per variant the finished, passed, failed, errored and running attempts, the spend, p50 and p95 latency, the primary metric with its 95% interval, and the top 3 failure groups by error code or failed check (check:<id>), each with one example message. The full view grows with the number of variants: 23 variants take about 64 KB in full and about 18 KB as a summary. fields keeps only the named metrics and checks in either view; a name the series doesn’t have fails with INPUT_INVALID and the list of names it has.
  • Quote verdict.text as it is once the status is done. The server writes that sentence from the interval and the margin declared in the file. Repeat it; don’t round the numbers or put them in your own words. While a series is still running, its verdict is provisional.
  • include_cases: true adds per-case rows to series_get, but only for dev cases: failing cases first, case_limit rows a page (50 by default, up to 200), and next_cursor for the next page passed back as cursor. The rest, held-out cases included, are counted in hidden_cases: include_cases never shows a held-out case one by one, so a change can’t be tuned to it. Each row gives every variant’s tally and failed checks, and each attempt its outcome, error and run_id: open that run with run_get and run_events to find the first node that failed. divergent is true when the variants’ pass rates on that case differ, for example 3 of 3 repeats against 1 of 3; with one variant it is always false. The rows grow with the number of variants and attempts, so read view: "summary" first and ask for the rows only when a variant has failures.
  • series_outputs reads what the attempts on working cases produced, in bulk. One row per attempt, in attempt order: case, variant, repeat, split, outcome (passed, failed, error, waiting or running), error_code, cost_usd, latency_ms, run_id, output, node_outputs and checks ({check_id: value}). Filter by split, variant, case and outcome; page_size goes up to 200, and next_cursor comes back as cursor. Without fields each row carries the whole flow output. With fields, a JSON pointer such as /label puts {"/label": value} in output, a node id such as triage puts that node’s whole output in node_outputs, and triage/summary a path inside it; a path that isn’t there is null. The rows come from the same run records run_get reads, so use this, not one run_get per attempt, and never the files under .aqven/. Outputs of map items, branches and loop passes aren’t included: read those with run_get_node.
  • series_outputs never returns held-out attempts. Like include_cases, it leaves them out: a series on held-out cases gives no rows, and split: "holdout" is refused with REQUEST_INVALID. Held-out cases are read only as totals, through series_get (view: "summary" or the aggregates of the full view). A change tuned to held-out outputs makes the next held-out series measure the tuning, not the flow. The REST route and aqven series export leave them out too; only the owner can ask for them there (include_holdout, --include-holdout), to review a finding, never to tune.
  • series_list is the history, with totals. Newest first, filtered by experiment_id and status: series_id, experiment_id, flow_id, status, the verdict state, on, progress, spend_usd, the times and eta. Its stats total every series of the experiment, or of the project without experiment_id: series, finished attempts, requests (attempt runs plus judge check runs, not single model calls), tokens, spend_usd and wall_seconds in which at least one series was open.
  • A failure counts, an infrastructure error doesn’t. An attempt whose model broke its output type even after its retries (MODEL_RETRIES_EXHAUSTED, MODEL_SCHEMA_MISMATCH, invalid JSON) is a counted failure, like a provider refusing the output type as too complex (OUTPUT_SCHEMA_REJECTED), a failed check or a refusal. A missing provider key, a provider error, a timeout, a stalled stream (MODEL_STREAM_STALLED) or broken code is an infrastructure error. It stays out of the metrics, and above 5% of the attempts it makes the series invalid. An attempt that ends on a rate limit (provider_error, HTTP 429) runs once more at the end of the series before it counts. See How a series decides.
  • series_cancel takes a series_id and an optional reason. Queued attempts never start. Model calls already running finish and are paid for. No finding is written. Cancelling a series that has already stopped fails with SERIES_STATE_CONFLICT.

The server splits every dataset in half by a hash of the package name, the dataset id and the case name, the same way in every clone of the project. A second dataset built from the same inputs, under another id, splits them anew, so an input you explored on in one dataset can land in the held-out half of the other: see The same population in both halves. Iterate on dev as often as you need: a series on dev gives at most a signal, never a finding. Run holdout once, when the change is done and the question is fixed. A series on holdout whose verdict is anything but invalid writes experiments/<experiment_id>/findings/<series_id>.yaml once and regenerates FINDINGS.md at the module root. Both are generated files, so don’t edit them; aqven check reports an edited finding as E_FINDING_TAMPERED and a stale FINDINGS.md as W_FINDINGS_STALE.

VerdictMeaning
confirmedthe 95% interval clears the bound, or beats the baseline, by more than the declared margin
refutedthe effect is within the margin or reversed
inconclusivethe interval is too wide to decide; below_mde means more cases would decide it
signalthe series ran on dev, or the deciding check is a judge without validated_by
invalidcancelled, stopped by the spend cap, inputs changed during the series, more than 5% infrastructure errors, or no data

To see what a flow does on a few named cases, pass look instead of experiment_id: {"look": {"flow_id": "support_case", "dataset_id": "support_case_cases", "case_names": ["strip_flicker_credit"]}}. Each named case runs once, with no verdict and no finding, up to 500 cases. start_node and end_node narrow the runs to a range of top-level nodes, and the nodes above the range take their outputs from the case. A look runs the named cases on whichever side of the split they are. include_cases still shows only the working ones and counts the rest in hidden_cases.

In the showcase project, reply_overpromise_risk asks whether the polish range of support_case keeps its promises check above 0.97 with a margin of 0.01. Before starting it on two working (dev) cases, repeated twice, look at the launch plan series_start would return. Studio asks the same question with POST /api/experiments/reply_overpromise_risk/launch-plan and the body {"on": "dev", "cases": 2, "repeats": 2}. The response below is real, trimmed, from AQVEN’s example project lumen with no series history and no provider key, on a machine without network. The showcase template is the same project under your package name, and the split hashes the package name with the dataset id and the case name, so your count of working cases can differ by one or two:

JSON
{
"on": "dev",
"cases": 2,
"repeats": 2,
"variants": 1,
"attempts": 4,
"available": 6,
"half_width": 0.24352108850049436,
"margin": 0.01,
"recommended": {
"cases": 1187,
"repeats": 2,
"reason": "short_of_cases",
"text": "about 1187 cases are needed for a half-width within the 0.01 margin, but only 6 are available"
},
"below_recommended": true,
"needs_approval": false,
"project_cap_usd": "1.00",
"project_cap_source": "project",
"cap_usd": "1.00"
}

The plan has no dollars: what an attempt costs depends on the models it calls, so the series learns it from its own attempts as they finish, and the cap keeps the total in check. project_cap_source says where the project cap came from: project for aqven.yaml, as here, override for a local override, default when neither sets it. No cap_usd was passed, so needs_approval is false and the series cap is the project cap: series_start with the same on, cases and repeats starts the series as running at once, with a cap of $1.00, and every attempt calls the model live. Without a provider key those attempts fail as infrastructure errors, and the series ends failed with the missing key named in error.

The recommended.text is the warning to pass on: with six working cases, no series of this experiment can settle a margin that narrow. If a series is already running, series_cancel stops it: queued attempts never start, the status becomes cancelled, and the verdict is invalid with the reason cancelled. A second series_cancel on the same series fails with SERIES_STATE_CONFLICT.

The same series from a terminal is aqven series reply_overpromise_risk --cases 2 --repeats 2. It starts the project server if it isn’t running, then waits and prints the progress with the time left, and the verdict. It exits with 0 when the series is done, whatever the verdict, and with 1 when it was cancelled or failed. It exits with 3, printing a Studio link, when the series waits for approval or paused near its cap, and with 4 when an attempt waits for a person. When it loses contact with the server it retries for up to 5 minutes, then exits with 5, naming the error; the series keeps running on the server. --cap sets the series’ own cap, and --json prints the final state as one JSON line. aqven series export <series_id> --format csv --fields /label triage writes the same rows as series_outputs, one per attempt on working cases, as JSON lines or CSV. How to run a series lists every flag and exit code.