aqven
Use cases

What AQVEN is for.

Five situations from building multi-step AI workflows, and what you and your coding agent do about each in AQVEN.

Get startedGitHub

Who does what

You

You set the direction, read the evidence, agree what counts as a failure and decide what ships.

Agent

Your coding agent edits the workflow files, writes cases and experiments, and lines up runs for you to read.

AQVEN

AQVEN checks the files, runs flows and series, writes the verdicts and keeps the findings.

Find1 / 5

Find what you'd miss by hand.

“What am I missing?”

The situation

The workflow almost works. But you hand-picked a few examples, and you can't say how it behaves on inputs you haven't tried.

triagegemini-2.5-flash-liteoutput.retries: 2
types/records/observation.yaml · value: maxLength 200
try 1observations[].value over 200 charsrefused
try 2observations[].value over 200 charsrefused
try 3observations[].value over 200 charsrefused

MODEL_RETRIES_EXHAUSTED: counted as schema_invalid, a failed attempt

Triage on gemini-2.5-flash-lite wrote more than the 200 characters its type allows, on all three tries.

What you do in AQVEN

  1. 1

    Hand your agent a question.

    You

    Say what the flow is for, what counts as done and how much it may spend.

  2. 2

    It writes cases, you name the failures.

    AgentYou

    Cases span risks such as length or channel. You read the failing runs first and agree the failure modes.

  3. 3

    Test the riskiest one.

    AgentAQVEN

    The agent writes an experiment, metric first. AQVEN runs it on working cases: a signal, not yet a finding.

What it won’t do

The agent can tell a plausible story, and nothing in the engine checks that you read the traces.

Read more

Explain2 / 5

Explain where a run went wrong.

“Why did this fail?”

The situation

The final answer fails a check. The cause could be the context, one step, the model, a contract between steps or the infrastructure. Rewriting the last prompt is a guess.

1
collect_orders3 orders read from queue
2
classify_intentcategory: refund, confidence: 0.41
3
route_to_queuesent to: escalations

One card per step, with what it received and returned. Read back from the output.

What you do in AQVEN

  1. 1

    Open the run in Studio.

    You

    A Failed steps panel lists every failed execution in plain words, with its error code and a hint.

  2. 2

    Walk back to the first failure.

    You

    Click a step: what it read, the prompt as sent, what came back. Later failures often follow from the first.

  3. 3

    Test the suspicion.

    YouAgent

    Rerun just that step with a changed prompt. Then an experiment checks the fix beyond the one run you saw.

What it won’t do

A trace shows where a failure appeared, not why.

Read more

Compare3 / 5

Compare changes on your own cases.

“What should I change?”

The situation

A new model came out, or the current version costs too much and passes one day, fails the next. You want variants compared on your own cases.

collect_orders$0.00 · 12ms
classify_intent$0.02 · 840ms
route_to_queue$0.00 · 3ms

Each finished step logs its model, time, tokens and cost. The expensive one stands out.

What you do in AQVEN

  1. 1

    Write the comparison down first.

    AgentYou

    One experiment, one factor. The metric and your margin go in the file before any data.

  2. 2

    Run both on the same cases, several times.

    AQVEN

    Every variant runs every selected case, with repeats. The series pauses near your spend cap.

  3. 3

    Read quality, cost and latency side by side.

    You

    Each metric gets a value and a 95% interval. Filter to the cases where variants disagree and open their runs.

What it won’t do

It measures your criteria on your cases, not a leaderboard, and doesn't check that variants got an equal budget.

Read more

Confirm4 / 5

Confirm a fix before you ship it.

“Can I trust this change?”

The situation

Your agent proposed an improvement, and the examples you work with pass. That can be a real effect, or a fix tuned to the cases it saw.

promises above 0.80, margin 0.02
gpt
0.91
mistral
0.89
holdout: confirmed
angry_refund_requestdatasets/route_ticket_cases.yaml
regression: yes

A held-out verdict against a margin set before the data, and the case that broke, tagged regression.

What you do in AQVEN

  1. 1

    Explore on working cases.

    Agent

    Every dataset is split in half. On the working half the agent iterates freely and gets signals, never findings.

  2. 2

    Confirm once on held-out cases.

    AQVEN

    When the question is frozen, one series runs on the held-out half, cases the agent never sees one by one.

  3. 3

    Read the verdict AQVEN writes.

    AQVENYou

    Confirmed, refuted or inconclusive, from the 95% interval and your margin. It lands in FINDINGS.md with its scope.

What it won’t do

A green verdict is not a production guarantee: it holds for the cases, checks and versions it measured.

Read more

Build5 / 5

Build it so it stays readable.

“How do I keep it readable as it grows?”

The situation

Your coding agent builds a new AI process fast. A few weeks in, the prompt, the routing and the logic are spread across code nobody on the team can follow.

$ aqven checkflows/support_case/nodes/triage/triage.node.yaml:8:3: error E_REF_MISSING in[0].from: reference $prepare.out.messages: the value has no field messageserrors: 1, warnings: 0

Real output from the docs: one typo in a binding, reported with file, line and rule before any model call.

What you do in AQVEN

  1. 1

    Flows, nodes and prompts as typed files.

    Agent

    A flow is a folder in your repo. Prompts live in Markdown files, and every change shows in your diff.

  2. 2

    aqven check before a run.

    AQVEN

    It validates every file and binding, then simulates each flow with a stand-in model. No network, no API key.

  3. 3

    Your agent edits the same files.

    Agent

    Claude Code, Codex or Cursor connect over MCP. Structural changes go through flow_patch: every file lands, or none does.

What it won’t do

A passing check means the structure holds together, not that the logic is right.

Read more

Start with one question.

Run a small experiment. Inspect the evidence. Decide what comes next.

GitHubGet started