Skip to content

Scanning the literature before you test

Before you rank hypotheses, look up what is already known about the riskiest assumptions — where to search, what to write down, how to mark what nobody has confirmed, and why the web is a source for the domain but never for the engine.

Many risky assumptions have been tested before by someone else: whether speech-to-text holds up on noisy phone calls, whether a model family reads small print on a scanned page, how a judge drifts, which public dataset has the labels you need. Spend a short scan on the riskiest assumptions before you design experiments. Write down each source with its date, the claim it makes and how strong the evidence is. Mark what nobody confirmed as unverified, and mark domain facts that need an expert. The web is a source for the domain and for methods. It is never a source for how AQVEN works: those facts come from the documentation of the version you run.

  • After error analysis, before you rank hypotheses. You know the failure modes, and you are about to decide which claim to test first. A known result can move a hypothesis up the list, or remove it.
  • When an assumption could kill the product. “The model finds the termination clause in a 200-page contract” or “the model reads the total on a crumpled receipt photographed with a phone” is worth an hour of reading before a day of series.
  • When you need labelled data. A public dataset with labels from its source beats any labels you derive.
  • When a result surprises you. Check whether others saw the same effect before you call it a finding.
SourceGood forWatch out for
papers and preprintsmeasured effects, benchmark numbers, known failure patternspreprints are not reviewed; a benchmark may not match your inputs
model cards and provider documentationinput limits (context window, file size, audio length, image resolution), supported features, output modesmarketing claims; a feature listed is not a feature proven on your task
public labelled datasetsreal inputs with labels from their sourcethe license text; how the labels were made; whether the population matches yours (vet it)
practitioner write-upsevaluation methods, error analysis, pitfallsone team’s experience; numbers without a method
domain referenceswhat a rule, a category or a term means in the fielddefinitions that differ between sources; facts that need an expert to apply

For every source, one short entry:

  • Link and date opened. A page can change after you read it.
  • The claim, in one sentence, with its number if it has one.
  • The evidence: measured on data, observed in a few cases, or only stated.
  • How it applies here: same kind of input, same model family, same population, or only similar.
  • What it changes: a hypothesis moved up or down, a new negative control, a dataset to use, a method for a check.

Keep the entries where the next reader looks: in the experiment.md of the experiment they shaped, and in the “Open questions” or “How we measure” section of the project’s research journal.

  • A claim you found but could not trace to data gets a plain UNVERIFIED: in front of it. It may still guide a hypothesis. It never becomes a fact in a report.
  • A domain assumption that needs an expert, such as which clauses make a contract high-risk, gets “needs expert sign-off”. Labels built from it carry the same mark until someone signs off.
  • A dataset without a license you can read is not used, however good its labels look.

Summaries of web pages lose details and sometimes invent them. That is tolerable for the domain, where you mark what you could not confirm. It is not tolerable for the engine you run: a binding syntax, a key of aqven.yaml or a node kind learned from a web summary may not exist in your version. Engine facts come from the documentation of that version and from aqven check, whose messages name the fix.

A scan entry in an experiment’s experiment.md, before the experiment on whether a speech-to-text step keeps intent detection accurate on recorded support calls:

MARKDOWN
## Literature
- <paper title and link>, opened 2026-09-24. Claim: word error rates of speech-to-text
models rise sharply on accented speech over a phone line. Evidence: measured on
a public benchmark. Applies: same kind of audio, other languages. Changes: the dataset
gets `accent` and `line_quality` tags, and the experiment compares native audio input
against a transcript step on phone-quality recordings, not studio ones.
- UNVERIFIED: <forum post and link> says a provider mixes stereo calls down to mono. Changes:
cases with one speaker per channel are added, to see whether the model still tells the
caller from the agent.
- <consumer protection guide and link>: which statements in a sales call count as a
binding promise in one country. Needs expert sign-off before the "promise made" labels
built from it count as ground truth.

The same entry shape works for any input: a paper on how long-context models lose facts in the middle of a long document, a provider page on how many frames per second it samples from a video, a benchmark of table reading on scanned forms.

  • Scan the riskiest assumptions after error analysis and before you rank hypotheses.
  • Write each source with its date, claim, evidence and what it changes.
  • Mark unverified claims and expert-dependent assumptions, and carry the mark into labels and reports.
  • Learn the engine from its own documentation and aqven check, never from web summaries.