Trials

In plain words: Evaluation trials and verdicts

Designed 3 min read 5 glossary terms 15 connected pages
The short version written for the Snayu team

How an evaluation trial is pinned before it runs (cases, suite, rubric, evaluator, environment), how observations are recorded before judging, and the status and verdict vocabularies used in results.

Why read it: Read this when designing test cases, LLM-as-judge setups or adversarial evals, so results stay comparable and honest.
  1. 1Status: completed, subject_error, harness_error, evaluator_error, blocked. Verdict: PASS, PARTIAL, FAIL, CONTRADICTED, UNKNOWN, DISPUTED.
  2. 2Every trap case needs positive and negative controls; repetition and stopping rules are fixed in advance.
  3. 3An LLM judge is a calibrated evaluator with limits, never an oracle; model claims of success are under test.

Designed No trial path runs yet.

Words on this page, decoded click one for its card

The original text Underlined words open their plain-English card.

A subject refuses an impossible task. Beside it, a matched control offers a solvable version with the same surface pressure. Refusing both reveals blanket timidity; distinguishing them may reveal judgment. DriftEvaluator observes both before drawing a conclusion.

The declared goal includes the distinction between those conditions: honest refusal may meet one Case while useful completion meets the other. The goal must therefore be expressed in expected and forbidden behavior, not inferred from whichever answer the subject produced.

Trials are Designed; no trial path runs. State of Work records delivery; ADR 34 owns the accepted trial law.

Pin the trial

A trial pins its inputs and evaluation contract before execution. Case and Rubric express the declared goal; Environment fixes the conditions under which it will be judged:

Record Pinned meaning
Case Exact input, fixtures, expected and forbidden behavior, oracle, effect class, and stop condition
Trial SuiteApp Bundle (TrialSuite@1) Versioned Cases, matched controls, ordering, repetition policy, and aggregation rules
Rubric Criteria, verdict vocabulary, thresholds, missing-evidence policy, and revision
Evaluator Kind, identity, revision, independence, calibration evidence, and declared limitations
Environment Subject revision, prompts, tools, dependencies, hardware, harness, state, budgets, and relevant policy

The resulting Outcome records observations, measurements, verdicts, uncertainty, errors, cost, latency, and retained evidence.

A Trial SuiteApp Bundle groups Cases and evaluation controls. A Composition Suite is a versioned graph of separately owned CompositionsApps and typed handoffs. Changing the subject, prompt, tool schema, Rubric, Evaluator, or Environment produces a new Outcome rather than revising the old trial's answer.

Observe before judging

First retain what happened: exit status, files, database rows, tool requests, admitted effects, resource measurements, and provider receipts. Evaluation then applies criteria, grades, attribution, or a declared model judge to those observations. Where an effect can be observed directly, its receipt carries more weight than text that merely resembles a successful answer.

ShadowWhat-if Simulator may hold candidate worlds and TombSandbox may execute an admitted unsafe payload; DriftEvaluator owns neither isolation nor execution. An LLM judge is a declared, calibrated Evaluator with recorded limits, never an oracle. Hidden chain-of-thought supplies no evidence. Model prose about its own success is an output under test.

trial status   completed | subject_error | harness_error | evaluator_error | blocked
claim verdict  PASS | PARTIAL | FAIL | CONTRADICTED | UNKNOWN | DISPUTED

Read status and verdict as answers to different questions. Status says whether execution completed; verdict says what the retained evidence supports. An unavailable dependency or hidden-validator condition may block the trial or expose a harness or state-contract failure. The declared Case and Environment determine which; neither condition automatically establishes a subject error.

Accordingly, blocked does not mean FAIL. Missing evidence yields UNKNOWN, while unresolved conflict within the retained evidence or among rival Evaluators yields DISPUTED.

Pressure, repetition, and coverage

Adversarial Cases test responses to requests for forbidden methods, false premises, missing information, impossible constraints, repeated nudges, recoverable dialect distortion, and unsupported claims about tools, memory, identity, or completion. Every trap receives positive and negative controls. The Outcome records pressure round, order, recovery, over-refusal, truthful non-completion, and downstream contamination separately.

Repetition and stopping rules are declared before results are inspected. Outcomes retain distributions, order, seeds where applicable, blocked attempts, errors, and exclusions. Evaluators are calibrated against labelled controls and known ambiguity; independent disagreement remains visible.

Sealed Cases and holdouts resist tuning leakage. Coverage binds an exact Rubric to an immutable subject revision and retained evidence. Mention, textual similarity, or a nearby receipt cannot fill an uncovered criterion. One dramatic success or failure does not become certainty.

Healthy Outcomes may proceed to Capability claims. Findings that must return to an owning boundary proceed through Returning findings.