34. Evaluation

Decision

DriftEvaluatorEvaluator is LychD's singular evaluation jurisdiction: it evaluates how an exact subject revision meets or departs from a declared goal. It defines Cases and Trial SuitesApp BundlesApp Bundles, captures observations, applies versioned Rubrics, reports uncertainty, and returns bounded findings. It does not execute unsafe payloads or own TombSandboxSandbox; select AnimatorModel RuntimeModel Runtime/capability; authorize spend, publication, repair, or deployment; define Persona; mutate PatternWorkflowWorkflow, CompositionAppApp, or artifact; admit training; or promote a SoulstoneLocal ModelLocal Model. Execution, DispatcherTraffic ControllerTraffic Controller, Toll, SpellweaverWorkflow EngineWorkflow Engine, MirrorPersona KeeperPersona Keeper, Smith, SoulforgeModel TrainerModel Trainer, and HitLApproval StepApproval Step keep those effects. Evaluation is evidence offered to policy, never policy disguised as a score.

The declared goal

Before a trial, its requesting owner states expected and forbidden behavior in the Case and criteria, thresholds, and missing-evidence policy in the Rubric. Applicable authority, constraints, and permitted effects remain part of the trial conditions. DriftEvaluatorEvaluator evaluates that declared contract. Its findings cannot revise the goal or authorize a way of reaching it.

Deviation from the goal may be categorical, qualitative, or measured along separate axes. A first trial needs no earlier version or time series: its reference is the declared goal. Baseline/candidate comparisons and repeated trials show how that relation changes. An unchanged failure remains a failure; a large behavioral change may be an improvement. Missing evidence establishes neither success nor a measured distance from the goal.

A revised goal requires revised Cases or Rubrics and new Outcomes. Earlier findings retain their original contract; the subject cannot move the acceptance criteria after seeing the result.

Trial contract

Changed subject, prompt, schema, Rubric, Evaluator, or Environment creates a new Outcome; it cannot rewrite an earlier result. Libraries implement this port but do not own evidence or routing. DriftEvaluatorEvaluator first retains what was observed: process exits, files, rows, tool requests, admitted effects, resource measurements and provider receipts. A criterion match, quality grade, attribution claim or judge score is a judgment over those observations and must remain separately identifiable. Model self-report is output under test. Mechanically observable receipts outrank textual similarity, and missing evidence stays missing. ShadowWhat-if SimulatorWhat-if Simulator may isolate candidates and TombSandboxSandbox may execute them; each contributes typed observations to this judgment.

trial status: completed | subject_error | harness_error | evaluator_error | blocked
claim verdict: PASS | PARTIAL | FAIL | CONTRADICTED | UNKNOWN | DISPUTED

Unavailable dependency is not subject failure; a hidden validator precondition is a harness/state contract candidate. Refusal on an impossible task is not success unless matched solvable controls show action when action is possible.

Adversarial evidence and calibration

Adversarial Cases pressure a boundary: forbidden authoritative requests, false premises, demanded certainty amid missing evidence, contradictions/impossible completion, repeated nudges, recoverable distortion, and attempted tool/memory/identity/completion claims beyond supplied evidence. Matched positive and negative controls record pressure round/order, recovery, over-refusal, truthful non-completion, and downstream contamination separately—not a mutable “integrity” scalar. MagusYou, the OperatorYou, the Operator dialect perturbations are still test data with provenance, scope, and release rules.

A Trial SuiteApp BundleApp Bundle declares repetitions and stopping rules before it runs. It retains distributions, trial order, applicable random seeds, blocked and error trials, and exclusions. Non-deterministic Evaluators are calibrated against labelled controls and known ambiguity. Qualitative work records independent agreement and disagreement where the judgment warrants it.

An LLM judge is a declared, bounded Evaluator. Its prompt, revision, inputs, lineage, calibration and conflicts belong to the Environment; hidden chain-of-thought is never required. Sealed Cases and holdouts protect against tuning. Leakage, duplicates, unstable harnesses and changes in Evaluator behavior invalidate the claims they undermine while leaving independently supported claims intact.

Evaluating a workflow change

Workflow improvement applies this trial contract to an exact score, prompt, schema, ContextBriefing PackBriefing Pack policy or implementation revision. The claim must include relevant trajectory evidence, failures and human burden as well as terminal artifact quality. It pins baseline and candidate, matched budgets, controls, uncertainty and stopping rules before comparison. Observations and qualitative judgments remain distinguishable.

Cached evidence can support only its unchanged input and evidence closure. Regrading a retained artifact creates a judgment over that artifact; it cannot establish that a changed workflow produces the same trajectory. Changed actions or relevant state require fresh admitted trials. The owner may adopt, reject or investigate the resulting candidate under its authority, without rewriting earlier Outcomes or a live ScrollWorkflow VersionWorkflow Version. A later training hypothesis requires separate Soulforge admission and protected holdout evidence.

Historical exploration replay

DriftEvaluatorEvaluator may evaluate a versioned exploration strategy against an immutable collection of recorded branch histories. The Environment pins the histories, observation-disclosure and transition rules, cost accounting and stopping conditions. The subject strategy sees only observations revealed by its preceding replay decisions; hidden descendants and their outcomes remain unavailable. An absent continuation stays unknown and cannot become an invented observation or a witnessed failure.

Such an Outcome measures strategy behavior on that recorded search space. It does not establish that changed requests, dependencies, timing or parallel execution would reproduce those outcomes. Keeping the baseline among candidates can prevent a worse selection on the same replay objective and history; it gives no guarantee for later histories or live work. Effectiveness outside replay requires fresh admitted trials, protected evaluation and independently observed state. Report represented execution costs separately from actual replay, strategy-development and validation costs. The operating passage shows how to prepare that comparison.

Evidence of recursive improvement

A recursive-improvement claim must show reuse of a changed mechanism: exact baseline/candidate revisions, motivating evidence, adoption decision, and a later improvement attempt that actually used it. Showing effectiveness requires a further comparison. Compare the successors produced by original and revised mechanisms from comparable starting subjects and evidence, under matched total budgets and independent evaluation. Keep transfer results, regressions, uncertainty, and stopping decisions alongside improvement across rounds. A better task artifact alone does not establish a better mechanism for future improvements.

Development feedback remains separate from protected assessment. Revising an Evaluator creates a separately calibrated candidate and new Outcomes; the subject cannot redefine its own acceptance criteria. Ouroboros connects these claims to the owning creation and adoption passages.

Capability claims and routing

DriftEvaluatorEvaluator may derive a scoped claim from a healthy Trial SuiteApp BundleApp Bundle. Each claim pins:

• AnimatorModel RuntimeModel Runtime, model, adapter, tool, and configuration revisions;
• task class, Cases, Rubric, Evaluators, and Environment;
• sample, controls, distribution, uncertainty, and noise;
• cost and latency per admitted success, including failures;
• creation, expiry, and evidence references.

There is no universal rank: accuracy, latency, VRAM, cost, restraint, and tool behavior are distinct policy-valued axes. DispatcherTraffic ControllerTraffic Controller may consume fresh admitted claims only after WardSecurity DeskSecurity Desk, compatibility, availability, privacy, and authority construct an eligible set. Missing/stale evidence leaves the DispatcherTraffic Controller's documented deterministic readiness order in force rather than inventing an intelligence floor. Dispatcher selection owns that order, including the treatment of stale or erroneous cached observations. It selects one candidate; failure of that selection does not retry another candidate. Toll may use the same measures in spend policy without making one local or frontier win universal routing authority.

Evaluation before and after training

SoulforgeModel TrainerModel Trainer defines a training proposal's expected change, baseline Outcomes, holdout evidence, and unacceptable regressions. Post-training evaluation uses that contract or makes every change visible; training-facing improvement cannot promote a candidate. DriftEvaluatorEvaluator returns evidence without selecting a corpus or registering a model. Passing an identity/behavior Trial SuiteApp BundleApp Bundle grants no Persona, SigilID BadgeID Badge, tool, or privileged route.

Returning findings across a Composition Suite

An exact, version-pinned Composition Suite may return a consumer consequence as evidence. This requires the member CompositionAppApp and PatternWorkflowWorkflow revisions, handoffs, failing observation, Rubric, Evaluator, Environment, verdict and uncertainty, and the declared artifact and evidence dependencies. Returning evidence does not reverse execution or merge member rows, secrets, SigilsID BadgesID Badges, approvals, policies or effect authority.

DriftEvaluatorEvaluator follows declared dependencies backwards to the smallest cut supported by the evidence. The nearest producer is not automatically the cause. Reuse requires the same complete input closure, artifact revisions, Rubric, Evaluator, relevant Environment and evidence contract. A failing consumer alone cannot condemn shared artifacts.

Missing lineage, flakiness, contagion, capture or rival explanations may leave the verdict UNKNOWN or DISPUTED. They can justify at most a broader bounded trial; they cannot justify reconstructed history or convenient blame.

The returned records authorize no spend, publication, deletion, training, promotion or mutation. SpellweaverWorkflow EngineWorkflow Engine may admit a new forward InvocationJob RequestJob Request under ordinary policy and HitLApproval StepApproval Step. Earlier RunsJob RecordsJob Records and Outcomes remain in its lineage.

Consequences

Acceptance evidence

DriftEvaluatorEvaluator remains Designed until one versioned Trial SuiteApp BundleApp Bundle with controls distinguishes subject/harness/ evaluator failure, reproduces an Outcome, calibrates each non-deterministic Evaluator, preserves raw evidence/uncertainty, and proves routing/repair consumers reapply their own policy. State of WorkWhat Actually WorksWhat Actually Works, not this ADR alone, records promotion.