Training Run

The metal remembers the fire.

A Training RunJob RecordJob Record turns admitted training inputs into candidate bytes while keeping custody and evidence explicit. It receives a Training IntentRequestRequest, immutable Dataset Manifest, exact base-model digest, objective, and Recipe. Its result is a frozen Candidate Bundle with a receipt showing what execution produced. The whole passage is Designed; no Training RunJob Record path runs. State of Work owns delivery, and ADR 33 owns execution law.

Admit the strike

Entry requires the immutable Dataset Manifest produced by Corpus Admission and Compilation, base-model and tokenizer/config digests, the objective, Recipe, and Training IntentRequestRequest with resource envelope. The trainer receives these exact inputs. It cannot expand the corpus, pull ambient traces, or alter them after admission.

Request the resources this run actually needs: exact accelerators, memory, storage, network policy, duration, and conflict-domain requirements. OrchestratorStage ManagerStage Manager uses fresh observations and the compiled conflict graph to determine which existing work is affected. Training has no universal priority, and the operator may postpone or refuse it. If a physical transition is required, only the affected leases drain; unused devices stay undisturbed.

Orchestrator owns hardware readiness, admission closure, affected-lease drain, conflict resolution, convergence, and restoration. Corpus meaning and trainer success remain outside that authority.

The trainer receives only admitted secrets and network access, and writes to fresh Run-owned locations. A container or CovenModel GroupModel Group label cannot prove isolation: filesystem, credential, network, process, and resource controls must enforce it and leave evidence. Local and remote execution share the same custody bar. Local work has no privacy exemption; remote work records egress, provider custody and retention, network and secret handling, returned artifacts, and receipts.

Pin the evidence

A Training RunJob RecordJob Record pins input digests, trainer and dependency revisions, environment, randomness, Recipe-defined precision, budgets and stop conditions, resource observations, logs, checkpoints, terminal status, and artifact digests. A friendly name never replaces a digest.

At completion, the trainer emits candidate bytes and its receipt. Freezing them into a Candidate Bundle binds the base digest, model or adapter bytes, tokenizer/config changes, Dataset Manifest and Recipe lineage, trainer receipt, and compatibility claims. An adapter must remain tied to its exact base and compatible runtime, so its bytes cannot be presented as an independent model.

Trainer loss, telemetry, development improvement, and trainer-authored samples remain diagnostics. The job cannot replace a serving revision, enlarge its resource envelope, or promote its output; independent evaluation begins only after custody freezes.

Settle failure

Cancellation or failure preserves terminal truth, logs, completed checkpoints, and quarantined partials for diagnosis. A partial never becomes a Candidate Bundle. Retry receives a new Training RunJob RecordJob Record identity and receipt rather than appending to or overwriting history.

If a resource transition fails, OrchestratorStage ManagerStage Manager restores the exact prior world before admission reopens; otherwise uncertainty stays visible and contained. Process exit alone is not clean recovery.

The passage ends with either a frozen Candidate Bundle and receipt ready for independent handoff, or exact failure or cancellation truth with every partial quarantined.