Stasis and return
In plain words: Pausing and resuming workflows
How a workflow pauses (Stasis) and comes back: live waits for hardware keep the worker, durable waits for approval or a delegated job save a checkpoint and re-queue later, and finished Runs never return.
- 1Hardware waits stay live, capped at eight convergence calls per Run and three in a row per capability.
- 2Approval or delegate waits checkpoint state; return gets a fresh queue key and reads stored truth, not the payload.
- 3Missing checkpoint fails as 'stasis lost'; once terminal status commits, the Run cannot be revived.
Partial Local runs available; real restart proof for checkpoint plus approval is still missing.
Words on this page, decoded click one for its card
Pause Stasis
A job safely paused, waiting for a model to load, a human approval, or another agent.
It's like: Putting a phone call on hold. Live hold keeps the line open; durable hold means "we will call you back".
Click a dot to open that part's card.
Between the pages, a ribbon holds the place where the rite fell quiet.
A pause must retain both the owner of the wait and the boundary through which work can return. If that continuity breaks, the IntentRequest is never rerouted or restarted. ADR 28 owns this law.
Three boundaries
| Boundary | State movement | Custody | Return |
|---|---|---|---|
| Live hardware wait | RUNNING → AWAITING_HARDWARE → RUNNING |
Resident GhoulWorker; OrchestratorStage Manager owns readiness | Same worker hop; no queue re-admission |
| Durable Gate or delegate wait | RUNNING → AWAITING_CONSENT or AWAITING_DELEGATE |
Checkpoint and exact wait owner | Fresh enqueue and worker claim, including in the same VesselEngine Room |
| Terminal RunJob Record | DONE, FAILED, or CANCELLED |
Canonical ledger | None |
A Gate or DelegatedAgentNode makes its PatternWorkflow durable, but hardware
waiting stays Live: the worker remains resident without a capability lease. A RunJob Record may make up to
eight hardware convergence calls in total and three consecutively for one capability. A further HardwareTransitionRequired fails before another OrchestratorStage Manager call. The
initial ordinary node execution is not a counted hardware convergence request.
Live and Durable
Stasis owns the distinction.
Durable parking ends the current worker execution while the VesselEngine Room may keep running. If the process also dies, Reanimation reconciles the committed RunJob Record, queue, checkpoint, and wait owner in its replacement. Whole-body snapshot restoration is a separate Restoration rite; consult its implementation evidence before relying on it.
What crosses the threshold
The consent or job identity exists before GraphRunner snapshots. The worker then commits the exact
wait owner and status without terminal DONE. Consent has released the Agent lease before its
Gate, and return reads the durable verdict before requesting another grant.
A checkpoint is Run-keyed GraphStep Map history. A NodeSnapshot carries typed GraphStep Map state,
next node, and execution status; an EndSnapshot carries final state and GraphStep Map result. Live
grants, secrets, runtime dependencies, provider handles, events, and AgentJob truth stay
elsewhere. It is declared state, neither process image nor effect receipt.
Every workflow state that can request hardware carries a typed hardware_resume_budget in that
checkpoint. Its total and per-capability counters therefore survive a durable park and a replacement
GraphRunner; reconstructing a runner cannot reset them. Checkpoints written before this field decode
with zero counters and cannot prove attempts that happened before the upgrade.
Memory returns defensive copies. The Postgres adapter replaces one JSONB history document per RunJob Record, but no real Postgres consent-plus-checkpoint restart receipt proves that path.
Return makes a new claim
Either verdict may win AWAITING_CONSENT → QUEUED; the resumed hop reads stored truth, never an
enqueue payload. Delegate return accepts only the terminal AgentJob owning the current wait.
The successful return allocates a fresh monotonic enqueue sequence, from which its Run-specific queue key derives. Status change and pending delivery commit together. If publication fails, startup or the runtime relay can publish that exact queued hop; failure neither recreates the wait nor permits reuse of a possibly escaped key. Duplicate verdicts and callbacks remain inert.
Bridge admits one approval call per model round. Resumed rounds may chain, bounded to three.
Multiple approvals in one response create no consent row; they settle DONE with an honest
bottleneck.
A missing checkpoint fails exactly as stasis lost. An invalid document fails the claimed hop
with validator text, not a stable public code. Pinned-manifest mismatch, including
checkpoint-schema identifier mismatch, fails as pinned Pattern unavailable; Pattern
lifecycle owns compatibility, migration, and refusal. Process death during
RUNNING or AWAITING_HARDWARE never guesses a replay: startup recovers only an exact first-node
ConsentApproval Step or delegate park, otherwise contains correlated effects before failing the RunJob Record.
An exact pending or already-decided ConsentApproval Step survives the pre-park startup window; decided parked consent is re-fired. The equivalent exact delegated checkpoint is parked, and startup also refreshes each durable delegated wait and re-admits it only when its exact owning job is terminal. Delegated agents owns that boundary.
Truth closes first
Once terminal status commits, the RunJob Record cannot return, even if checkpoint cleanup fails. The worker settles its exact delivery before releasing ContextBriefing Pack and attempting cleanup, so retained stasis cannot revive it. Ghouls follows the terminal-event and cleanup sequence under Graph's commit law.
Cancellation fences RunJob Record claims and delivery rotation while the parent broker job, delegated jobs,
and ConsentApproval Step authority are contained. Only acknowledged containment permits CANCELLED; failure
leaves honest, retryable CANCELLING truth. Completion that already won makes cancellation a no-op.
Every effectful station needs its own idempotency identity, completion receipt, cancellation rule, compensation or refusal path, and illegal-repeat boundary. A checkpoint cannot settle an uncertain external effect.
Topology-A local runs are Available. Graph Stasis and consent re-admission and delegated execution are Partial. Real checkpoint-plus-Consent/delegate restart proof, distributed fencing, and general periodic workflow recovery remain absent. Workers owns custody; Spellweaver routes the subsystem.