Reanimation
In plain words: Crash recovery after restart
Explains Crash Recovery (Reanimation): how a new server process reads the database after the old one died and decides which jobs may resume. It separates this from pausing for a model switch and from full-system restore.
- 1Three different things: Pause for model switch, Crash Recovery after process death, and full Restore (still Designed).
- 2There is no graceful drain command proving zero loss; stop submissions and let work settle before restarting.
- 3Never hand-edit generated systemd units; change config and run Apply Config (lychd bind).
Partial Restart recovery exists; real systemd/Podman proof and full restore are pending.
Words on this page, decoded click one for its card
Crash Recovery Reanimation
After a crash or restart, the new process reads the records and decides what can safely continue.
It's like: A new shift reads the handover notes before touching anything.
Click a dot to open that part's card.
Read the ashes before you call the dead by name.
ReanimationCrash Recovery brings a new VesselEngine Room to the records committed before its predecessor died. Systemd can raise the process; the PhylacteryMemory Vault, queue, checkpoint, and wait owners determine which work may return. Volatile memory, subscribers, leases, and uncommitted frames are gone.
A durable park can return through a fresh worker claim in the same VesselEngine Room. ReanimationCrash Recovery is needed only when the application process has died.
Three rites that must not be confused
| Need | Rite | Continuity boundary |
|---|---|---|
| A model or VRAM transition during live work | Live StasisPause | The resident RunJob Record releases its lease, waits for OrchestratorStage Manager convergence, then resumes itself. |
| A new application process after VesselEngine Room death | ReanimationCrash Recovery | A new process reconciles committed truth before admitting work. |
| An earlier whole body, including database, code, and configuration | Restoration | Coordinated snapshot, freeze, restore, and reconciliation remain Designed. |
AAn VesselEngine Room restart neither substitutes for a model transition nor stops every SoulstoneLocal Model or clears VRAM. Snapshot State keeps the whole-body boundary separate.
Perform one bounded reanimation
Begin only after the four Awakening observations agree. ReanimationCrash Recovery cannot complete a host's unfinished first life. Run the commands below from the stable Summoning checkout, under the same operator user and XDG environment.
Before restarting, stop submissions and let Bridge-visible work settle. Active work must either
reach a supported durable boundary or have its loss accepted. There is no complete RunJob Record census
or graceful drain command proving zero-loss restart; if unseen work or continuity matters, do
not restart. State records the current status and
stop limits. Direct Systemd action bypasses OrchestratorStage Manager admission and lease drain.
The generated body
lychd bind and the Scribe project CodexSettings Folder intent into owned units atomically. Edit that intent,
never paste or hand-edit a generated Quadlet. The generated VesselEngine Room has a restart policy and a
migration gate before start; real systemd/Podman embodiment
still needs operator validation.
When CodexSettings Folder or unit intent changed, bind first. Otherwise restart the existing projection:
# Only when Codex or generated-unit intent changed:
uv run --extra postgres-binary lychd bind --dry-run
uv run --extra postgres-binary lychd bind
systemctl --user restart lychd-vessel.service
uv run --extra postgres-binary lychd status
systemctl --user show lychd-migrate.service \
--property=Result --property=ExecMainStatus
uv run --extra postgres-binary lychd logs services --lines 200
The current stop cannot stop an active VesselEngine Room without the authenticated lifecycle port.
status establishes unit and mount truth; the migration unit must report Result=success and
ExecMainStatus=0. Neither proves model warmth. Repeat NexusModel Status Board readiness and a benign same-host
Bridge reply. Disagreement among state, readiness, and reply
returns the investigation to The Awakening.
Reanimation — a new Vessel judges durable truth
Before publishing its substrate or AltarControl Panel services, the VesselEngine Room connects runs and rites,
constructs services, warms the registry, synchronizes standing policy, and reconciles durable
state. Required PostgreSQL recovery must finish cleanly; failure or degradation aborts startup.
The memory profile has no cross-process truth and remains best-effort.
AWAITING_CONSENTremains parked. A pending verdict waits. A verdict committed while the process was down is re-fired; one admission atomically creates its next delivery and later reads durable verdict and checkpoint. The pre-park crash window is recoverable only when the first resumable snapshot binds this RunJob Record and its exact latest ConsentApproval Step id, with that owner still non-cancelled. An already-decided owner is parked and then re-fired.QUEUEDremains Run-ledger-owned. Its exactRunDeliverysays fresh versus resume, queue, priority, and publication state.HELDis refused because initiating context never became publishable. Current-boot work is retained; a proven pre-boot active generation is terminally fenced and re-probed; an absent job is republished under the same key; a terminal broker record rotates to a fresh sequence without changing delivery mode. Missing or mismatched delivery truth, unprovable active ownership, queue absence, or probe failure makes recovery degraded rather than inventing an outcome.- Previous-process
RUNNINGandAWAITING_HARDWAREdo not replay. Startup may recover the exact first-node ConsentApproval Step or delegate pre-park boundary; otherwise it contains correlated effects and recordsFAILED / ghoul lost. Checkpoint deletion and one sequence-correct durably drained terminal event follow. AWAITING_DELEGATErefreshes its exact owner. Only a terminal owningAgentJobmay re-admit it. Missing coordination or owner identity fails required PostgreSQL startup.- Missing resume checkpoints fail as
stasis lost. The GraphStep Map never restarts from its first node or original IntentRequest. CANCELLINGstill owes containment. Startup continues the elected cancellation; a broker or delegate whose containment remains uncertain cannot be declared terminal merely to open the AltarControl Panel.
Startup repairs missing terminal evidence from canonical terminal RunsJob Records. Conflicting terminal evidence instead aborts required PostgreSQL startup; recovery does not overwrite it. Cancellation cleanup may still remove a checkpoint if evidence fails. Ghouls explains terminal ordering. Conversely, a checkpoint surviving failed cleanup cannot revive a terminal RunJob Record.
Lifespan-owned delivery, consent, and delegated-wait relays continue repair after startup. They retain degraded keyset pages while scanning newer owners. There is no periodic workflow scheduler, generic automatic SAQ retry, public failed-Run retry, same-boot worker-failure custody watchdog, or transactional Step/event outbox.
The disposable two-boot factory receipt proves a narrower PostgreSQL lifecycle with offline collaborators. It does not prove a real Consent-plus-Checkpoint restart. Graph stasis and consent re-admission remain Partial, and BridgeChat accepts only one approval call per model round.
Read the ashes
A pending consent card is a preserved wait. A decided consent back in QUEUED has won a resume
hop, but still awaits a worker claim. FAILED / ghoul lost identifies work that crossed death
without a supported durable boundary. A surviving or republished SAQ job preserves queued labor,
not the old in-memory event stream.
Retain the host, unit, image, database, queue, RunJob Record, shutdown, and recovery observations before promoting the result to an operator receipt.
Inscribe the breath before it passes beyond recall.