Reanimation

In plain words: Crash recovery after restart

Partial 4 min read 14 glossary terms 16 connected pages
The short version written for the Snayu team

Explains Crash Recovery (Reanimation): how a new server process reads the database after the old one died and decides which jobs may resume. It separates this from pausing for a model switch and from full-system restore.

Why read it: Read this before restarting a running installation or changing recovery logic.
  1. 1Three different things: Pause for model switch, Crash Recovery after process death, and full Restore (still Designed).
  2. 2There is no graceful drain command proving zero loss; stop submissions and let work settle before restarting.
  3. 3Never hand-edit generated systemd units; change config and run Apply Config (lychd bind).

Partial Restart recovery exists; real systemd/Podman proof and full restore are pending.

Words on this page, decoded click one for its card

The AssistantMemory VaultPauseEngine RoomCrash Recovery
Where it sits in LychD

Crash Recovery Reanimation

After a crash or restart, the new process reads the records and decides what can safely continue.

It's like: A new shift reads the handover notes before touching anything.

Click a dot to open that part's card.

The original text Underlined words open their plain-English card.

Read the ashes before you call the dead by name.

ReanimationCrash Recovery brings a new VesselEngine Room to the records committed before its predecessor died. Systemd can raise the process; the PhylacteryMemory Vault, queue, checkpoint, and wait owners determine which work may return. Volatile memory, subscribers, leases, and uncommitted frames are gone.

A durable park can return through a fresh worker claim in the same VesselEngine Room. ReanimationCrash Recovery is needed only when the application process has died.

Three rites that must not be confused

Need Rite Continuity boundary
A model or VRAM transition during live work Live StasisPause The resident RunJob Record releases its lease, waits for OrchestratorStage Manager convergence, then resumes itself.
A new application process after VesselEngine Room death ReanimationCrash Recovery A new process reconciles committed truth before admitting work.
An earlier whole body, including database, code, and configuration Restoration Coordinated snapshot, freeze, restore, and reconciliation remain Designed.

AAn VesselEngine Room restart neither substitutes for a model transition nor stops every SoulstoneLocal Model or clears VRAM. Snapshot State keeps the whole-body boundary separate.

Perform one bounded reanimation

Begin only after the four Awakening observations agree. ReanimationCrash Recovery cannot complete a host's unfinished first life. Run the commands below from the stable Summoning checkout, under the same operator user and XDG environment.

Before restarting, stop submissions and let Bridge-visible work settle. Active work must either reach a supported durable boundary or have its loss accepted. There is no complete RunJob Record census or graceful drain command proving zero-loss restart; if unseen work or continuity matters, do not restart. State records the current status and stop limits. Direct Systemd action bypasses OrchestratorStage Manager admission and lease drain.

The generated body

lychd bind and the Scribe project CodexSettings Folder intent into owned units atomically. Edit that intent, never paste or hand-edit a generated Quadlet. The generated VesselEngine Room has a restart policy and a migration gate before start; real systemd/Podman embodiment still needs operator validation.

When CodexSettings Folder or unit intent changed, bind first. Otherwise restart the existing projection:

# Only when Codex or generated-unit intent changed:
uv run --extra postgres-binary lychd bind --dry-run
uv run --extra postgres-binary lychd bind

systemctl --user restart lychd-vessel.service
uv run --extra postgres-binary lychd status
systemctl --user show lychd-migrate.service \
  --property=Result --property=ExecMainStatus
uv run --extra postgres-binary lychd logs services --lines 200

The current stop cannot stop an active VesselEngine Room without the authenticated lifecycle port. status establishes unit and mount truth; the migration unit must report Result=success and ExecMainStatus=0. Neither proves model warmth. Repeat NexusModel Status Board readiness and a benign same-host Bridge reply. Disagreement among state, readiness, and reply returns the investigation to The Awakening.

Reanimation — a new Vessel judges durable truth

Before publishing its substrate or AltarControl Panel services, the VesselEngine Room connects runs and rites, constructs services, warms the registry, synchronizes standing policy, and reconciles durable state. Required PostgreSQL recovery must finish cleanly; failure or degradation aborts startup. The memory profile has no cross-process truth and remains best-effort.

  • AWAITING_CONSENT remains parked. A pending verdict waits. A verdict committed while the process was down is re-fired; one admission atomically creates its next delivery and later reads durable verdict and checkpoint. The pre-park crash window is recoverable only when the first resumable snapshot binds this RunJob Record and its exact latest ConsentApproval Step id, with that owner still non-cancelled. An already-decided owner is parked and then re-fired.
  • QUEUED remains Run-ledger-owned. Its exact RunDelivery says fresh versus resume, queue, priority, and publication state. HELD is refused because initiating context never became publishable. Current-boot work is retained; a proven pre-boot active generation is terminally fenced and re-probed; an absent job is republished under the same key; a terminal broker record rotates to a fresh sequence without changing delivery mode. Missing or mismatched delivery truth, unprovable active ownership, queue absence, or probe failure makes recovery degraded rather than inventing an outcome.
  • Previous-process RUNNING and AWAITING_HARDWARE do not replay. Startup may recover the exact first-node ConsentApproval Step or delegate pre-park boundary; otherwise it contains correlated effects and records FAILED / ghoul lost. Checkpoint deletion and one sequence-correct durably drained terminal event follow.
  • AWAITING_DELEGATE refreshes its exact owner. Only a terminal owning AgentJob may re-admit it. Missing coordination or owner identity fails required PostgreSQL startup.
  • Missing resume checkpoints fail as stasis lost. The GraphStep Map never restarts from its first node or original IntentRequest.
  • CANCELLING still owes containment. Startup continues the elected cancellation; a broker or delegate whose containment remains uncertain cannot be declared terminal merely to open the AltarControl Panel.

Startup repairs missing terminal evidence from canonical terminal RunsJob Records. Conflicting terminal evidence instead aborts required PostgreSQL startup; recovery does not overwrite it. Cancellation cleanup may still remove a checkpoint if evidence fails. Ghouls explains terminal ordering. Conversely, a checkpoint surviving failed cleanup cannot revive a terminal RunJob Record.

Lifespan-owned delivery, consent, and delegated-wait relays continue repair after startup. They retain degraded keyset pages while scanning newer owners. There is no periodic workflow scheduler, generic automatic SAQ retry, public failed-Run retry, same-boot worker-failure custody watchdog, or transactional Step/event outbox.

The disposable two-boot factory receipt proves a narrower PostgreSQL lifecycle with offline collaborators. It does not prove a real Consent-plus-Checkpoint restart. Graph stasis and consent re-admission remain Partial, and BridgeChat accepts only one approval call per model round.

Read the ashes

A pending consent card is a preserved wait. A decided consent back in QUEUED has won a resume hop, but still awaits a worker claim. FAILED / ghoul lost identifies work that crossed death without a supported durable boundary. A surviving or republished SAQ job preserves queued labor, not the old in-memory event stream.

Retain the host, unit, image, database, queue, RunJob Record, shutdown, and recovery observations before promoting the result to an operator receipt.

Inscribe the breath before it passes beyond recall.