ExLlamaV3 through TabbyAPI

In plain words: Engine: ExLlamaV3 via TabbyAPI

Operator validation 1 min read 3 glossary terms 4 connected pages
The short version written for the Snayu team

How LychD runs ExLlamaV3 through its TabbyAPI server as a dynamically loaded local model, using Tabby's lifecycle API to load models and verify the active one.

Why read it: Read this when setting up ExLlamaV3/TabbyAPI-backed models.
  1. 1The container starts without a model, so capabilities start as activatable, not warm.
  2. 2Only the Chat Completions surface is supported; responses-surface declarations are rejected.
  3. 3API and admin keys are mandatory and stored as Podman secrets.

Operator validation Contracts tested; real GPU/model receipt still needed.

Words on this page, decoded click one for its card

The original text Underlined words open their plain-English card.

ExLlamaV3 is the inference library; TabbyAPI is its official OpenAI-compatible server. LychD keeps that server outside the daemon dependency graph and registers it as the dynamic animator/exllamav3 SoulstoneLocal Model runtime.

The generated sample pins the TabbyAPI image by digest. The container starts without a model, so its declared capabilities become ACTIVATABLE, not WARM. LychD sends the declared model directory and backend through Tabby's lifecycle API; context, cache, split, and reserve remain TabbyAPI defaults unless a model-local tabby_config.yml overrides them.

Keep the identity used by LychD separate from the directory TabbyAPI loads. A stable [[models]].id names the LychD model; its path must name a validated direct child of the mounted model directory. The Connector takes that path's basename as Tabby's runtime name and translates between them for both lifecycle control and data calls.

The current LychD TabbyAPI adapter executes the Chat Completions surface. A model declaration with capabilities.surface = "responses" is rejected at RuneService Config File validation because that surface is outside this adapter's admitted contract.

Authentication is mandatory even inside the unpublished private pod. The two distinct API and admin keys remain Podman-secret material and are never placed in browser code or ordinary RuneService Config File fields. Core binds only the managed container VesselEngine Room; the former bind --uncaged option is retired.

Follow model loading through the whole POST /v1/model/load server-sent-event stream, then verify the active model with GET /v1/model. LychD performs both steps because a disconnected client cannot establish that a detached load failed. An indeterminate stream keeps the result contained for reconciliation or reset through the owning VesselEngine Room boundary.

Focused contracts cover the runtime, control plane, connector, registration, authentication, and containment behavior. The named GPU/model/runtime receipt remains operator validation.

See ExLlamaV3, TabbyAPI, and Soulstone Disciplines.