37. Audio

Status

Audio admission is Partial. The current v1 compatibility spine can carry immutable audio ArtifactRef metadata, express audio input through modalities_in, and declare stt and tts capability families. Media-type projection to input modality remains Designed. Those labels prove schema only, not a working transcription or synthesis call.

A pure EchoVoiceVoice inspector also checks supplied bytes against an exact artifact reference and the bounded PCM subset below. The separate explicit local Reliquary adapter can retain supplied WAV bytes in quarantine. A trusted EchoVoice consumer reruns the inspector, requires separate admission, and reads under current authority. These unregistered helpers supply no capture or speech path.

An explicit engineering probe can inspect and send canonical WAV to pinned local audio.cpp or whisper.cpp transcription bindings, retain attributed transcription or generated audio.cpp WAV, and optionally capture a finite utterance through a separately invoked Linux recorder helper. Explicit Horizon and companion lab entries can select bounded local playback after capture, with owned player-group settlement and distinct player-exit facts. These helpers issue no native speech grant or playback receipt. Bridge does not forward audio modality into dispatch. No credential-backed audio admission, speech timeline, streaming transport, native STT/TTS grant, selectable EchoVoice extension, or Audio CovenModel GroupModel Group ships. State of Work owns this boundary and the local custody limits.

Decision

EchoVoiceVoice owns the speech lifecycle. Its first profile, record and send, begins with one explicit, bounded utterance. The utterance becomes an immutable artifact before an admitted provider returns text or another audio artifact.

There is no audio capability family. Audio is material; current stt, tts, and chat are v1 compatibility families, while the Designed general-service path uses exact interfaces, profiles, operations, and call/session grants.

Engine, model, protocol, and language

Speech integration keeps four axes separate:

A shared file format or OpenAI-compatible route does not make an unknown model architecture loadable. Engine support means that the exact family has a loader, preprocessing, decoding, and tested execution path. Likewise, two servers exposing /v1/audio/transcriptions or /v1/audio/speech may accept different fields and provide different streaming or cancellation semantics; each admitted endpoint declares and proves its dialect.

The first planned local engine profile is audio.cpp behind the bounded OpenAI audio batch subset. It is one initial runtime choice, not a new universal audio abstraction and not evidence that every audio.cpp family works. Its RuneService Config FileService Config File must pin the runtime revision, model family, checkpoint and digest, backend and device, declared and verified languages, license, and the exact stt or tts behavior admitted by a bake. The engineering qualification probe may explicitly select a separate pinned whisper.cpp transcription binding while retaining audio.cpp synthesis. Each operation owns its endpoint, runtime revision, dialect, model identity and language policy; a common HTTP route does not merge their provenance. Existing single-runtime profiles retain their original identity. This additional probe binding grants no native Ear or Voice admission and permits no automatic runtime fallback. EchoVoiceVoice owns the current candidate models and promotion criteria; native delivery remains Not yet.

Language is neither a host-global switch nor a property inferred from the engine. An explicit request language wins, then the admitted Principal or Persona preference, then model-supported detection when policy allows it. A model may route a language only after declaring it; a verified language additionally passed the relevant LychD bake. STT detection never silently authorizes translation, and TTS may not substitute another language or voice when the selected output locale is unavailable. Translation is a separate semantic act.

AltarControl PanelControl Panel interface locale is independent of this speech policy: a Slovak interface may capture an English utterance and request a Japanese Voice, provided each exact step is admitted. Musical lyrics, singing, arrangement, and musical acceptance belong to Riffmaw rather than EchoVoice; they reuse the same no-global-language and no-silent-translation law without turning a song generator into TTS.

Language Edition owns an attributed timed-media language edition: aligned source and target words, casting, performance direction, dialogue timing, caption fit, and acceptance of a constrained language-track replacement. It may place STT, Translation, and TTS or human-performance steps in one PatternWorkflowWorkflow, but that orchestration does not make EchoVoice the application or Translation an ambient feature. Language Edition may alter only declared language tracks against a locked master; Broadcast retains editorial recutting, final assembly, and publication.

Capture and custody

An explicit engineering qualification probe may exercise the selected local runtime before a native Ear or Voice is admitted. Its fixed literal-loopback dialect, model identity, bounded inputs and outputs, private external workspace and cancellation limits must be explicit. Such a probe issues no AnimatorModel RuntimeModel Runtime grant, registers no ambient provider or extension, and cannot establish BridgeChatChat audio admission, Resonance Session delivery or remote-egress authority. It exists to measure the exact binary/model pair and supply evidence for those later boundaries. The local qualification contract owns its operation.

Capture authority is explicit, visible, time-bounded, and revocable. Consent to one utterance is not permanent microphone access; device indicators and server state must agree whether capture is armed, active, stopped, or failed.

Voice cloning is not implied by TTS support. A cloning request separately binds the authorized speaker or source, reference-audio custody, permitted words and use, model and license, and the derived voice artifact. A convincing result grants no Persona, Principal, or attribution authority.

Reliquary custody binds recording or synthesis identity, digest, media type, byte size, validated duration/codec facts, classification, Principal, retention, and derivation from recording, transcript, or request. Bytes never enter a GraphStep MapStep Map checkpoint, queue payload, event envelope, or log—only references and bounded timeline state do. Decoders and transcoders treat them as hostile: format, duration, channels, sample rate, decompression, parser resources, and metadata are bounded before provider use.

Bounded PCM inspection

EchoVoiceVoice's first local material-inspection contract accepts supplied immutable bytes and an existing ArtifactRef. It checks the exact byte length and SHA-256 digest against that reference; it neither retrieves the artifact nor establishes custody or permission to use it.

This first subset is deliberately exact: audio/wav, a 44-byte RIFF/WAVE header with a 16-byte PCM fmt chunk immediately followed by one data chunk, signed 16-bit little-endian samples, one channel, and 16,000 frames per second. The RIFF length, data length, byte rate, block alignment, and sample width must agree with the actual bytes. Accept 1 through 480,000 complete frames (up to 30 seconds) and no more than 1 MiB in the supplied artifact. Refuse ancillary chunks, extended headers, compression, trailing bytes, truncation, empty data, and other WAV variants; there is no implicit conversion or resampling.

A successful inspection returns immutable source and PCM facts. It establishes only that these bytes match this reference and this structural subset. Silence is valid PCM; inspection proves neither speech, origin, consent, semantic eligibility, model compatibility, nor an executed transcription. Construction of a facts record alone does not prove inspection. This boundary starts after a caller has obtained bytes; it does not bound an upstream upload or retrieval.

Record, send, account

explicit capture → immutable source artifact → eligible Ear or audio-capable Mind
→ attributed transcript or native observation → ordinary Agent step → optional Voice
→ delivery receipt

A transcript retains source, available time regions, provider and revision, language assumptions, and uncertainty; it interprets a recording rather than replacing it. Synthesized audio is a new artifact. “Generated” says bytes were produced, not that a client received, buffered, played, or completed them.

When speech is required, the admitted PatternWorkflowWorkflow's finish policy declares the required result, the evidence needed to settle it, and any explicitly configured accepted fallback. Failure or missing evidence at that required boundary produces partial or non-completion under the pinned score unless an explicitly admitted fallback is fulfilled. Preserved text does not fulfill a required speech obligation; text-only completion is permitted only when that fallback was explicitly admitted. An observed playback completion establishes the reported playback boundary, not that the person heard or understood the response.

The PatternWorkflow supplies exact interface/operation, material facts, classification, priority, and authority. The Dispatcher chooses an exact Ear, Voice, or Mind profile; current v1 can express only the stt, tts, or chat compatibility projection. For an otherwise eligible managed binding that is not WARM, DispatcherTraffic ControllerTraffic Controller returns HardwareTransitionRequired; the requesting RunJob RecordJob Record enters GraphStep MapStep Map StasisPausePause while Orchestrator converges readiness, then re-dispatches.

In current source, even WARM v1 stt and tts declarations fail closed at grant issue because neither has a typed call surface. Readiness convergence cannot turn those labels into Ear or Voice.

EchoVoiceVoice may not preempt, revoke another grant, make its own continuing session, or silently select a remote provider. A PortalCloud ModelCloud Model needs egress eligibility, consent where required, and a cost bound. Local execution still requires capture, retention, and tool authority; source influence persists under Context privatization.

Streaming without pretending a socket is a protocol

Record-and-send establishes the minimum passage. Half-duplex can then stream one response after capture closes. Full-duplex requires more: simultaneous capture and playback, VAD, barge-in, echo cancellation, and contested turn ownership must all be accounted for.

Voice-activity detection answers whether speech is present; it does not prove that an activation phrase was spoken. The first profile therefore uses visible push-to-talk. A future wake-word backend belongs to the Listener and emits a bounded activation event; it is neither an stt AnimatorModel RuntimeModel Runtime nor ambient permission to retain or transcribe surrounding audio.

A Resonance Session keeps one monotonic timeline for segments and attempts: capture armed/active/stopped/failed; transcription and synthesis in progress/settled/generated; delivery offered/accepted/failed; playback started/completed/interrupted/unknown; and cancellation, expiry, or retry. WebSocket, WebRTC, or another transport can carry these records but supplies none of their semantics. Reconnect resumes by acknowledged sequence and artifact identity, never by guessing from a live socket.

Cancellation names its scope: stopping playback need not erase an artifact, revoking capture stops new input, and cancelling ana InvocationJob RequestJob Request settles every admitted provider and transport attempt. Closing local result adoption or a transport connection does not prove that provider execution stopped. After submission, retain execution ownership until completion or an owned stop and reconciliation proves settlement; uncertain execution cannot authorize a retry or release a compute reservation. Late output may settle execution while remaining ineligible for adoption. A future Companion client can own physical capture, playback, route selection, and low-level device state; a private Tether still supplies transport, not application authority, classification, or consent.

Application-owned live audio

EchoVoiceVoice's Resonance Session owns speech chronology. Applications retain their own records over that chronology: Riffmaw may own a musical performance session and clock overlay, while Language Edition may own a language-edition timeline over exact speech attempts. Neither duplicates the capture, synthesis, delivery, or playback ledger. Every live route still follows Audio custody, transport, interruption, and uncertain-playback law; EchoVoice does not absorb every application that carries sound.

Avatar may use an authorized exact voice profile or derived artifact inside one profile or Morphe. Avatar can narrow purpose, audience, words, disclosure, retention, and revocation, but cannot create speaker authority, copy a Persona, widen the cloning grant, or claim that synthesized speech was played. EchoVoice retains those lifecycle facts. Language Edition may cast the same eligible reference for one exact language edition, but owns only role fit, pronunciation, timing, disclosure, and performance acceptance; it cannot widen either Avatar's presentation envelope or the underlying performer and voice-use authority.

Monitoring, recording, retention, analysis, transformation, remote transmission, and machine response are independently admitted scopes. A visible armed input proves none of the others. A live route binds an exact route and clock epoch, monotonic and sample-frame mapping, sequence, scheduled window, absolute deadline, cancellation fence, and output acknowledgement. Late or stale frames are rejected locally after expiry, disconnect, cancellation, or epoch change; a restarted device, peer, model, or plug-in requires reconciliation and a newly admitted live InvocationJob RequestJob Request.

The realtime callback never waits on a model, disk, network, control-plane action, or unsafe plug-in operation. Declared direct or otherwise safe monitoring, output limiting, feedback protection, safe bypass, and immediate local stop remain available without model cooperation. Model and planning work occurs outside that callback and may emit only through the deadline fence. PCM, MIDI streams, device handles, and plug-in state never enter GraphStep MapStep Map checkpoints or events; only artifact references, bounded sequence state, epochs, and receipts cross the durable seam.

A tempo or phase protocol does not establish audio transport or a shared sample clock. A remote audio link does not establish recording consent. Each application records mappings, clock-rate divergence, resampling, loss, jitter, latency, discontinuity, and uncertainty rather than naming two devices “synchronized” by assertion.

Consequences and acceptance

Independent Ears, Voices, Minds, and Listeners can compose because EchoVoiceVoice keeps generation, delivery, and playback distinct. Doing so requires sensitive custody, storage and retention, and explicit handling of timing, cancellation, and egress races.

Bounded material inspection can be accepted through actual byte-parsing and refusal tests without claiming a speech runtime. Usable record-and-send execution additionally requires evidence for visible capture consent, custody, hostile-audio limits at acquisition and decoding, modality forwarding, local and PortalCloud ModelCloud Model policy, STT/TTS conversion, transcript provenance, delivery/playback receipts, interruption, retention/deletion, and disconnect recovery.