Native audio input#

The optional audio component accepts speech and other sounds as native model input. CUDA validates the source, decodes PCM, resamples, builds log-mel values and runs the trained audio encoder. The language decoder receives the projected audio rows at each source reference. No speech transcript or generated sound description replaces those rows.

The supported pair is Qwen2-Audio-7B-Instruct with its F16 audio encoder. The language file uses Q4_K_M weights. Base text conversations and the separate image component remain available.

Before starting#

Prepare the exact audio pair and a build with enough weights, feature storage and KV pages. A combined image/audio store needs both parent models and separate requests for each parent. Use the gateway for authenticated WAV uploads.

The supported workflow checks source ownership, model routing and recovery. Model replies remain subject to transcription and sound-classification errors.

Prepare the model store#

Add the audio language file to manifest.jsonl with role language-audio. Create audio.jsonl with exactly two entries:

1. Copy that language entry from manifest.jsonl.
2. Add the matching encoder entry with role audio.

Each entry requires name, role, path, source, revision, license, SHA-256 and byte count. The parent entry must match exactly. The two names, paths and digests must differ. The component reader checks all 489 tensor shapes, types and extents before allocation.

Set the explicit chat wrap described in model files. Use the Qwen im_start and im_end message spans, without thinking spans. The end IDs are 151645 and 151643. The generation head is <|im_start|>assistant followed by a newline. Select the audio parent at startup:

aotx_boot --models STORE --roles language-audio,embedding --journal JOURNAL

A combined store can select language,language-audio,embedding and include the image pair. Each request keeps its model, vocabulary, chat wrap and native feature width. An audio reference selects language-audio, including references in retained context. A prompt that requires both supported media parents is refused. Use separate conversation slots for requests that require different parents. Raw trained audio markers are refused without resolved sources.

Store-wide voice, steer and probe assets belong to the default language model. They do not apply to a separate specialist model through matching token IDs or vector widths. The optional per-agent affect state and temperature controls remain independent of those assets. When audio is the default language model, its compatible store assets can apply to it.

Submit audio#

Use the terminal or the local attach interface:

audio load 0 private wav /absolute/path/sound.wav
audio load 0 private pcm16 48000 2 /absolute/path/samples.pcm
audio load 0 private f32 16000 1 /absolute/path/samples.f32

Raw samples use little-endian byte order. Channels are interleaved. The remainder of the command is the pathname; spaces need no quotes. The file reader rejects symlinks, nonregular files and files changed during transfer. It reads bytes on the runtime host. It does not interpret samples.

The console gives a source reference and transfer ID. A ready notice means that native rows are available:

audio: ready [audio:SHA256] transfer TRANSFER_ID
say [audio:SHA256] Describe the sound briefly.

Replace SHA256 with the displayed 64 lowercase hexadecimal characters. Several audio references can appear in one prompt, in order. The decoder receives the trained audio markers and rows; the source digest and pathname are omitted. Missing, malformed, retired or inaccessible sources refuse the complete prompt. A valid source that is still processing keeps the prompt waiting.

The private, room and shared scopes have the same rules as image sources. Without a cognitive binding, private belongs to the addressed base conversation slot. An active sequence keeps a feature lease until it ends. Cancel an owned transfer or retire an unused source:

audio cancel 0 TRANSFER_ID

Audio adds these hexadecimal refusal statuses to the common media statuses:

The zero-signal refusal is exact. It is not a speech detector or a quiet-audio accuracy guarantee. Model replies can be incorrect, including sound labels and speech transcriptions.

Audio formats#

WAV input supports RIFF/WAVE PCM16 and IEEE F32, including validated extensible PCM/F32 headers. Supported rates are 16000, 44100 and 48000 Hz, with one or two channels. The parser checks the complete declared file extent, chunk padding, sample alignment and format fields. It refuses duplicate data chunks, inconsistent lengths, unsupported encodings and nonfinite samples.

Each source can contain at most 30 seconds. A source must produce at least one pooled audio row. This requires more than 320 canonical samples at 16000 Hz. Longer recordings can be submitted as separate sources. Microphone drivers, compressed audio codecs, video decoding and network retrieval are outside this interface. A producer can submit captured samples through the common media ring.

The raw transport source begins with a 32-byte header:

All fields are little-endian. The exact sample bytes follow the header. The digest covers the header and samples. The raw file commands supply this header automatically. The common begin record uses format 3 for WAV or 4 for raw PCM; width and height are zero.

Stereo channels are averaged. Resampling operates over the complete source timeline. Transport chunks do not restart the filter or spectral window. The encoder preserves valid input lengths and masks padded attention keys. Each source produces at most 750 rows, each containing 4096 F32 values. These rows count toward the conversation token and KV limits.

Conversation capacity#

The supplied conductor role has a 160-page quota; the worker role has 32 pages. Audio prompts can need more pages than these quotas permit. On the 12g profile, increase the quota for each unbound conversation that needs it:

agent 0 pages 640
agent 1 pages 640

Bound conversations use the page quota in their cognitive binding. Each request must fit both that quota and the shared physical pool.

Select each required language role at startup. The model load ROLE NAME command can replace an active base language role. It cannot add a language role whose vocabulary, working memory and decoder are absent.

Device capacity and recovery#

The common media profile bounds source descriptors and encoded bytes for both media types. Audio features and worker storage have separate capacities:

An audio feature row costs 16384 bytes. Model weights and worker buffers require additional VRAM. Capacities are independent. Increasing source slots does not allocate one encoder workspace per slot.

Requests reserve their complete prompt and reply cache requirement before decoding. An audio runtime reserves its configured physical KV pool at startup. Startup refuses allocations that do not fit; it does not silently offload a model or active memory.

An optional audio.profile selects capacities within the build limits. It contains exactly 64 bytes:

All fields are little-endian. An omitted directory profile selects the build defaults. Complete runtime files include the audio pair, audio profile and common source profile. The required audio flag is 4 in the runtime index feature field. Activation checks the selected parent, exact component bytes and supported capacities.

Combined 12 GiB configuration#

The reference configuration holds the audio pair, Qwen3.5-0.8B Q8_0 image parent and Qwen3-Embedding-0.6B Q8_0. It uses the matching F16 vision component and one workspace per media component. Configure the shared cache before building:

cmake -S . -B build -DAOTX_KV_POOL_PAGES=384
cmake --build build --parallel

Create both capacity profiles in the prepared model store. Replace /path/to/store with that store's absolute path:

from pathlib import Path
import struct

store = Path("/path/to/store")
media = struct.pack("<8sIIQIIIII4xQ24x", b"AOTXIM01", 1,
    128, 67108864, 16384, 1, 4194304, 8192, 4096, 8388608)
audio = struct.pack("<8sIIII40x", b"AOTXAU01", 1, 8192, 1, 1440000)
(store / "media.profile").write_bytes(media)
(store / "audio.profile").write_bytes(audio)

These profiles use 64 MiB for shared source bytes, 16384 image rows and 8192 audio rows. The 384 physical cache pages reserve 768 MiB. This profile was checked with one active media conversation in a 64-slot build. It does not establish simultaneous media capacity for every slot.

Set each conversation page quota for its intended prompt and reply size, as described above. The complete runtime packager includes both profiles. Available VRAM must cover the complete runtime and other device allocations.

Source recovery#

The GPU retains active source bytes and feature rows. The drain writes the recovery copy behind it. Recovery rebuilds native audio rows from the recorded canonical bytes before new requests use them. Source retirement frees GPU capacity but preserves historical source records in the recovery log. The complete file cap, publication costs and durable-ready rules still apply.