Supported architectures and model families#
AOTX 0.3.5 runs supported GGUF model files through its native CUDA backend. The current model paths cover dense Qwen3, Qwen2/Qwen2.5, Llama 3.2, Qwen3.5 hybrid models and OLMoE. Specific trained pairs provide image and audio input.
This page describes model architectures in the public development and test pre-release. For GPU hardware and build profiles, see support and qualification.
Read the support levels#
| Level | Meaning |
|---|---|
| Implemented path | The loader and CUDA runtime contain the required layer operations. |
| Checked file | A specific model conversion has recorded execution checks. The result applies to that file and configuration. |
| Qualified feature | An exact model, wrapper and processor meet the requirements of a specific feature. |
A family name does not qualify every size, conversion, fine-tune or task. The model file must also pass tensor, tokenizer, chat-wrap, memory-capacity and runtime checks. A catalog entry identifies a file; it does not establish runtime qualification.
Text model families#
These paths have been exercised with the examples below. The examples identify existing coverage; they are not a complete list of compatible files. Read model files for exact file identities and measured limits.
| Family | GGUF architecture and runtime path | Checked examples |
|---|---|---|
| Qwen3 dense | qwen3; attention with query/key normalization |
Qwen3-4B Q8_0 and Q4_0; Qwen3-1.7B Q4_0. |
| Qwen2 / Qwen2.5 | qwen2; attention with query, key and value biases |
Qwen2.5-0.5B-Instruct Q8_0. |
| Llama 3.2 | llama; attention without query/key normalization |
Llama-3.2-1B-Instruct Q5_K_M and Q8_0. |
| Qwen3.5 hybrid | qwen35; linear-delta layers and gated attention |
Qwen3.5-0.8B Q8_0. |
| OLMoE | olmoe; attention and expert feed-forward layers |
OLMoE-1B-7B-0924-Instruct Q4_K_M. |
Qwen3 dense support does not include Qwen3 mixture-of-experts models.
The current expert loader accepts the OLMoE routing rule only.
The hybrid loader accepts the qwen35 layer layout and checks its dimensions and tensor types.
The checked Qwen2.5 file does not give exact token agreement with the processor reference on every measured continuation. The release also retains unresolved numerical accuracy comparisons. Execution support does not imply numerical parity with another backend or correct answers to every task.
Image and audio models#
Each media component requires the exact matching language model and encoder or projector. Text support for a family does not enable every vision or audio variant.
| Input | Supported trained pair | Setup and limits |
|---|---|---|
| Images | Qwen3.5-0.8B Q8_0 with its F16 vision GGUF component | Native image input. |
| Audio | Qwen2-Audio-7B-Instruct Q4_K_M with its F16 audio encoder | Native audio input. |
A combined store can hold both supported media parents. Use separate requests when their language parents differ. A prompt that requires both parents is refused. The media guides specify the exact source revisions, file hashes and capacity requirements.
Embedding and reranker models#
The reference catalog includes Qwen3-Embedding-0.6B Q8_0 and Qwen3-Reranker-0.6B Q8_0. These roles have separate requirements from the language role.
An embedding model can use a different vocabulary from the language model. The runtime uses a separate tokenizer table when required. A reranker must share the language model's token IDs and vocabulary. Matching vocabulary sizes or family names does not prove compatibility.
The checked Qwen3 reranker is incompatible with the checked Llama 3.2, Qwen2.5, Qwen3.5 and OLMoE vocabularies.
Select language,embedding for those stores and omit the incompatible reranker.
See conversation setup.
Recognized names and limits#
The model descriptor recognizes these architecture names for rotary-position pairing:
llama, olmo, qwen2, qwen3, qwen3moe, olmoe and qwen35.
This table selects a rotary rule; it is not a complete architecture-support list.
| Recognized name | Limit |
|---|---|
qwen3moe |
A rotary rule exists, but the expert loader refuses its routing architecture. |
olmo |
A rotary rule exists. This release's checked family examples do not establish complete OLMo model support. |
| Other names | A name outside the rotary table is refused. |
The current descriptor permits at most 64 layers. Tensor dimensions, state storage and available GPU memory impose additional limits. Larger build profiles do not establish that every larger model fits or has runtime qualification.
The inspector requires tokenizer.ggml.model = gpt2 for the text tokenizer.
Its compiled pre-tokenizer patterns cover Qwen2, Llama 3, GPT-2 and Qwen3.5 splits.
Several metadata names select each pattern.
For example, deepseek-r1-qwen is a pre-tokenizer name; it does not establish support for a DeepSeek model architecture.
Inspect the architecture and tokenizer fields separately.
Tensor types and quantization#
The current tensor type table contains:
| Group | Tensor types |
|---|---|
| Floating point | F32, F16 |
| Legacy quantized blocks | Q4_0, Q4_1, Q5_0, Q5_1, Q8_0 |
| K quantized blocks | Q2_K, Q3_K, Q4_K, Q5_K, Q6_K |
Each layer still checks the types required for its individual tensors.
Some scalar and normalization tensors require F32.
A label such as Q4_K_M describes a model conversion, not one uniform tensor type.
Inspect every tensor type in the file.
Types outside the compiled table, including BF16 and IQ types, are not accepted by this build.
Check another model file#
- Select a fixed model revision and read its license.
- Inspect the local GGUF file before adding it to a store:
build/aotx_models inspect /path/to/model.gguf
- Check the architecture, pre-tokenizer, tensor types and layer sets in the report.
- Verify the complete file size and SHA-256 digest.
- Register the file and its exact chat wrapper with the model store.
- Check startup, prompt processing and recovery with the required build profile and roles.
build_support=yes covers the reported header fields and tensor sets only.
The inspector reports run_verified=no; it does not run inference or qualify a task.
Require spans=pass ends=pass prefill=pass and usable=yes on the startup wrapper report.
The GGUF specification defines the file metadata and tensor descriptors. AOTX's compiled loader determines which combinations this release accepts.
Optional feature qualification#
Ordinary conversation support does not qualify automatic memory, appraisal or numerical controls. Those features require their exact model, wrapper, processor or control identities. A fine-tune or different quantization changes the file identity and needs its own checks.
Read support and qualification before enabling these features. Use model files for registration and wrapper procedures.