System overview#

A local inference operating system designed in CUDA.

AOTX-1 runs agents, model inference, memory selection and a message bus on an NVIDIA GPU. CUDA owns the live system state. An asynchronous disk journal preserves completed changes for recovery. Local window and terminal clients expose the same system. A separate HTTP gateway provides text and media access for application clients.

This source is version 0.3.5. The changelog records its changes and earlier releases.

Model files are separate downloads with their own licenses and verified digests.

Capabilities#

Area	Current behavior
Inference	Batched text generation, paged attention, routed experts and supported hybrid layers.
Agents and tools	Conductor, worker and verifier roles; explicit tool selection and operator grants.
Persistent memory	Typed GPU objects, scoped recall, source evidence, corrections and cold storage.
Model controls	Identity modules, voice profiles and controls bound to exact qualified model packages.
Task reviews	Saved task outcomes and exact evidence for later matching tasks.
Media	CUDA image and audio preparation with compatible native model paths.
Applications	Standard chat completions plus native requests, media and persistent shared resources.
Recovery	Ordered journal replay and complete CCIR runtime files with embedded assets.

A runtime uses one GPU. Profiles configure 32, 64, 128 or 256 agent slots; they do not combine memory across cards. Available VRAM, model shape and configured capacities determine which workloads fit.

Qualification

Automatic memory and numerical controls require exact qualified components. Appraisal acceptance applies to the documented model configuration. Ordinary model support does not establish semantic qualification.

See support and qualification for the current boundaries.

Build from source#

The reference environment uses Linux, CUDA Toolkit 13.2 and a GPU with compute capability 8.6. The build requires compute capability 8.0 or above, a compatible driver, CMake 3.28 or later, Ninja and host development libraries. Read build requirements before configuration.

From the repository root:

export PATH=/usr/local/cuda-13.2/bin:$PATH
bash tools/profile-detect.sh

Use the reported profile and architecture. This example targets the 12 GiB reference card:

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DAOTX_PROFILE=12g -DAOTX_ARCH=86
cmake --build build
build/aotx_boot --version

A larger profile is not a promise that every model combination fits that card. The 24g and 48g profiles compile; full-capacity execution on those cards remains unqualified.

Build options cover context, memory, media and optional components. Testing explains focused checks, required model files and display tests.

Start a local conversation#

Fetch and activate a language model from the supplied catalog:

build/aotx_models --dir models list
build/aotx_models --dir models fetch language
build/aotx_models --dir models activate language language

Create aotx.settings:

ini

journal.dir = build/run
models.dir = models
models.roles = language
tui.on = 1

Start the runtime:

build/aotx_boot --settings aotx.settings

Enter say what is a tick in the console. Enter quit to stop and flush the run. Use a new journal directory for a new instance. Operation covers settings, model selection, tools, shutdown and restore.

Attach another terminal to this run:

build/aotx_tui --attach build/run --settings aotx.settings

The graphical control client, build/aotx_ctrl, can create, attach to and stop local instances. The GPU window starts with --window.

Connect an application#

The HTTP gateway runs separately from the CUDA runtime. It provides standard chat completions, streaming output, owned media and exact request cancellation. The native protocol defines request state and output cursors.

Applications that need persistent participants, spaces and conversations use the shared service. It provides durable operation identities, scoped memory and saved responses. Clients discover available features through /aotx/v1/capabilities. No particular frontend is required.

Preserve and restore state#

The journal records authoritative inputs and device decisions in order. Recovery applies recorded tokens and selections without repeating completed tool actions. Only complete durable ticks survive a failed run.

A complete runtime file packages models, modules, settings and recovery state in one CCIR container.

Checkpoints and cold memory preserve typed state beyond the live payload store. These files can contain private conversations and executable modules; keep them under the intended account's control.

Documentation#

Start at the documentation index for procedures, reference manuals and shared terms. Read architecture for state ownership, security for trust boundaries, and support for measured limits.

Task	Guide
Select or inspect a model	Model files
Add roles, skills or tools	Modules and tool SDK
Enable scoped memory	Live memory and semantic memory
Configure identity and controls	Control bindings
Use images or audio	Image input and audio input
Inspect saved state	Journal format and CCIR format
Understand earlier performance figures	Measurements

Source layout

text

cuda/     device modules and host glue; host glue names end in _host.cu
ptx/      embedded CUDA driver modules
disk/    file, journal, model-store and transport programs
ctrl/     graphical control client and its vendored ImGui source
gateway/  HTTP transport, credentials and deployment configuration
modules/  supplied role and identity modules
sdk/      tool contracts and examples
share/    model catalog and terminal artwork
tests/    structural, numerical and runtime checks
docs/     guides and format references
tools/    source gates and authoring tools

Apache License, Version 2.0. See LICENSE.