prain/README.md

# prain

Serve a mixture-of-experts model with vLLM when its experts do not fit in VRAM. A TOML file places
each expert of each layer in VRAM (pinned or in a cache slot), in RAM, or in a pack file on disk.
Missing experts are copied to the GPU or computed on the CPU (`[miss]`). Eviction, heat, prefetch
and miss policies are configurable, and `prain tune` picks them from a routing trace. Experts can be
served as bf16, int8 or int4 (`quant`).
Target: Gemma 4 26B-A4B (30 layers x 128 experts, 42.5 GiB of bf16 experts) on an 8 GB RX 6600 XT.

## Build (GPU host, user space)

    scripts/build.sh wrappers sdk fetch torch torchvision vllm prain

- `sdk`: venv with AMD's ROCm SDK wheels for `PRAIN_GPU_ARCH` (default gfx1032)
- `torch`, `torchvision`: from source with clang and `-march=native` (hours)
- `vllm`: v0.32.0 from source with gfx1032 added and `patches/vllm-*.patch` applied
- `prain`: editable install and the test suite

Check the stack: `scripts/smoke.sh MODEL_DIR --attention-backend TRITON_ATTN`

Tune vLLM's fused-MoE kernel for the GPU once (~45 min of Triton compiles the first time;
`scripts/env.sh` points vLLM at the files). It keeps vLLM's default wherever no candidate beats it:

    python scripts/tune_moe.py build/moe-configs

## Serve

    . scripts/env.sh
    prain serve examples/gemma4-26b-a4b-rx6600xt.toml MODEL_DIR -- --served-model-name gemma4 \
        --language-model-only --max-model-len 16384 --max-num-batched-tokens 1024 \
        --enable-prefix-caching --attention-backend TRITON_ATTN \
        --enable-auto-tool-choice --tool-call-parser gemma4

The first start writes the bf16 pack (42.5 GiB, ~3 min); with `quant = "int8"` or `"int4"` it
then quantizes that pack once on the GPU (~3 min). Once the int4 pack is written, the bf16 files
(`layerNNN.w13`, `.w2`, `.done`) are not read and can be deleted; a later bf16 or int8 start
writes them again. Chat with `prain tui`, or `prain web --config FILE` for several users with
drafts (`--host 0.0.0.0` serves it to other machines; it has no login). Tool calls work through
vLLM's `gemma4` parser, so coding agents and MCP clients can use the OpenAI-compatible API.

## Expert formats

| quant | expert | all experts | weights | VRAM kernel | CPU kernel |
|---|---|---|---|---|---|
| bf16 | 11.34 MiB | 42.5 GiB | as trained | vLLM Triton | torch bf16 matmul |
| int8 | 5.69 MiB | 21.3 GiB | fp32 scale per row | vLLM Triton (w8a16) | `torch._weight_int8pack_mm` |
| int4 | 3.01 MiB | 11.3 GiB | bf16 scale per 64, MSE clip | vLLM Triton (GPTQ/AWQ, w4a16) | `torch._weight_int4pack_mm_for_cpu` |

Measured on the RX 6600 XT behind Thunderbolt (PCIe 3.0 x4, 2.5 GB/s), i9-13950HX (16 CPU
threads for experts), 31 GiB RAM, quiet host. Every format got 1.66 GiB of expert VRAM, its own
placement from `prain tune` and tuned kernels; 4k context, 0.75 GiB KV:

| | bf16 | int8 | int4 |
|---|---|---|---|
| experts in VRAM (cache slots) | 120 | 270 | 540 |
| decode, tok/s (median of 3 x 48 tokens) | 1.04 | 1.42 | 1.62 |
| first token, short chat prompt (median) | 22.9 s | 5.0 s | 4.9 s |
| 3 greedy answers x 64 tokens | 221 s | 143 s | 123 s |
| greedy answers identical to bf16 | - | 1 of 3 | 1 of 3 |

The smaller formats win twice: more experts fit in VRAM, and every miss moves fewer bytes over
the link and out of the page cache (the int4 pack fits in free RAM, the bf16 one does not). On
the GPU itself a decode round costs about the same in every format (2.3-2.8 ms). These decode
numbers predate the dense-layer fix below, which makes every format faster.

## Dense layers on RDNA2

rocBLAS runs bf16 GEMVs at 13-16 GB/s on gfx1032 (RDNA2 has no bf16 dot instructions), and vLLM
uses its own skinny-GEMM kernels only on gfx9/gfx11/gfx12. A decode step spent ~350 ms just
reading the dense weights (lm_head alone: 93 ms). On gfx10 prain sends every unquantized GEMM
with at most 4 rows through a Triton GEMV (`prain/gemv.py`, ~210 GB/s; `PRAIN_GEMV=0` turns it
off). With the agent setup below, decode went from 1.58 to 2.77 tok/s with unchanged answers
(same quality score). Prefill keeps rocBLAS.

## Trading quality for speed

`[miss] min_weight` (default 1) is the share of each token's router weight that must be
computed. Below 1, a token skips its lightest experts that are not in VRAM once it has that
share, and its other routes are scaled up to its full weight: fewer copies, a slightly different
model. Measured on the agent setup: int4, 16k context, 1 GiB KV and 450 cache slots (1.32 GiB) as
in the example config, per-layer slot counts from `prain tune`. Quality is the mean
negative log-likelihood of the min_weight = 1 answers to six chat prompts (`scripts/quality.py`;
0.11 is that config scoring its own answers):

| min_weight | answer NLL | decode tok/s | first token, short prompt | first token, 2.5k-token prompt |
|---|---|---|---|---|
| 1.0 | 0.11 | 2.77 | 4.6 s | 28.5 s |
| 0.6 | 0.19 | 3.15 | 2.7 s | 23.0 s |
| 0.4 | 0.40 | 3.36 | 2.0 s | 19.9 s |

At 0.6 the answers stay correct but take other words; at 0.4 they show slips ("Hereon any of the
following one-liners"). Pinning only the hottest experts and never copying (`prain plan
--cache-slots 1`, min_weight 0) broke the model (NLL 9) for 25% more decode speed (measured before
the dense-layer fix).

## Context length and coding agents

Gemma 4 has 25 sliding-window layers (window 1024) and 5 full-attention layers, so KV grows slowly
with context: per request ~20 KiB per token plus ~0.6 GiB for the sliding layers, which also
hold the tokens in flight (2 x `--max-num-batched-tokens`). `kv_cache_gib = 1.0` holds 18,169
tokens, 1.11 requests of 16k (measured).

With the example config (int4, 16k, tools) a cold 4.7k-token prompt took 70 s and a 12k-token
one 176 s: every 1024-token prefill chunk needs nearly all experts, copied over the 2.5 GB/s link.
Agent turns reuse the prefix cache: turns 2 and 3 of a conversation found 4,704 and 4,736 of
~4,750 prompt tokens cached and took ~20 s each (24-token answers). vLLM keeps sliding-window
state only where earlier prompts ended, so the second conversation that shares just the system
prompt recomputes it once (176 s); later ones hit (4.5 s).

## Tune for a machine

    python scripts/measure.py PACK_DIR/layer029.w13 --quant int4 --threads 16 > cost.json  # server stopped
    prain sim TRACE CONFIG MODEL_DIR/config.json --cost cost.json
    prain tune TRACE MODEL_DIR/config.json --cost cost.json --quant int4 --expert-vram-gib 1.4 \
        --ram-gib 0 --pack-dir PACK_DIR --cpu-threads 16 > tuned.toml
    python scripts/bench.py http://127.0.0.1:8000 [--context FILES...]          # speed
    python scripts/quality.py http://127.0.0.1:8000 make ref.json   # reference answers, then
    python scripts/quality.py http://127.0.0.1:8000 score ref.json  # on another config

Judge quality on chat answers, not raw-text perplexity: this instruction-tuned model scores raw
code at a negative log-likelihood of 5-9 per token on a correct engine.

The simulator treats every disk-tier expert as a disk read, so it under-predicts bf16 (0.6 vs 1.04
tok/s measured, the page cache helps) and over-predicted int4 (2.4 vs 1.62 tok/s).

## Profile

Serve with `PRAIN_PROFILE=/abs/dir`. Writing a number N into `dir/start` records the next N engine
steps: spans around the engine step, every module of the model and prain's MoE phases (route,
plan, copy: h2d/repack/d2d, round), each synced with the GPU. `python -m prain.profile dir`
writes `decode.svg`/`prefill.svg` flame graphs and prints self time and calls per step. The
syncs remove the CPU/GPU overlap: a decode step took 395 ms profiled, 307 ms without.

## vLLM patches

- `vllm-0.32-rocm-no-cuda-families`: gfx1032 reports capability 10.3, and vLLM's CUDA "family"
  checks took it for Blackwell (B200 attention tiles: 128 KiB of LDS on a 64 KiB GPU).
- `vllm-0.32-gemma4-fit-8gb`: Gemma 4 shares `embed_tokens` as `lm_head` instead of allocating a
  second 1.4 GiB copy while loading, and sizes rope caches by `max_model_len` (saves ~0.4 GiB).