fabolous005
d1ccc5c294
prain: int4 MoE on RDNA2: GEMV-style decode kernel, fp16 rounds, staged expert copies
...
Measured on the live 16k agent setup (A/B with runtime flags, greedy answers vs reference):
decode 3.49 -> 5.24 tok/s, first token 4.3 -> 3.3 s, 2.3k-token prompt 27.5 -> 20.6 s,
same answers. Dropped after measuring: route stats on the CPU, a Triton router GEMV,
pregate prefetch from disk (all within noise or slower).
The GEMV keeps one config instead of autotuning: Triton benchmarks with a 256 MiB buffer
that torch keeps cached, which left 8 MiB of VRAM and made lazy kernel loads fail
(hipErrorNoBinaryForGpu). prain also returns torch cached blocks after materializing and
whenever the driver has < 128 MiB free. quality.py score --greedy-only skips prompt logprobs,
which OOM the 16k config.
2026-10-29 14:20:23 +01:00
fabolous005
0f3911afb5
prain: span profiler with flame graphs (PRAIN_PROFILE: engine step, every model module, MoE route/plan/copy/round, call counts; python -m prain.profile); prain web --host
2026-10-29 08:39:28 +01:00
fabolous005
216cd76a77
prain: Triton GEMV for decode GEMMs on gfx10 (rocBLAS ran bf16 GEMVs at 6% of the bandwidth; agent setup 1.58 -> 2.77 tok/s, same answers)
2026-10-28 22:07:25 +01:00
fabolous005
708e152549
prain: int8/int4 experts quantized once on the GPU, rounds through vLLM's expert_map, [miss] min_weight skips experts not in VRAM; scripts: tune_moe, quality (answer NLL on identical token ids), bench --context, measure --quant
2026-10-28 22:07:25 +01:00
fabolous005
16c3bf81f7
prain: pin the RAM tier per expert (big pinned blocks take minutes on eGPU hosts); sim charges disk reads on the CPU path; measure CPU cost on cold experts
2026-10-28 00:33:34 +01:00
fabolous005
01dce2ffc6
prain: tune searches the auto CPU threshold and writes --cpu-threads; vLLM patch: fit Gemma 4 in 8 GB (tied lm_head without a copy, rope caches sized by max_model_len)
2026-10-27 23:52:53 +01:00
fabolous005
036f4611cc
prain: vLLM tiered experts layer and plugin
2026-10-27 15:55:57 +01:00
fabolous005
7bab776a16
prain: tiered forward with CPU path
2026-10-27 15:54:11 +01:00
fabolous005
1cf3c18a82
prain: draft rows from the batch layout
2026-10-27 13:52:25 +01:00
fabolous005
cf9a347792
prain: web UI with sessions and drafts
2026-10-27 13:52:25 +01:00
fabolous005
0531aaed44
prain: terminal UI
2026-10-27 13:52:16 +01:00
fabolous005
041398d849
prain: chat client with cancellable warm-up
2026-10-27 13:52:14 +01:00
fabolous005
7a78cdee79
prain: command line
2026-10-27 13:52:10 +01:00
fabolous005
168251e81c
prain: traces, simulator and tuner
2026-10-27 13:52:09 +01:00
fabolous005
582601101b
prain: slot cache with eviction, scan, prefetch and CPU policies
2026-10-27 13:52:07 +01:00
fabolous005
8b34961308
prain: stats planner and TOML writer
2026-10-27 13:52:06 +01:00
fabolous005
a5f011dcb8
prain: placement and option tables
2026-10-27 13:52:06 +01:00
fabolous005
db1736c42d
prain: expert id ranges
2026-10-27 13:52:05 +01:00
fabolous005
c8d5706956
prain: skeleton
2026-10-27 13:52:04 +01:00
fabolous005