prain: personas for the chats; README on quantization levels and personas

prain web/tui --persona FILE sends the file as the system message ahead of every warm-up and
answer; the page shows its name. Three examples in examples/personas (task, how to answer,
environment, how to use it). A persona costs 2.0-3.2 s once (241-467 tokens), then at most
0.3 s per conversation from the prefix cache.

README rewritten around launching: quick start, the vLLM flags, a quantization table measured
today at equal expert VRAM (bf16 0.90 / int8 2.19 / int4 5.19 tok/s; 2.3k-token prompt 69.5 /
32.3 / 20.5 s; 4 of 6 int8 and 0 of 6 int4 answers identical to bf16, one int4 answer wrong),
how to switch formats, the VRAM budget, writing personas, the chats and the monitor.
This commit is contained in:
2026-10-29 23:55:36 +01:00
parent a14e3ade0b
commit a67238f66b
13 changed files with 464 additions and 150 deletions
+3 -1
View File
@@ -23,7 +23,9 @@ pack_dir = "../models/packs/gemma4-26b-a4b" # bf16 pack (42.5 GiB) written on t
quant = "int4" # then quantized once into an int4 pack (11.3 GiB):
# bf16 | int8 (5.69 MiB/expert) | int4 (3.01 MiB)
default_tier = "disk" # runs on any RAM size; `prain tune` does better
cache_slots = 15 # 30 x 15 x 3.01 MiB = 1.32 GiB of VRAM
cache_slots = 15 # 30 x 15 x 3.01 MiB = 1.32 GiB of VRAM. The same VRAM
# holds 8 int8 or 4 bf16 experts per layer: change
# quant and cache_slots together
stats_path = "../models/packs/gemma4-26b-a4b/stats.json" # prain plan and the web monitor read it
stats_every = 8 # MoE forward passes between stats/trace writes (~3 ms each)
trace_path = "../models/packs/gemma4-26b-a4b/trace.jsonl.gz" # routing trace for sim/tune