prain: personas for the chats; README on quantization levels and personas

prain web/tui --persona FILE sends the file as the system message ahead of every warm-up and
answer; the page shows its name. Three examples in examples/personas (task, how to answer,
environment, how to use it). A persona costs 2.0-3.2 s once (241-467 tokens), then at most
0.3 s per conversation from the prefix cache.

README rewritten around launching: quick start, the vLLM flags, a quantization table measured
today at equal expert VRAM (bf16 0.90 / int8 2.19 / int4 5.19 tok/s; 2.3k-token prompt 69.5 /
32.3 / 20.5 s; 4 of 6 int8 and 0 of 6 int4 answers identical to bf16, one int4 answer wrong),
how to switch formats, the VRAM budget, writing personas, the chats and the monitor.
This commit is contained in:
2026-10-29 23:55:36 +01:00
parent a14e3ade0b
commit a67238f66b
13 changed files with 464 additions and 150 deletions
+13
View File
@@ -124,6 +124,19 @@ class Web(unittest.TestCase):
"<done>"])
thread.join()
def test_a_persona_leads_every_warm_up_and_answer(self):
self.start(system="You answer in German.", persona="german")
self.warm("s1", "Wie spät")
wait_for(lambda: self.stub.bodies)
with self.ask("s1", "Wie spät ist es?") as r:
r.read()
system = {"role": "system", "content": "You answer in German."}
self.assertEqual([b["messages"][0] for b in self.stub.bodies], [system, system])
self.assertEqual(self.stub.bodies[1]["messages"][1:],
[{"role": "user", "content": "Wie spät ist es?"}])
with urllib.request.urlopen(self.url + "/status") as r:
self.assertEqual(json.load(r)["persona"], "german")
def test_refine_none_keeps_the_draft_and_more_full_slots_mean_no_draft(self):
self.start(draft=Draft(refine="none"))
self.stub.delay = lambda body: 0.0 if "vllm_xargs" in body else 0.5