prain: personas for the chats; README on quantization levels and personas

prain web/tui --persona FILE sends the file as the system message ahead of every warm-up and
answer; the page shows its name. Three examples in examples/personas (task, how to answer,
environment, how to use it). A persona costs 2.0-3.2 s once (241-467 tokens), then at most
0.3 s per conversation from the prefix cache.

README rewritten around launching: quick start, the vLLM flags, a quantization table measured
today at equal expert VRAM (bf16 0.90 / int8 2.19 / int4 5.19 tok/s; 2.3k-token prompt 69.5 /
32.3 / 20.5 s; 4 of 6 int8 and 0 of 6 int4 answers identical to bf16, one int4 answer wrong),
how to switch formats, the VRAM budget, writing personas, the chats and the monitor.
This commit is contained in:
2026-10-29 23:55:36 +01:00
parent a14e3ade0b
commit a67238f66b
13 changed files with 464 additions and 150 deletions
+13
View File
@@ -68,6 +68,19 @@ class Cli(TmpCase):
main(["web", "--host", "0.0.0.0"])
self.assertEqual(web.call_args.kwargs["host"], "0.0.0.0")
def test_a_persona_file_reaches_web_and_tui(self):
path = self.write("gentoo-admin.md", "You look after a Gentoo box.\n\n")
with mock.patch("prain.web.serve") as web:
main(["web", "--persona", path])
self.assertEqual((web.call_args.kwargs["system"], web.call_args.kwargs["persona"]),
("You look after a Gentoo box.", "gentoo-admin"))
main(["web"])
self.assertEqual((web.call_args.kwargs["system"], web.call_args.kwargs["persona"]),
(None, None))
with mock.patch("prain.tui.main") as tui:
main(["tui", "--persona", path])
self.assertEqual(tui.call_args.kwargs["system"], "You look after a Gentoo box.")
def test_synth_sim_and_tune(self):
hf = self.write("config.json", json.dumps(GEMMA4))
trace = os.path.join(self.dir, "trace.jsonl.gz")