prain: serve with the string chat template content format

2026-11-04 21:50:46 +01:00
parent a67238f66b
commit 1e3a412563
3 changed files with 10 additions and 3 deletions
@@ -53,8 +53,11 @@ it:

 `prain serve PLACEMENT MODEL_DIR -- VLLM_FLAGS` validates the placement file and starts
 `vllm serve` with prain's plugin. prain adds `--enforce-eager --moe-backend triton
--scheduling-policy priority`, and it takes `--max-num-seqs` and `--kv-cache-memory-bytes` from
-the file's `[serve]` table (it refuses them after `--`). The flags in `$FLAGS` above:
+--scheduling-policy priority --chat-template-content-format string`, and it takes
+`--max-num-seqs` and `--kv-cache-memory-bytes` from the file's `[serve]` table (it refuses them
+after `--`). vLLM would otherwise pick the `openai` content format for Gemma 4, whose template
+then prints a system prompt as a Python list (`[{'type': 'text', 'text': '…\n…'}]`, newlines
+escaped); a flag after `--` still overrides it. The flags in `$FLAGS` above:

 | flag | why |
 |---|---|