prain: web UI with a model monitor
A clear chat page (markdown, code blocks with copy, stop, new chat, kept in the browser, drafts shown as such) and a monitor sidebar, a sheet on phones: tokens/s, a live map of the 30 x 128 experts (in VRAM, used now, copied in), VRAM hit rate, copies, context, prompt cache, GPU VRAM/load/temperature/power/fan. GET /status merges vLLM /metrics, amdgpu sysfs and the stats file, which now lists the experts in VRAM. /warm answers 202 when a warm-up starts. README: the int4 kernels, their A/B, 2048-token chunks as a trade, prompt logprobs limit.
This commit is contained in:
@@ -41,10 +41,13 @@ class StubVllm:
|
||||
self.delay = 0.0
|
||||
self.pause = 0.0
|
||||
self.fail = None
|
||||
self.metrics = None # Prometheus text served at GET /metrics
|
||||
stub = self
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path == "/metrics" and stub.metrics is not None:
|
||||
return self.send(200, "text/plain", stub.metrics)
|
||||
self.send(200, "application/json", {"data": [{"id": "stub-model"}]})
|
||||
|
||||
def do_POST(self):
|
||||
|
||||
Reference in New Issue
Block a user