Skip to content

Latest commit

 

History

History
49 lines (37 loc) · 2.51 KB

model-server.md

File metadata and controls

49 lines (37 loc) · 2.51 KB

Model server

What this app assumes about the local LLM server, and how those assumptions were verified.

The app talks to oMLX (/Applications/oMLX.app, process omlx-server) on 127.0.0.1:11435 over its OpenAI-compatible API.

Note the spelling: oMLX, not "OLMX". It is not Ollama, and OpenAI-compatible shims differ in the details, so the server was probed directly before any code was written. These are measured facts, not assumptions from the model card:

Question Answer
Model id the server accepts Qwen3-VL-8B-Instruct-MLX-4bit — the bare id, not the lmstudio-community/… HuggingFace repo path
Does streaming work Yes. Progressive data: frames, terminating in data: [DONE]
Is auth required Yes. The server log reports authentication: enabled; an absent or wrong key returns 401
Reasoning leaking into content None. No <think> tags, no reasoning_content field. This is the Instruct (non-thinking) variant
Unknown request fields Tolerated. Both stream_options and a deliberately bogus field returned 200
system role Supported

Two behaviours worth knowing before you debug something that is not broken:

  • A keepalive primer frame arrives first, carrying "model":"keepalive", "created":0, and "content":"". The SSE generator's if msg.content: guard suppresses it. That guard is load-bearing — do not delete it as redundant.
  • Deltas are chunked, not per-token. A ~60-word answer arrived in 15 content deltas of roughly 4–5 words each. Replies appear in visible progressive chunks, not word by word. This is the server's behaviour, not a buffering bug in this app.

Other models present on the server, unused by this app: Qwen3-Coder-30B-A3B-Instruct-MLX-8bit, gpt-oss-20b-MXFP4-Q8.

Reproducing the probe

K=oMLX   # the local API key; see Configuration below

# Which model ids does the server advertise?
curl -s -H "Authorization: Bearer $K" http://127.0.0.1:11435/v1/models

# Does streaming work, and what do the deltas look like?
# -N is required: without it curl buffers and the test proves nothing.
curl -sN http://127.0.0.1:11435/v1/chat/completions \
  -H "Authorization: Bearer $K" -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-VL-8B-Instruct-MLX-4bit","messages":[{"role":"user","content":"Count to five."}],"stream":true}'

If a request fails in a way the HTTP response does not explain, ~/.omlx/logs/server.log usually does — it records the loaded model, auth failures, and request errors.