Choose a starting point. Try Gemma or Qwen3 in your terminal and over HTTP. For a tool-using client, run Nemotron with Deep Agents.
Start one model on localhost
Complete Linux setup and run Qwen2 once. From the same checkout, install the server dependencies into its Python environment. Reuse the resulting model bundle for a single local client. Keep this terminal open.
.venv/bin/python -m pip install -r server/requirements.txt
version/v8/scripts/cks-v8-run serve \
hf://Qwen/Qwen2-0.5B-Instruct-GGUF/qwen2-0_5b-instruct-q4_k_m.gguf \
--host 127.0.0.1 --port 8080 --model-name cke-local --no-buildVerify the server and send a request
In a second terminal check that health returns mode: live and inference: true. The model list should include the name you chose.
curl -fsS http://127.0.0.1:8080/v1/health
curl -fsS http://127.0.0.1:8080/v1/models
curl -fsS http://127.0.0.1:8080/v1/responses \
-H 'Content-Type: application/json' \
-d '{"model":"cke-local","input":"What is the capital of France?","temperature":0,"max_output_tokens":24}'The local smoke test returned “The capital of France is Paris.” Read the message in the response’s output array. Use Ctrl+C in the server terminal when finished.
Settings you are likely to change
- Port and clients. Pick a free port and use the same port in your client’s base URL, including
/v1. Keep the listener on loopback. The development server has no built-in authentication. - Model name.
--model-nameis the identifier a client sends. The positional model argument selects the actual weights. Check both if a request fails. - Context. With
--no-build, the bundle must already exist. A requested context above its compiled capacity fails. Build a separate run directory using--runand the intended--context-lenbefore serving longer requests. - Sampling. Set temperature and output length per request. Enable streaming with
"stream":trueand usecurl -Nto display events as they arrive. - Chat templates. A normal chat bundle needs its matching tokenizer and template. A missing template calls for checking the conversion and bundle; a raw-prompt override changes how the model reads the conversation.
- Concurrency and persistence. Start with one generation at a time. The optional two-slot mode is restricted to tested KV-only bundles. Conversation and response records are process-local; keep any history you need before stopping the server.
Supported client behavior and configuration
Configure flags and limits · Agent health and discovery checks · Inspect the generated model