Skip to content
C-Kernel-Engine docs Antshiv Open Source

C-KERNEL-ENGINE · GUIDE

Run a local inference server

Choose a starting point. Try Gemma or Qwen3 in your terminal and over HTTP. For a tool-using client, run Nemotron with Deep Agents.

Start one model on localhost

Complete Linux setup and run Qwen2 once. From the same checkout, install the server dependencies into its Python environment. Reuse the resulting model bundle for a single local client. Keep this terminal open.

Examplebash
.venv/bin/python -m pip install -r server/requirements.txt
version/v8/scripts/cks-v8-run serve \
  hf://Qwen/Qwen2-0.5B-Instruct-GGUF/qwen2-0_5b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 --model-name cke-local --no-build

Verify the server and send a request

In a second terminal check that health returns mode: live and inference: true. The model list should include the name you chose.

Examplebash
curl -fsS http://127.0.0.1:8080/v1/health
curl -fsS http://127.0.0.1:8080/v1/models
curl -fsS http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"cke-local","input":"What is the capital of France?","temperature":0,"max_output_tokens":24}'

The local smoke test returned “The capital of France is Paris.” Read the message in the response’s output array. Use Ctrl+C in the server terminal when finished.

Settings you are likely to change

  • Port and clients. Pick a free port and use the same port in your client’s base URL, including /v1. Keep the listener on loopback. The development server has no built-in authentication.
  • Model name. --model-name is the identifier a client sends. The positional model argument selects the actual weights. Check both if a request fails.
  • Context. With --no-build, the bundle must already exist. A requested context above its compiled capacity fails. Build a separate run directory using --run and the intended --context-len before serving longer requests.
  • Sampling. Set temperature and output length per request. Enable streaming with "stream":true and use curl -N to display events as they arrive.
  • Chat templates. A normal chat bundle needs its matching tokenizer and template. A missing template calls for checking the conversion and bundle; a raw-prompt override changes how the model reads the conversation.
  • Concurrency and persistence. Start with one generation at a time. The optional two-slot mode is restricted to tested KV-only bundles. Conversation and response records are process-local; keep any history you need before stopping the server.

Supported client behavior and configuration

Configure flags and limits · Agent health and discovery checks · Inspect the generated model