Start with a short reply on loopback. Once it works, change one setting at a time. Run the commands from your CKE checkout after Linux setup.
Choose weights, address and budget
.venv/bin/python -m pip install -r server/requirements.txt
MODEL=hf://unsloth/gemma-3-270m-it-GGUF/gemma-3-270m-it-Q5_K_M.gguf
RUN="$HOME/cke-runs/gemma3-270m-1k"
version/v8/scripts/cks-v8-run serve "$MODEL" --run "$RUN" \
--host 127.0.0.1 --port 8080 --model-name cke-local \
--context-len 1024 --max-tokens 64 --temperature 0 \
--request-output-cap 128 --request-prompt-byte-cap 8192- Which model? The positional argument picks the weights.
--rungives this build its own directory. Once the bundle works, add--no-buildto reopen it. Keep separate directories for different checkpoints, context builds and serving variants. - Which address?
--hostand--portselect the listener. The local development service defaults to 127.0.0.1:8080 and has no built-in authentication. An agent on the same host can use that address. An agent in another container has its own loopback. - Which name?
--model-namesets the ID the client sends. Choose a descriptive name when running multiple endpoints. A friendly name alone cannot identify the exact artifact. - How much fits?
--context-lenis the token budget. The formatted prompt includes instructions, history, role markers and tool definitions. The server needs room for that input plus the requested reply. Reopening a compiled bundle cannot increase its capacity. - How much may it answer?
--max-tokensdefaults the HTTP reply allowance to 512 if omitted. A client can request a different allowance withmax_output_tokens.--request-output-capclamps that allowance.--request-prompt-byte-caplimits rendered UTF-8 input bytes independently. A multibyte character, a byte and a model token are different units.
Choose sampling and stop conditions
--temperature 0 picks the highest-scoring next token. For sampling, --temperature defaults to 0.7 and --top-p to 1.0. Clients can override these with temperature and top_p. Keep the model, template, prompt and settings fixed when comparing outputs. The launcher accepts several compatibility flags, including --top-k and --repeat-penalty, and warns that its native HTTP sampling path ignores them.
Use repeatable --stop-on-text for a decoded delimiter specific to your application. A reply can also finish early at the model's end token. Check the response status and token usage before increasing the limit.
Templates, tools and concurrent clients
Start with the checkpoint's chat template. --serving-variant tools selects a declared tools variant at build time for models such as Nemotron. --chat-template-file supplies an explicit replacement for a matching model. A wrong template can produce plausible text with incorrect role or tool boundaries. Check the model and harness matrix before configuring a tool client.
The default path handles one active generation at a time. Exact-prefix reuse (--prefix-checkpoint-cache-bytes) and experimental two-slot HTTP (--batch-http-extra-bytes) require suitable generated bundles and budgets. Check the artifact-specific evidence before enabling either for an agent. Use --no-stats to disable console metrics or --no-viz to disable the live request view.
version/v8/scripts/cks-v8-run serve --helpCheck the running endpoint · CKE server flags at the reviewed revision