Вернуться обратно

Stdio JSON Protocol

Stdio JSON Protocol

Скачать · Исходный код · О проекте

Stdio JSON Protocol

LevPRO AI runtime exposes a line-delimited JSON (NDJSON) protocol for host applications (e.g. Electron). The subprocess reads requests from stdin and writes protocol messages to stdout. Logs and diagnostics go to stderr only.

Output buffering

Outbound messages are written by a background writer task: a bounded asyncio.Queue drained in FIFO order, flushing stdout once per drained batch instead of once per event/chunk. This keeps a slow pipe reader from stalling the server.

  • Events (streaming chat events like llm_chunk) are best-effort: if the bounded queue is full they are dropped (logged at DEBUG as STDIO_WRITER_DROP) so a slow reader never blocks the LLM stream.
  • Results and errors are never dropped: they wait for the writer to drain the queue.
  • If stdout is closed/broken, the write failure is logged once (ERROR_STACKTRACE) and the server keeps processing requests (output is discarded); shutdown flushes the remaining queue with a bounded timeout (default 2s).

On shutdown (or stdin EOF) the writer drains the outstanding queue before the process exits.

Launch

local-ai serve-stdio [--config config.yaml]

Message format

Each line is one JSON object.

Host → Core (request)

{"id": "1", "method": "ping", "params": {}}
Field Required Description
id yes Correlation id (string or number)
method yes Method name
params no Method-specific object (default {})

Core → Host (response types)

Event (streaming, mainly during chat):

{"id": "1", "type": "event", "event": "llm_start", "step": 0}

Result (success):

{"id": "1", "type": "result", "result": {"status": "ok"}}

Error:

{"id": "1", "type": "error", "code": "model_plan_error", "message": "..."}

Methods (v1)

Method Description Starts llama-server
initialize Negotiate protocol version and report host capabilities no
ping Health check no
doctor System diagnostics no
plan Launch plan preview no
chat Single agent turn with event stream yes (first call)
cancel Cancel an in-flight chat by request id (per-run cancel_event) no
shutdown Graceful app shutdown (shutdown_event) and exit loop stops if started

chat params

Param Type Default Description
user_id string default_user Session user
session_id string random 8-char Session id
agent string first agent in config Agent name
input string "" User message
resume bool false Resume L2 checkpoint
approve_execution bool false Auto-approve MCP tools with requires_approval

chat result

{
  "response": "assistant text",
  "session_id": "abc123",
  "trace_id": "deadbeef",
  "warnings": []
}

warnings may include runtime notices (e.g. corrupt session history reset). Salvageable tool-parse failures are re-prompted up to app.max_parse_errors before abort.

cancel params

Param Type Default Description
request_id string or number — Request id of the active chat to cancel

For backward compatibility, request_id may be omitted when exactly one chat is active. It is required when multiple chats are active.

Example:

{"id":"cancel-1","method":"cancel","params":{"request_id":"chat-1"}}

chat events

Events reuse AgentLoop names:

Event Typical fields
run_start user_input, trace_id
memory_start —
llm_start step
llm_chunk step, delta
llm_done step
tool_start step, tool, args
tool_done step, tool, status, preview, retryable, ms
final step, response
loop_abort step, reason

Lifecycle

  1. Host spawns local-ai serve-stdio.
  2. Host sends NDJSON requests on stdin.
  3. First chat (or explicit warm-up) starts llama-server and builds memory index.
  4. Host sends shutdown or closes stdin → core stops supervisor and exits.

Example (PowerShell)

$proc = Start-Process local-ai -ArgumentList "serve-stdio","--config","config.yaml" -RedirectStandardInput -RedirectStandardOutput -NoNewWindow -PassThru
# Write requests to $proc.StandardInput, read lines from $proc.StandardOutput

Error codes

Code Meaning
config_missing No config.yaml
binary_missing llama-server path invalid
model_missing GGUF path invalid
model_plan_error Model does not fit hardware
unknown_method Unsupported method
invalid_request / invalid_json Malformed request
cancelled Request cancelled
internal_error Unexpected failure

Protocol initialization

Hosts should call initialize after spawning Core and before sending other requests:

{"id":"init-1","method":"initialize","params":{"protocol_versions":[1]}}

Core selects a mutually supported version and returns capabilities such as parallel chat requests and targeted cancellation. Existing v1 hosts may continue without this handshake.

See also docs/bootstrap.md for embedding patterns.