Architecture Documentation

Overview

LevPRO AI runtime is a production-grade local agent runtime built on llama-server. It provides a ReAct agent loop with tool calling, persistent memory, and a CLI interface while keeping all processing local.

Architecture Layers

The system follows a 4-layer architecture with clear separation of concerns:

┌─────────────────────────────────────────────────────────────┐
│                    [ APP ]                                  │
│  • AgentLoop (ReAct loop)                                  │
│  • MemoryRetrieverProtocol → NullMemoryRetriever (default)   │
│    or MemoryRetriever when [memory] + memory.enabled       │
│  • SkillsRetrieverProtocol → NullSkillsRetriever (default) │
│    or SkillsRetriever when [skills] + skills.enabled       │
│  • ToolExecutor (+ optional extra tools)                   │
├─────────────────────────────────────────────────────────────┤
│                    [ CORE ]                                 │
│  • LocalLLMClient or EnsembleRouter ([ensemble])           │
│  • ContextBuilder (prompt assembly)                          │
│  • TokenCounter                                             │
│  • ProcessSupervisor(s) (llama-server watchdog)              │
│  • ModelPlanner (launch parameter planning, llama-tools)   │
├─────────────────────────────────────────────────────────────┤
│                    [ INFRA ]                                │
│  • DirectoryGuard (sandbox security)                         │
│  • SessionStore (JSON persistence)                           │
│  • (optional [memory]) MemoryStore + MemoryManager (L0–L3)  │
└─────────────────────────────────────────────────────────────┘

Optional pip extras (wired in app.py when installed + enabled):
  [memory] [ensemble] [skills] [monitor] [autotune]
  MCP tools: local-ai-mcp hard dep; servers from mcp.json only

Layer Responsibilities

[INFRA] Layer

Provides foundational services that other layers depend on.

Modules:

• security/guard.py - DirectoryGuard: path sandbox for Core runtime (and optional workspace skills root)
• session/store.py - SessionStore: JSON-based chat history storage
• packages/local-ai-memory/memory/ (optional [memory]) — MemoryStore, MemoryManager (L0–L3), WorkingMemoryCompressor

Purpose:

• Enforce security boundaries
• Provide persistent storage
• Handle concurrent access safely

Dependencies:

• Python standard library (pathlib, asyncio)
• No external dependencies

[CORE] Layer

Core business logic and infrastructure services.

Modules:

• core/llm_client.py - LocalLLMClient: HTTP client for llama-server
• core/supervisor.py - ProcessSupervisor: llama-server process watchdog
• core/context_builder.py - ContextBuilder: prompt assembly with token budgeting
• core/token_counter.py - TokenCounter: token counting via llama-server
• core/logging_config.py - logging configuration and event tagging

Planner (separate package llama-tools):

• llama_tools/planner/ - ModelPlanner: launch parameter planning from GGUF metadata

Purpose:

• Manage llama-server lifecycle and health
• Compute optimal launch parameters for models
• Build prompts within token budgets
• Communicate with llama-server API

Dependencies:

• httpx (async HTTP client)
• psutil (system probing)
• gguf (GGUF metadata parsing)
• cpuinfo (CPU feature detection)

[APP] Layer

Application-specific logic for the agent loop and tool execution.

Modules:

• agents/loop.py - AgentLoop: ReAct agent loop orchestration
• core/protocols.py - MemoryRetrieverProtocol / NullMemoryRetriever (default); real MemoryRetriever from [memory]
• core/protocols.py - SkillsRetrieverProtocol / NullSkillsRetriever (default); real SkillsRetriever from [skills]
• agents/tool_parser.py - ToolParser: strict JSON tool call parsing
• tools/executor.py - ToolExecutor: tool validation and execution
• tools/meta_tools.py / session tools — native recovery tools (report_inability, load_message, …)
• MCP capability tools via local-ai-mcp (mcp_* from mcp.json)

Purpose:

• Execute the agent ReAct loop
• Retrieve relevant memory/context
• Parse and execute tool calls
• Build message context for LLM

Dependencies:

• All CORE layer modules
• DirectoryGuard for Core runtime path I/O (not MCP tools)
• JSON Schema validation (jsonschema)

Cross-Cutting Concerns

Security

• DirectoryGuard sandboxes Core-owned paths under runtime_data_dir (sessions, memory, archives) and, when set, the optional workspace root used for skills storage
• MCP filesystem/shell tools are not constrained by DirectoryGuard — hosts configure MCP server roots and requires_approval
• Do not treat optional workspace as a general project file sandbox for the agent

Logging

• Structured logging with event tags
• Two loggers: app (application) and llama (llama-server)
• Event tags: LLM_REQUEST, LLM_RESPONSE, TOOL_CALL, TOOL_RESULT, TOOL_RESULT_STORED,

MEMORY_RETRIEVAL_STAGE1, MEMORY_RETRIEVAL_STAGE2, MCP_CONNECT, MCP_TOOL_CALL, SKILL_STORED, SKILL_RETRIEVAL,
SUPERVISOR_RESTART, ERROR_STACKTRACE

Concurrency

• All file I/O uses asyncio.to_thread() for non-blocking execution
• SessionStore uses per-file asyncio.Lock for concurrent access
• ProcessSupervisor watchdog runs as asyncio task

Bootstrap and Embedding

LevPRO AI runtime separates sync construction (create_app()) from async runtime (supervisor.start() → ctx.startup() → agent work → ctx.shutdown()). All interfaces share a single AgentLoop from AppContext.

See docs/bootstrap.md for the full dependency graph, lifecycle checklist, selective component embedding, and manual wiring cookbook.

Data Flow

Agent Request Flow

1. User Input
   ↓
2. SessionStore.append() - Add user message to history
   ↓
3. memory_retriever.retrieve() — no-op when NullMemoryRetriever; indexed RAG when [memory] enabled
   ↓
4. Load L1 working memory (WorkingMemoryCompressor / MemoryManager)
   ↓
5. ContextBuilder.build() - Assemble prompt with token budgeting
   ↓
6. AgentLoop.run() - Main loop
   ├─ Build messages (system + tools + working memory + RAG + history + input)
   ├─ LocalLLMClient.chat() or EnsembleRouter.chat() ([ensemble]) → response
   ├─ ToolParser.parse_tool_call() → tool call or final answer
   ├─ ToolExecutor.execute() → ToolResult
   │    └─ large success? → MemoryManager L3 + compact reference JSON
   └─ Update history with tool observation (inline or reference)
   ↓
7. SessionStore.append() - Add assistant response
   ↓
8. WorkingMemoryCompressor.update_summary() - Compress run into L1
   └─ SessionStore.compact() when session exceeds history_limit
   ↓
9. Return final response

Model Launch Planning Flow

1. ModelPlanner.plan()
   ↓
2. GgufMetadataReader.read() - Parse GGUF file
   ↓
3. SystemProbe.detect() - Detect hardware (RAM, CPU, GPU)
   ↓
4. Compute optimal parameters:
   ├─ ctx_size (context window that fits)
   ├─ gpu_layers (max layers for VRAM)
   ├─ n_batch / n_ubatch (batch sizes)
   ├─ threads (CPU allocation)
   ├─ flash_attn (flash attention setting)
   └─ cache_type_k/v (cache dtype)
   ↓
5. fits_in_memory() validation
   ↓
6. Return PlanResult with estimates

File Organization

packages/local-ai-core/
├── core/                  # llm_client, supervisor, token_counter, context_builder, mcp_json
├── agents/                # loop, tool_parser, registry
├── session/               # SessionStore, SessionRunner
├── security/guard.py      # DirectoryGuard
├── tools/                 # executor, meta/session tools, lazy_tool
├── interfaces/            # cli, stdio_server
├── config/loader.py
├── app.py                 # AppContext factory
└── main.py                # local-ai entry

packages/local-ai-memory/          # [memory] — store, retriever, manager, compressor
packages/local-ai-mcp/             # hard dep — lazy stdio + HTTP MCP bridge
packages/local-ai-ensemble/        # [ensemble] — multi-model EnsembleRouter
packages/local-ai-skills/          # [skills] — skill capture + retrieval
packages/local-ai-monitor/         # [monitor] — RAM/VRAM polling + graceful degradation
packages/local-ai-autotune/        # [autotune] — launch-param calibration
packages/llama-tools/llama_tools/  # planner, doctor, plan, inspect, benchmark CLI

Key Design Decisions

1. Single AgentLoop Instance

All interfaces share a single AgentLoop instance from AppContext. This ensures:

• Consistent state management
• No duplication of agent logic
• Proper session tracking

2. Strict JSON Tool Parsing

Tool calls must be strict JSON (no regex fallback). This ensures:

• Predictable parsing behavior
• Security (no ambiguity)
• Compatibility with all models

3. 3-Stage Memory Retrieval

Memory retrieval uses three stages:

• Stage 1: BM25 or hybrid index search (no LLM, top 5)
• Stage 2: LLM selects from filtered candidates
• Stage 3: Budgeted content load

This balances speed and relevance.

4. Token Budgeting

ContextBuilder enforces a token budget to prevent:

• LLM context overflow
• Excessive memory usage
• Slow responses

5. Process Supervision

ProcessSupervisor watches llama-server with:

• 5-second health check interval
• Restart on crash or 3 consecutive failures
• Exponential backoff (1s → 30s max)

6. Path sandbox (DirectoryGuard)

Core-owned file I/O goes through DirectoryGuard, which:

• Resolves paths to absolute
• Validates paths stay under its base (runtime_data_dir, or optional workspace for skills)
• Raises PermissionError for escapes

Agent capability tools (read/write project files, shell, etc.) come from host MCP servers and are outside this guard.

Performance Considerations

• Async-first design: All I/O is async, blocking operations use asyncio.to_thread()
• Token budget: Prevents unbounded context growth
• Index-based memory retrieval: Fast initial filtering
• Health checks: Non-blocking supervisor checks
• Exponential backoff: Prevents resource exhaustion on restarts

Scalability

The architecture supports scaling through:

• Multiple sessions: Each session is isolated in JSON
• Multiple users: User ID separation in storage
• Memory categories: Organized by category for retrieval
• Configurable tools: Tools are registered and enabled per agent

Security Boundaries

1. File System: DirectoryGuard sandboxes Core runtime/skills paths; MCP servers own their own roots (not Core isolation)
2. MCP approval: ApprovalGate for MCP tools with requires_approval in mcp.json
3. Network: llama-server for the agent runtime; MCP HTTP servers use host-configured URLs/headers
4. Memory: User/session isolation
5. Configuration: Typed config with validation
6. Tool Execution: JSON Schema validation before execution

Security model

Capability tools (filesystem, shell, etc.) are provided by host MCP servers — Core does not ship a process sandbox package.

Logging and Observability

Log Locations

• logs/app.log - Application events
• logs/llama.log - llama-server stdout/stderr

Event Tags

Use core/logging_config.log_event() for structured logging:

• LLM_REQUEST - Before LLM call
• LLM_RESPONSE - After LLM response
• TOOL_CALL - Tool invocation
• TOOL_RESULT - Tool result
• TOOL_RESULT_STORED - Large tool output persisted to L3
• MEMORY_RETRIEVAL_STAGE1 - Index filtering
• MEMORY_RETRIEVAL_STAGE2 - LLM selection
• WORKING_MEMORY_COMPRESSED - L1 working memory updated after agent run
• SCRIPT_EXEC_START / SCRIPT_EXEC_DONE - Sandboxed script execution
• SCRIPT_EXEC_DENIED - User or policy denied script execution
• SUPERVISOR_RESTART - Supervisor restart event
• ERROR_STACKTRACE - Error details

Configuration

Configuration is loaded from YAML and typed with dataclasses:

llama:
  binary_path: "..."
  models:
    main:
      path: "..."
      port: 8080
  optimization_mode: balanced
  os_reserve_mb: 4096
  disk_reserve_mb: 4096

# workspace: "./workspace"  # optional; skills storage root (not a Core FS index)

agents:
  assistant:
    tools: []   # mcp_* from mcp.json are auto-granted

mcp:
  enabled: false
  external_path: null

memory:
  enabled: true
  categories: ["knowledge"]
  search_mode: bm25
  working_memory_target_tokens: 768
  working_memory_enabled: true
  working_memory_compact_keep_last: 6

app:
  history_limit: 12
  max_steps: 20
  context_token_budget: 6000
  tool_result_inline_limit: 6144
  max_parse_errors: 5

See config.yaml for full reference.

Testing Strategy

All modules have comprehensive tests:

• Unit tests with mocked LLM
• Security sandbox tests
• Session concurrency tests
• Tool executor validation tests
• Tool result store / L3 externalization tests
• Model planner tests
• Memory retriever tests
• App bootstrap tests

Run tests with: python -m pytest tests/ -v