Project Philosophy

Overview

LevPRO AI runtime is built on a philosophy of local-first, minimal-dependency, and architecture-aware design. Every decision serves the goal of running sophisticated AI agents entirely on local hardware without cloud dependencies or framework bloat.

Local AI-First Approach

Why Local?

1. Privacy - Your data never leaves your machine
2. Control - Full control over what runs and how
3. Cost - No API calls, no monthly subscriptions
4. Latency - No network round trips
5. Reliability - Works offline, no service dependencies

Local-First Principles

Never (by default):

• Send prompts to external APIs
• Rely on cloud services
• Use APIs with rate limits or costs
• Trust external services with sensitive data

Opt-in exceptions (explicit config + host MCP):

• Network/HTTP capability tools only via host mcp.json (stdio or HTTP MCP servers) — Core does not ship a built-in http_fetch / [net] package
• No cloud LLM APIs; any network use is user-controlled through MCP servers and local-first by default

Always:

• Process locally (LLM inference via llama-server)
• Store locally (sessions, memory; optional workspace root for skills)
• Control locally (config, MCP allowlists, ApprovalGate for requires_approval tools)
• Own your data

Use of llama.cpp

Why llama.cpp?

1. Open source - MIT license, no restrictions
2. Efficient - Highly optimized C++ implementation
3. Portable - Works on Windows, Linux, macOS
4. Quantization - Supports multiple quantization levels
5. GPU support - CUDA, ROCm, Metal
6. Active development - Regular updates

llama.cpp Integration

HTTP Server:

• OpenAI-compatible /v1/chat/completions
• Native /tokenize endpoint
• /health for monitoring
• Standard HTTP (no special protocol)

Launch Parameters:

• Context size (ctx-size)
• GPU layers (n-gpu-layers)
• Batch sizes (batch-size, ubatch-size)
• CPU threads (threads, threads-batch)
• Flash attention (flash-attn)
• Cache types (cache-type-k/v)

Why HTTP?

• Language-agnostic (Python, Go, C++, etc.)
• Standardized (OpenAI compatibility)
• Easy to monitor and restart
• No language binding required

Use of Markdown Memory

Why Markdown?

1. Human-readable - Read without special tools
2. Portable - Works everywhere, no format dependencies
3. Version control - Git-friendly
4. Searchable - Easy to index and search
5. Toolable - Easy to parse and process
6. Backed up - Simple copy for backup

Memory Design Principles

Structure:

• Organized by category
• Separated by user
• Indexed for fast retrieval
• Versioned (Git)

Content:

• Plain text with Markdown formatting
• No binary formats
• No proprietary databases
• No encryption (unless user wants)

Retrieval:

• 3-stage indexed RAG (BM25 or hybrid → LLM select → load)
• Default memory.search_mode: bm25 — no embeddings required
• Optional memory.search_mode: hybrid — BM25 + locally computed embeddings (stored as JSON under memory_data/.embeddings/, via llama-server embed endpoint; falls back to BM25 when unavailable)
• Token-budget aware

Why Not Vector Databases?

1. Overkill - BM25 index search is sufficient for most local knowledge bases
2. Complexity - No separate vector DB process or proprietary storage
3. Cost - No specialized hardware or cloud embedding APIs
4. Latency - File-based RAG index + optional local embeddings is fast for typical knowledge-base sizes
5. Simplicity - Embeddings are plain JSON files alongside Markdown; easy to inspect, backup, and version

When to enable hybrid search:

• Semantic similarity matters beyond keyword overlap
• Larger knowledge bases where BM25 alone misses relevant files
• Still local-only: embeddings computed and stored on disk, not in a cloud vector service

Avoiding Mandatory Cloud Services

No Cloud Dependencies

Current design:

• No mandatory external API calls or cloud LLM services
• No network dependencies required beyond local llama-server (optional HTTP tools via host MCP)
• Zero cloud services required
• Zero authentication services

What this means:

• Works fully offline with default config (core + llama-server only)
• No account required
• No data collection
• No privacy concerns from cloud inference
• No dependency on external service uptime for the agent loop

llama-server Requirements

Minimal external dependency:

• Only needs llama-server binary
• No cloud registration
• No API keys
• No authentication

What you need:

• llama.cpp build with HTTP support
• GGUF model file
• That's it

Minimizing Dependencies

Dependency Philosophy

Rule of thumb:

• If Python standard library can do it, use it
• If it's essential, use well-maintained packages
• Never add dependencies for convenience

Current Dependencies

Monorepo layout (2.0): Core stays lean. Capability tools come from MCP (local-ai-mcp hard dep + host mcp.json). Optional extras are memory, ensemble, skills, monitor, and autotune — wired in app.py when installed and config-enabled.

local-ai-core dependencies:

• llama-tools — planner/doctor stack (transitive: gguf, psutil, py-cpuinfo, httpx, pyyaml)
• local-ai-mcp — lazy MCP bridge (stdio + HTTP)
• httpx — async HTTP client to llama-server
• jsonschema — JSON Schema validation for tool args
• pyyaml — YAML configuration
• jinja2 — per-agent prompt_files template stack
• json-repair — salvageable tool-call JSON in ToolParser cascade (no regex extraction)

Optional validation:

• pydantic — optional stricter config validation when installed (config/pydantic_models.py)

Optional extras (separate packages):

• local-ai-memory, local-ai-ensemble, local-ai-skills, local-ai-monitor, local-ai-autotune, local-ai-quantize (via llama-tools[quantize])

Embed via SessionRunner or use built-in interfaces (local-ai cli, local-ai serve-stdio, local-ai health).

Why These?

1. Well-maintained - Active development
2. Well-tested - Production-grade
3. Community-supported - Bug fixes, contributions
4. No forks - Use upstream, not forks
5. No enterprise - No proprietary dependencies

Avoiding Frameworks

No LangChain:

• Too heavy for simple use cases
• Cloud dependencies
• Proprietary components
• Bloated dependencies

No LlamaIndex:

• Overkill for local use
• External dependencies
• Heavy memory footprint

No LangGraph:

• Unnecessary complexity
• State management already handled
• No distributed features needed

Modular Monorepo Principles

Optional by Default

Pattern:

• Core ships with native recovery tools, session persistence, supervised llama-server, ReAct loop, and MCP bridge
• Extended stacks (memory, ensemble, skills, monitor, autotune) are separate pip packages activated by install + config flag
• Capability tools come from host mcp.json via hard-dep local-ai-mcp
• Protocol interfaces with null implementations (NullMemoryRetriever, NullSkillsRetriever) keep core tests and minimal installs working without optional deps
• Single AppContext / SessionRunner / AgentLoop — no duplicated agent logic across interfaces or extras

Why:

• Minimal install for embedding and CI
• Clear security boundaries (DirectoryGuard, MCP requires_approval, host-owned mcp.json)
• Users pay dependency and complexity cost only for features they enable

---

Project Scaling Principles

Design for Growth

Current scale:

• Single or multi-model (ensemble) via optional [ensemble] extra
• Capability tools via MCP (mcp.json); native recovery tools in Core
• Markdown memory (optional [memory] extra) with L0–L3 runtime state and indexed RAG in memory_data/
• Built-in crash/OOM GracefulDegradation tiers in core; optional [monitor] extra adds proactive RAM/VRAM polling

Future considerations:

• Database memory (optional, alongside file-based L0–L3)
• Distributed execution across instances
• Richer multi-agent coordination

Scaling Guidelines

Horizontal scaling:

• Multiple instances of app
• Shared state via filesystem
• Session isolation

Vertical scaling:

• More RAM for larger models
• More GPU for faster inference
• More CPU for parallel processing

Tool scaling:

• Register tools dynamically
• Validate each tool
• Rate limit if needed

Memory scaling:

• Category-based organization
• User-based isolation
• Index-based retrieval

Backward Compatibility

Configuration:

• Forward-compatible (extra keys ignored)
• Backward-compatible (old configs work)
• Migration paths documented

API:

• Versioned endpoints
• Deprecation warnings
• Migration guides

Storage:

• Versioned format
• Migration scripts
• No breaking changes

Documentation as Code

Self-documenting:

• Type hints everywhere
• Docstrings for public APIs
• Examples in code

External documentation:

• Architecture docs
• API reference
• User guides
• Examples

Decision Rationale

Why Not SQLite?

1. File-based is sufficient - JSON works for our needs
2. No locking complexity - Avoid database locking
3. Easier to backup - Simple file copy
4. No migrations - Schema changes are code changes
5. Debugging - Readable data without tools

Why Not JSON-RPC?

1. HTTP is standard - Everyone knows HTTP
2. Browser compatibility - Works in browsers
3. Proxy support - Works through proxies
4. Firewall friendly - Port 80/443
5. Standard tools - curl, Postman, etc.

Why Not gRPC?

1. HTTP is simpler - Easier to debug
2. Language agnostic - Any language can consume
3. Standard tools - curl, wget, etc.
4. No code generation - Less build complexity
5. Better error messages - HTTP status codes

Trade-offs Acknowledged

Simplicity vs. Features

We chose:

• Simple, understandable code
• Fewer features initially
• Easy to maintain
• Easy to extend

We accept:

• Manual configuration
• Less automation
• More manual work

Performance vs. Clarity

We chose:

• Clear, readable code
• Explicit over implicit
• Easy to understand

We accept:

• Some performance overhead
• More verbose code
• Less abstraction

Flexibility vs. Type Safety

We chose:

• Type hints for clarity
• Runtime validation
• JSON Schema for tools

We accept:

• Some runtime errors
• Manual validation needed
• Type checking not compile-time

Future Considerations

When to Evolve

Add features when:

• Clear need exists
• Well-understood use case
• Minimal impact on existing
• Backward compatible

Don't add features when:

• Solving non-existent problems
• Adding complexity for edge cases
• Breaking existing functionality
• Introducing new dependencies unnecessarily

Evolution Path

Phase 1 (Current — 2.0 monorepo):

• Pip monorepo under packages/ with optional extras wired via protocols and create_app()
• Local agent with MCP tools + native recovery; optional extras: memory, ensemble, skills, monitor, autotune, quantize
• Markdown memory: L0–L3 runtime state + indexed RAG in memory_data/; BM25 default, optional hybrid local embeddings
• Multi-model ensemble (EnsembleRouter); multi-agent orchestration via host MCP if needed
• Built-in GracefulDegradation (crash/OOM tier recovery); optional [monitor] for proactive RAM/VRAM pressure
• GGUF quantization via llama-tools quantize / local-ai-quantize
• MTP speculative decoding via llama.models.*.mtp
• Per-agent Jinja prompt_files (host-owned); mode-aware bundled system.md / system_native.md fallback when omitted
• Message archive + load_message; large tool results externalized to L3 + load_tool_result
• Optional workspace root for skills storage (no Core workspace file index; project FS via MCP)
• Session checkpoint/resume (--resume), token-budget context trim
• CLI: local-ai cli, local-ai serve-stdio, local-ai serve-http ([http]), local-ai health
• Standalone planner tools: llama-tools doctor, plan, inspect, benchmark, quantize
• SessionRunner embedding for host applications

Phase 2 (Future):

• Optional database-backed memory (alongside file-based L0–L3)
• Deeper multi-agent coordination beyond single-level delegation
• Distributed execution preparation (shared filesystem state, session isolation)

Phase 3 (Later):

• Federated or distributed local inference
• Cross-instance knowledge sync without cloud APIs

Core Values

1. Local-first - Privacy and control
2. Minimal dependencies - Less complexity
3. Type safety - Catch errors early
4. Async-first - Non-blocking design
5. Security - Sandbox everything
6. Simplicity - Easy to understand
7. Extensibility - Easy to add features
8. Documentation - Well-documented

Summary

LevPRO AI runtime embodies a philosophy of pragmatic minimalism - building sophisticated AI agents with minimal dependencies, maximum local control, and careful architectural decisions. Every component serves a clear purpose, and every trade-off is intentional.

The goal is not to replace cloud-based AI solutions, but to provide an alternative that respects user privacy, reduces costs, and works entirely offline.