Model Planner Documentation

Overview

The ModelPlanner is the single source of truth for all llama-server launch parameter calculations. It reads GGUF model metadata and system hardware information to compute optimal launch parameters that fit available resources.

Purpose

The ModelPlanner determines the best llama-server launch configuration for a given model and system, ensuring:

• The model fits in available RAM and VRAM
• Context size is appropriate for the mode and hardware
• GPU offloading is maximized without overflow
• Batch sizes are optimal for performance
• CPU allocation is efficient

Responsibilities

1. Read GGUF metadata - Parse model architecture, quantization, and size
2. Detect system hardware - RAM, CPU cores, GPU backend and VRAM
3. Compute launch parameters - ctx_size, gpu_layers, batch sizes, threads, flash_attn
4. Validate feasibility - Ensure parameters fit in available memory
5. Generate estimates - RAM/VRAM usage, tokens per second
6. Provide warnings - When requested parameters must be reduced

Dependencies

• gguf - GGUF metadata parsing
• psutil - System information
• cpuinfo - CPU feature detection
• llama_tools/planner/types.py - Dataclasses for results

Constraints

1. Single source of truth - All calculations must go through ModelPlanner
2. No duplicate calculations - Doctor, interfaces must use Planner results
3. Fail-safe - If model won't fit, return can_run=false with suggestions
4. Overrides validation - User overrides are validated and degraded when needed
5. Minimum context - Never allow ctx_size below MIN_CTX (512 tokens)

Extension Points

Adding a New Model Type

If supporting a new model format:

1. Add parsing logic to GgufMetadataReader
2. Update ModelMetadata dataclass if needed
3. Ensure GGUF format is maintained (llama.cpp standard)

Adding a New Optimization Mode

1. Add to OptimizationMode enum in types.py
2. Add batch defaults in MODE_DEFAULTS in estimator.py
3. Add target context fraction in mode_target_ctx_fraction()
4. Add flash_attn logic in flash_attn_for_mode()

Adding Hardware Detection

1. Add detection function in SystemProbe
2. Update SystemInfo dataclass if needed
3. Use detection results in max_gpu_layers() or fits_in_memory()

Module Details

llama_tools/planner/planner.py - ModelPlanner

The main orchestration class that:

• Reads or uses provided metadata
• Detects system (or uses provided)
• Applies user overrides with validation
• Computes optimal parameters
• Validates memory fit
• Returns PlanResult

Key methods:

• plan() - Main planning method

llama_tools/planner/gguf_reader.py - GgufMetadataReader

Reads GGUF metadata from model files:

• Architecture, layers, parameters
• Embedding length, context length
• Quantization type
• File size (handles sharded models)

llama_tools/planner/system_probe.py - SystemProbe

Detects system hardware:

• RAM available/total
• CPU physical and logical cores
• AVX2/AVX-512 support
• Accelerator backend (CUDA, ROCm/HIP on Linux and Windows, Vulkan, Metal) through provider probes
• GPU VRAM total/free
• All detected accelerators plus whether memory availability is measured or estimated
• GPU name

llama_tools/planner/storage_probe.py - StorageProbe

Path-scoped probe for the volume that holds the model GGUF (Windows / macOS / Linux):

• Whether the volume is an SSD (is_ssd; unknown → fail-closed, no budget expand)
• Free / total disk space on that volume
• OS swap / pagefile totals via psutil.swap_memory()

Attached onto SystemInfo.model_volume during ModelPlanner.plan().

llama_tools/planner/estimator.py - Estimator

Contains all calculation formulas:

• split_memory() / split_memory_detailed() - Calculate RAM/VRAM split (and CPU pinned vs weights)
• fits_in_memory() - Check if parameters fit (physical RAM + optional SSD virtual RAM)
• effective_ram_budget_mb() / ssd_virtual_ram_mb() - SSD-backed budget helpers
• max_ctx_that_fits() - Find max context
• max_gpu_layers() - Find max GPU layers
• estimate_speed() - Estimate tokens/sec
• build_estimate() - Build PlanEstimate
• build_minimum_resources() - Minimum requirements

llama_tools/planner/types.py - Dataclasses

Type definitions:

• ModelMetadata - GGUF model metadata
• ModelVolumeInfo - Model path volume (SSD / free space / swap)
• SystemInfo - System hardware info
• LaunchParams - Launch parameters
• LaunchOverrides - User overrides
• PlanEstimate - Resource estimates
• PlanWarning - Parameter warnings
• Recommendation - Advice messages
• PlanResult - Complete planning result
• MinimumResources - Minimum requirements

Calculation Details

Memory Split

The model and KV cache are split between RAM and GPU:

GPU:
  - GPU layers weights
  - Proportional KV cache (same ratio as layers)
  - GPU buffers for computation

RAM:
  - CPU layers weights
  - Remaining KV cache
  - CPU buffers

SSD-backed virtual RAM

When the model path is on an SSD, free disk space can expand the effective RAM budget so models larger than physical RAM may still plan as runnable:

physical_budget = ram_available_mb - os_reserve_mb
virtual_budget  = max(0, volume_free_mb - disk_reserve_mb)   # SSD only; else 0
effective_budget = physical_budget + virtual_budget

Rules:

• Non-SSD / unknown media → no expansion (fail-closed).
• CPU-side KV + compute buffers must fit in physical_budget; only CPU-resident weights may spill to SSD (mmap / paging).
• VRAM accounting is unchanged.
• Config: llama.disk_reserve_mb (default 4096), same spirit as os_reserve_mb.
• When virtual RAM is required, planner emits a PlanWarning on field ram_budget.

GPU Layers Calculation

Uses binary search to find maximum GPU layers that fit:

1. Start with requested or full offload
2. Test if parameters fit
3. Binary search for maximum
4. Respect VRAM safety margin (90%)

Context Size Calculation

1. Mode target (25%/50%/100% of model context)
2. Maximum that fits in memory
3. User override (if provided)
4. Round down to power of 2
5. Minimum MIN_CTX (512 tokens)

Batch Sizes

Mode-dependent defaults:

• SPEED: n_batch=2048, n_ubatch=512
• BALANCED: n_batch=1024, n_ubatch=256
• QUALITY: n_batch=512, n_ubatch=128

Adjusted to fit memory, halved if needed.

Speed Estimation

Based on:

• Quantization (Q4_K fastest, F32 slowest)
• GPU offload ratio
• AVX2/AVX-512 support
• Context size penalty (sqrt scaling)
• Flash attention (if available)

Usage Examples

Basic Planning

from core.planner import ModelPlanner

planner = ModelPlanner()
plan = planner.plan(
    model_path="C:/llama/model.gguf",
    mode="balanced",
)

With Overrides

from core.planner import LaunchOverrides

overrides = LaunchOverrides(ctx_size=4096, threads=8)
plan = planner.plan(
    model_path="C:/llama/model.gguf",
    mode="quality",
    overrides=overrides,
)

Programmatic Planning

from core.planner import ModelPlanner, LaunchOverrides

planner = ModelPlanner(os_reserve_mb=4096)

# Plan for model
plan = planner.plan(
    model_path="C:/llama/model.gguf",
    mode="balanced",
    overrides=LaunchOverrides(threads=12),
)

if plan.can_run:
    print(f"Recommended: ctx={plan.recommended.ctx_size}, "
          f"gpu={plan.recommended.gpu_layers}")
else:
    print(f"Cannot run: {plan.reason}")
    for suggestion in plan.minimum_required.suggestions:
        print(f"  - {suggestion}")

Launch Parameters

ctx_size

Context window size (tokens). Must be power of 2, minimum 512.

gpu_layers

Number of layers to offload to GPU. Maximum is model block_count.

n_batch

Batch size for processing. Affects throughput and memory.

n_ubatch

Micro-batch size for flash attention. Must be ≤ n_batch.

threads

Number of CPU threads for inference.

threads_batch

Threads for batched requests (same as threads by default).

flash_attn

Flash attention setting: "on", "off", or "auto".

cache_type_k / cache_type_v

KV cache data type: f16, q8_0, q4_0, etc.

Performance Estimates

PlanEstimate provides:

• ram_mb - Estimated RAM usage
• vram_mb - Estimated VRAM usage
• kv_cache_mb - KV cache memory
• tokens_per_second_min/max - Generation speed range

Warnings

PlanWarning is generated when:

• Requested parameter exceeds available resources
• GPU layers must be reduced
• Context size must be reduced
• Batch sizes must be reduced
• Plan relies on SSD-backed virtual RAM / mmap for CPU weights

Example:

PlanWarning(
    field="gpu_layers",
    message="Requested gpu_layers=35 exceeds VRAM; reduced to 28",
    requested=35,
    resolved=28,
)

Recommendations

Recommendation provides advice:

• code - Machine-readable identifier
• message - Human-readable message
• severity - "info", "suggestion", "warning", "error"
• field - Optional link to specific parameter

Integration Points

App Context

The planner is used in app.py to compute resolved model config:

planner = ModelPlanner(os_reserve_mb=config.llama.os_reserve_mb)
plan = planner.plan(...)

recommended = plan.recommended
model_config = ResolvedLlamaModelConfig(
    ctx_size=recommended.ctx_size,
    gpu_layers=recommended.gpu_layers,
    # ...
)

Doctor Interface

Doctor uses Planner (not its own calculations):

planner = create_planner()
doctor = Doctor(planner)
result = doctor.check_model(metadata, system, mode)

Plan Interface

Plan preview uses Planner directly:

planner = ModelPlanner(os_reserve_mb=config.llama.os_reserve_mb)
plan = planner.plan(...)

Testing

Key test cases:

• Model fits with various configurations
• Model doesn't fit (RAM/VRAM insufficient)
• Override validation and degradation
• GPU detection (with/without GPU)
• Context size optimization
• Batch size fitting
• Speed estimation accuracy

Run: python -m pytest packages/llama-tools/tests/test_model_planner.py -v

MTP (Multi-Token Prediction)

When LaunchOverrides.mtp_enabled=True (from llama.models.main.mtp.enabled):

1. Planner adds MTP warnings and MTP_ENABLED recommendation
2. Embedded MTP (no draft_path): adds MTP_VRAM_OVERHEAD_MB (2500 MB heuristic) to VRAM estimate
3. Separate draft (draft_path): reads draft GGUF, computes mtp_spec_draft_ngl via draft_ngl_for_remaining_vram()
4. If combined VRAM exceeds budget → can_run=False

Config keys: mtp.enabled, mtp.spec_draft_n_max (2 or 3), mtp.draft_path (optional).

Shared parsing lives in llama_tools/llama_config_parse.py.

Architecture Rules for Planner

1. Single Source of Truth - All calculations must be in Planner
2. No duplicate calculations - Doctor, interfaces must use Planner results
3. Fail-fast - If model won't fit, return can_run=false immediately
4. Graceful degradation - Reduce parameters when requested values don't fit
5. Document assumptions - All formulas and heuristics should be documented
6. Test-driven - Add tests for new calculations