Process Supervisor Documentation

Overview

The ProcessSupervisor manages the llama-server process lifecycle, providing health monitoring, automatic restart, and graceful shutdown. It ensures the LLM service is always available for the agent.

Purpose

1. Start llama-server - Launch the llama-server process with correct parameters
2. Health monitoring - Periodically check process health
3. Automatic restart - Restart on crash or health failure
4. Graceful shutdown - Cleanly terminate on app shutdown
5. Backoff handling - Exponential backoff on restart failures

Responsibilities

1. Process management - Start, stop, monitor subprocess
2. Health checking - HTTP health endpoint monitoring
3. Watchdog loop - Continuous health monitoring
4. Restart logic - Automatic recovery from failures
5. Logging - Stream process output to log file

Dependencies

• subprocess - Process management
• httpx - Health check requests
• core/logging_config - Structured logging

Constraints

1. Health check interval - 5 seconds between checks
2. Health fail threshold - 3 consecutive failures triggers restart
3. Timeout - 60 seconds to become healthy after start
4. Backoff - Exponential: 1s → 2s → 4s → ... → 30s max
5. Single instance - Only one supervisor per app
6. Trust environment - httpx with trust_env=False to avoid SOCKS proxy issues

Extension Points

Adding a New Health Check

If llama-server adds a new health endpoint:

1. Update is_healthy() to check new endpoint
2. Add fallback if old endpoint deprecated

Adding a New Restart Strategy

If you want different restart behavior:

1. Modify _watchdog_loop() restart logic
2. Consider: different backoff, restart limits, notifications

Module Details

core/supervisor.py - ProcessSupervisor

Manages llama-server process lifecycle:

Attributes:

• _process - subprocess.Popen instance
• _health_fail_count - Consecutive health failures
• _backoff_seconds - Current backoff delay
• _stopping - Shutdown flag
• _base_url - llama-server HTTP base URL

Methods:

start()

async def start() -> None:
    self._stopping = False
    await self._start_process()
    await self._wait_for_health(timeout=60.0)  # Wait for health
    self._watchdog_task = asyncio.create_task(self._watchdog_loop())

stop()

async def stop() -> None:
    self._stopping = True
    Cancel watchdog task
    self._terminate_process()  # Graceful termination

is_healthy()

async def is_healthy() -> bool:
    response = await client.get(f"{base_url}/health")
    return response.status_code == 200

_watchdog_loop()

async def _watchdog_loop() -> None:
    while not self._stopping:
        await asyncio.sleep(5)  # 5s interval
        
        if process_crashed:
            needs_restart = True
            reason = "process_crash"
        elif health_check_failed:
            self._health_fail_count += 1
            if self._health_fail_count > 3:
                needs_restart = True
                reason = "health_fail_count"
        else:
            self._health_fail_count = 0
            self._backoff_seconds = 1.0
        
        if needs_restart:
            log_event("SUPERVISOR_RESTART", reason)
            self._terminate_process()
            await asyncio.sleep(self._backoff_seconds)
            self._backoff_seconds = min(self._backoff_seconds * 2, 30.0)
            await self._start_process()
            await self._wait_for_health(timeout=60.0)
            self._health_fail_count = 0

_start_process()

async def _start_process() -> None:
    if process already running:
        return
    
    log_file = logs_dir / "llama.log"
    self._process = subprocess.Popen(
        self._build_args(),
        stdout=log_file,
        stderr=subprocess.STDOUT,
        text=True,
    )

_build_args()

def _build_args() -> list[str]:
    return [
        binary_path,
        "--model", model.path,
        "--port", str(model.port),
        "--ctx-size", str(model.ctx_size),
        "--n-gpu-layers", str(model.gpu_layers),
        "--batch-size", str(model.n_batch),
        "--ubatch-size", str(model.n_ubatch),
        "--threads", str(model.threads),
        "--threads-batch", str(model.threads_batch),
        "--flash-attn", model.flash_attn,
        # Optional cache types
    ]

Usage Flow

App Startup

# app.py
supervisor = ProcessSupervisor(binary_path, model_config)
await supervisor.start()
    - Starts llama-server
    - Waits for health (60s timeout)
    - Starts watchdog task

Agent Request

# api.py, cli.py, tui.py
if not await ctx.supervisor.is_healthy():
    raise HTTPException(503, "llama-server is not healthy")

App Shutdown

# interfaces/*.py
finally:
    await ctx.shutdown()
        - Cancels watchdog task
        - Terminates llama-server gracefully

Health Check Logic

Normal operation:

• Health check every 5s
• On success: reset fail counter to 0, backoff to 1s

Crash detected:

• process.poll() is not None (exit code set)
• Immediate restart with current backoff
• Log "restart_reason=process_crash"

Health failure:

• 3 consecutive failures → restart
• Log "restart_reason=health_fail_count"
• Reset counter after successful restart

Backoff progression:

• 1s → 2s → 4s → 8s → 16s → 30s (max)

Restart Scenarios

Scenario 1: Process Crashes

Time 0: Process starts
Time 5: Health check → OK
Time 10: Process crashes (segfault)
Time 10: Watchdog detects crash
Time 10: Terminate, backoff 1s
Time 11: Start new process
Time 11: Wait for health (60s)
Time 16: Health check → OK
Time 16: Reset fail counter, backoff 1s

Scenario 2: Health Check Fails

Time 0: Process starts
Time 5: Health check → OK
Time 10: Health check → FAIL (count=1)
Time 15: Health check → FAIL (count=2)
Time 20: Health check → FAIL (count=3) → RESTART
Time 20: Terminate, backoff 1s
Time 21: Start new process
...

Scenario 3: Shutdown

Time 0: shutdown() called
Time 0: Set _stopping = True
Time 0: Cancel watchdog task
Time 0: Terminate process gracefully
Time 0: Close log file

Configuration

Parameters controlled by ModelPlanner:

• binary_path - llama-server executable
• model.path - GGUF model
• model.port - HTTP port
• model.ctx_size - Context size
• model.gpu_layers - GPU offloading
• model.n_batch - Batch size
• model.n_ubatch - Micro-batch size
• model.threads - CPU threads
• model.threads_batch - Batch threads
• model.flash_attn - Flash attention

Logging

Log file: logs/llama.log

Events logged:

• "Starting llama-server: <args>" - Process start
• "SUPERVISOR_RESTART" - Restart event (reason, fail_count)

stdout/stderr: All llama-server output goes to log file

Error Handling

Startup Failure

Timeout (60s):

raise TimeoutError("llama-server failed to become healthy within timeout")

Health Check Failure

HTTPError:

• Returns False (treated as unhealthy)
• Triggers health fail counter increment

Network issues:

• Returns False
• May trigger restart if persistent

Testing

Key test cases:

• Process starts and becomes healthy
• Process crashes and restarts
• Health failures trigger restart
• Shutdown terminates cleanly
• Backoff progression
• Concurrent health checks

Run: python -m pytest packages/local-ai-core/tests/test_supervisor.py -v

Architecture Rules for Supervisor

1. Single instance - One supervisor per app context
2. Health monitoring - Always monitor process health
3. Automatic recovery - Restart on crash/failure
4. Graceful shutdown - Clean termination on stop
5. Exponential backoff - Prevent rapid restart loops
6. Log all events - Supervision events to structured logs

Integration with App Context

# app.py
supervisor = ProcessSupervisor(binary_path, model_config)
llm_client = LocalLLMClient(base_url=supervisor.base_url)

# interfaces/*.py
ctx = create_app()
await ctx.supervisor.start()
    - Starts supervisor
    - Waits for health
    - Starts watchdog
try:
    # Run agent
finally:
    await ctx.shutdown()
        - Cancels watchdog
        - Terminates process