Merge pull request #324 from lbedner/overseer-docs

Overseer Docs

Author: lbedner

README.md

@@ -8,6 +8,7 @@
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![Commits per Month](https://img.shields.io/github/commit-activity/m/lbedner/aegis-stack)](https://github.com/lbedner/aegis-stack/commits)
 [![Total Commits](https://img.shields.io/github/commit-activity/t/lbedner/aegis-stack)](https://github.com/lbedner/aegis-stack/commits)
+[![Monthly Downloads](https://img.shields.io/pypi/dm/aegis-stack)](https://pypi.org/project/aegis-stack/)
 [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json)](https://github.com/copier-org/copier)
 
 You need to ship reliable software, but management only gave you 2 weeks.
@@ -20,6 +21,11 @@ No time for health checks, proper testing, or clean architecture. Just enough ti
 
 Aegis Stack is a modular Python framework that grows with your projects — start with an API, add Auth, Scheduler, Workers, or AI services when you need them.
 
+## Prerequisites
+
+- **Python 3.11+**
+- **Docker & Docker Compose** - Required for the standard development workflow (`make serve`). Generated projects use Docker for consistent environments and service dependencies (Redis for workers, health monitoring, etc.).
+
 ## Quick Start
 
 ```bash
@@ -67,11 +73,13 @@ Most frameworks lock you in at `init`. Aegis Stack doesn't. See **[Evolving Your
 
 ## See It In Action
 
-### Overseer
+### Overseer - Built-In Health Monitoring
 
 ![Overseer](docs/images/overseer-dashboard-1-dark.png)
 
-Real-time monitoring dashboard showing how every component and service is performing with detailed health metrics and status information.
+**[Overseer](docs/overseer/index.md)** is the read-only health monitoring dashboard built into every Aegis Stack project. It provides real-time visibility into all your components (Backend, Database, Worker, Scheduler) and services (Auth, AI, Comms) through a web UI and CLI commands.
+
+No Datadog. No New Relic. No vendor lock-in. Just centralized monitoring you own from day one.
 
 ### CLI Health Monitoring
 

docs/about.md

@@ -26,7 +26,7 @@ Eventually, I started noticing the same patterns everywhere. Every project neede
 
 Years ago, I had a boss who got it. We'd sit in one of the conference rooms dreaming about automation tools, sketching ideas, making plans. He even started working on something on the side. But that manager was long gone by the time I brought it to fruition. 
 
-Post-pandemic, I discovered Streamlit and got tired of waiting for permission. So I built what I would later name, Overseer, myself. It actually worked - I actively used it to automate monotonous domain-specific tasks that were eating up our days.
+Post-pandemic, I discovered Streamlit and got tired of waiting for permission. So I built what I would later name, **[Overseer](overseer/index.md)**, myself. It actually worked - I actively used it to automate monotonous domain-specific tasks that were eating up our days.
 
 That experience taught me what happens when you stop waiting for permission and just build the tools you need.
 

docs/components/webserver.md

@@ -72,6 +72,11 @@ async def register_middleware(app: FastAPI) -> None:
 
 **No registration required** - just drop the file and restart. See the [Integration Patterns](../integration-patterns.md) for complete details.
 
+!!! example "Musings: Backend Middleware Auto-Discovery (November 22nd, 2025)"
+    I'm not sure how I ultimately feel about the auto-discovery middleware pattern. I don't have enough experience with FastAPI plugins yet, but it's something I'm thinking about as the architecture evolves.
+
+    The current approach works well for explicit component registration, but auto-discovery could reduce boilerplate at the cost of making the registration flow less obvious. Trade-offs worth considering as Aegis Stack matures.
+
 ## Integration
 
 FastAPI integrates with your application and provides:

docs/images/druski-shrug.gif

@@ -2,6 +2,32 @@
 
 Aegis Stack can be used in multiple ways depending on your needs and preferences.
 
+## System Requirements
+
+Before installing Aegis Stack, ensure you have the following:
+
+### Required
+
+- **Python 3.11 or higher** - Core runtime for Aegis Stack and generated projects
+- **Docker & Docker Compose** - Required for generated projects' development workflow
+
+### Why Docker?
+
+Generated projects use Docker for:
+
+- **Consistent development environments** - Same setup across all machines
+- **Service dependencies** - Redis for worker component, health monitoring infrastructure
+- **Standard workflow** - The `make serve` command uses `docker compose` under the hood
+- **Production parity** - Development closely mirrors production deployment
+
+!!! note "Docker Alternatives"
+    While the standard workflow uses Docker, generated projects are standard Python applications. Advanced users can manually run components (uvicorn for backend, direct Redis installation, etc.), but this workflow is currently undocumented and unsupported.
+
+### Installing Docker
+
+- **macOS/Windows**: [Docker Desktop](https://www.docker.com/products/docker-desktop/)
+- **Linux**: [Docker Engine](https://docs.docker.com/engine/install/) + [Docker Compose](https://docs.docker.com/compose/install/)
+
 ## Installation
 
 Choose the method that works best for your workflow:

docs/images/illiana_overseer_transparent.png

@@ -0,0 +1,135 @@
+# Overseer
+
+## Why This Exists
+
+**Nothing is more annoying than the shrug.**
+
+![Druski Shrug](../images/druski-shrug.gif)
+
+Something's broken in production. You ask what happened. You get a shrug. You ask when it started. Another shrug. You ask where the logs are. Shrug. You ask how we can fix it. The biggest fucking shrug you've ever seen.
+
+If something is wrong, I want to know **where**, **when**, **how**, **why**, and **how can we reconcile it**. Christ! Is that too much to ask?
+
+**It shouldn't be so fucking hard to know what happened, when, where.**
+
+You work with Datadog until management decides to migrate to New Relic. Or you're a solo dev who just wants to see if your background jobs are running without paying enterprise prices. Overseer solves this: centralized monitoring that you own, built into every Aegis Stack project from day one.
+
+## What It Is
+
+**Overseer is a read-only health monitoring dashboard** built into your Aegis Stack application. It provides real-time visibility into component and service health through a web UI and CLI commands.
+
+![Overseer Dashboard](../images/overseer-dashboard-1-dark.png)
+
+The dashboard displays:
+
+- **Component Cards**: Backend, Database, Worker, Scheduler health
+- **Service Cards**: Auth, AI, Comms health (when included)
+- **Header**: Overall health summary and theme toggle
+- **Auto-refresh**: Polls health endpoint every 30 seconds
+
+## Current Capabilities
+
+- Component health monitoring (Backend, Database, Worker, Scheduler)
+- Service health monitoring (Auth, AI, Comms)
+- System metrics (CPU, memory, disk usage)
+- Status hierarchy (Healthy, Warning, Unhealthy, Info)
+- Web dashboard with auto-refresh (30-second polling)
+- CLI health commands via your generated app
+
+## How It Works
+
+```mermaid
+sequenceDiagram
+    participant C as Components/Services
+    participant R as Health Registry
+    participant E as /health/ Endpoint
+    participant D as Dashboard UI
+
+    Note over C,R: Startup: Registration Phase
+    C->>R: register_health_check("backend", check_func)
+    C->>R: register_health_check("database", check_func)
+    C->>R: register_service_health_check("auth", check_func)
+
+    Note over E,D: Runtime: Monitoring Phase
+    D->>E: GET /health/ (every 30s)
+    E->>R: Run all registered checks
+    R->>C: Execute health check functions
+    C->>R: Return ComponentStatus
+    R->>E: Aggregate into SystemStatus
+    E->>D: Return health data
+    D->>D: Render component/service cards
+```
+
+**The Flow:**
+
+1. **Registration**: During app startup, components and services register their health check functions with the health registry
+2. **Aggregation**: The `/health/` endpoint runs all registered checks and aggregates results into a hierarchical status tree
+3. **Polling**: The dashboard polls the health endpoint every 30 seconds
+4. **Display**: Component and service cards render with real-time status, metrics, and details
+
+## Component & Service Cards
+
+Each card shows real-time health status, component-specific metrics, and configuration details. Click any card to open a detailed modal with diagnostics, performance data, and system information.
+
+## Health Status Indicators
+
+Each card displays a status indicator using the Overseer status hierarchy:
+
+| Status | Color | Visual | Meaning |
+|--------|-------|--------|---------|
+| **✅ Healthy** | Green | Solid green border | Component/service fully operational |
+| **ℹ️ Info** | Blue | Solid blue border | Informational status, not a problem |
+| **⚠️ Warning** | Yellow | Orange border | Operational but with issues |
+| **❌ Unhealthy** | Red | Red border | Component/service down or failing |
+
+**Status Propagation**: Parent components inherit the worst child status:
+
+- Any child **Unhealthy** → Parent **Unhealthy**
+- Any child **Warning** (no unhealthy) → Parent **Warning**
+- Any child **Info** (no unhealthy/warning) → Parent **Info**
+- All children **Healthy** → Parent **Healthy**
+
+## Theme Support
+
+The dashboard automatically adapts to light and dark themes:
+
+- **Light Mode**: White cards, dark text, subtle shadows
+- **Dark Mode**: Dark cards, light text, enhanced contrast
+- **Toggle**: Click the theme icon in the header to switch
+
+Images and status colors adjust automatically to maintain visibility in both themes.
+
+## CLI Health Access
+
+The same health data is accessible via CLI:
+
+```bash
+# View system health
+your-app health
+
+# Example output:
+┌────────────────────────────────────────┐
+│ System Health                          │
+├────────────────────────────────────────┤
+│ Components                             │
+│   ✅ backend    - FastAPI healthy      │
+│   ✅ database   - SQLite connected     │
+│   ✅ worker     - arq processing       │
+│   ✅ scheduler  - 3 jobs scheduled     │
+│                                        │
+│ Services                               │
+│   ✅ auth       - 42 users, HS256      │
+│   ✅ ai         - Anthropic/Claude     │
+└────────────────────────────────────────┘
+```
+
+## What's Coming
+
+Overseer is evolving into a full operational control plane. Want to know where this is headed and why I'm so confident it'll work?
+
+**[Read the full story →](story.md)** - How Overseer evolved from solving production problems at iHeartMedia (2022-2024) to becoming the built-in control plane for Aegis Stack.
+
+## Next Steps
+
+- **[The Overseer Story](story.md)** - Evolution from Streamlit to Aegis Stack, roadmap, and vision
+- **[Integration Guide](integration.md)** - Add health checks to custom components/services