Datasance · emirhandurmus · Jun 24, 2026 · Jun 24, 2026 · Jun 24, 2026 · Jun 24, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -71,7 +71,7 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 #### Database and distribution
 
 - **Greenfield schema** — **new install required**; no v3.7 → v3.8 database migrator.
-- PKI: central router/NATS local CAs; legacy per-agent CAs migrated via one-time **rotation job** (Plan 5).
+- PKI: central router/NATS local CAs; legacy per-agent CAs migrated via one-time **rotation job**.
 - **Node.js 24.x** required for dev and CI (was 16/18).
 - Dual-mirror container images: **`ghcr.io/eclipse-iofog/controller`** and **`ghcr.io/datasance/controller`** from the **same commit SHA**; publish on **`v*` tags only** via repo variable **`IMAGE_REGISTRY`**.
 
@@ -87,6 +87,17 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - Neutral in-tree identity: RBAC **`iofog.org/v3`**, default namespace **`iofog`**, `package.json` name **`controller`**.
 - NATS account/user rule **JWT Latin-1 validation** — rejects rules whose fields cannot be encoded in a NATS JWT (create/update on rules, applications, microservices, and NATS API).
 - RBAC **route catalog utils** (`isPublicCatalogRoute`) — shared lookup of public routes from `rbac-resources.yaml` (empty verb list = no auth required).
+- **Fog + service platform reconcile** — declarative router/NATS and service endpoint lifecycle replaces fire-and-forget `setImmediate` in `iofog-service.js` and `services-service.js`.
+- Tables: **`FogPlatformSpecs`**, **`FogPlatformStatuses`**, **`FogPlatformReconcileTasks`**, **`ServicePlatformReconcileTasks`**, **`HubRouterConfigLocks`** (greenfield migrations amended for sqlite, mysql, postgres).
+- **`platform-reconcile-worker-job.js`** — one worker, two DB-backed claim paths (fog + service); stale reclaim, exponential backoff, max attempts.
+- **`fog-platform-sweep-job.js`** — periodic drift detection for fog and service platform state.
+- **`GET /api/v3/iofog/{uuid}`** — optional **`platformStatus`** (`phase`, `generation`, `lastError`, conditions).
+- **`POST /api/v3/iofog/{uuid}/reconcile`** and **`POST /api/v3/services/{name}/reconcile`** — manual retry after failed or stuck reconcile.
+- Service **`provisioningStatus`** — hub semantics: **`ready`** when hub connector/listener and K8s Service reconcile succeed; edge TCP bridges converge asynchronously via fog platform reconcile fan-out.
+- **K8s control plane:** hub **`iofog-router`** ConfigMap patches serialized via DB lock; K8s Service create/update/delete with LoadBalancer watch timeout.
+- **`service-bridge-config.js`** — full recompute of service-derived TCP bridge config per fog on reconcile (preserves router base config).
+- **SQLite single-node production hardening** — WAL + `busy_timeout` pragmas, reconcile task claim retry on `SQLITE_BUSY`, staggered startup for reconcile-heavy background jobs (`settings.jobStartupDelaySeconds`).
+- **WebSocket exec & log session hardening** — quotas (1 exec / 3 log WS per resource), exec_b lifecycle, 60s/120s pending timeouts, 8h exec max, 30s graceful drain, OTEL metrics, HA AMQP fail-fast, integration tests, swagger WS protocol docs, operator guide (`docs/operations/ws-sessions.md`).
 
 ### Fixed
 
@@ -106,6 +117,13 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - Router microservice **`siteConfig.platform`** defaults to **`edgelet`** (was **`docker`**) when the agent uses the Edgelet runtime.
 - Boolean env vars (`TRUST_PROXY`, `SERVER_DEV_MODE`, `DB_USE_SSL`, `VAULT_ENABLED`, `ENABLE_TELEMETRY`, and other mapped flags) are parsed consistently from Kubernetes string values (`true`/`false`, `1`/`0`) via shared **`config.getBoolean()`** — fixes startup crash when **`TRUST_PROXY=true`** was passed as a string to Express.
 - Postgres/MySQL SSL reads canonical **`DB_SSL_CA`** (via config) instead of undocumented **`DB_SSL_CA_B64`**; **`database.*.useSSL`** config key honored (was **`useSsl`** typo).
+- Spurious **`routerMode`** / **`natsMode`** **`none`** on fog list/get when runtime rows were pending — read path now falls back to **`FogPlatformSpecs`** during reconcile.
+- **`PATCH /api/v3/iofog/{uuid}`** on system fogs with full config (potctl redeploy) — **400** **`Invalid NATS mode 'undefined'`** when `natsMode` was omitted from PATCH body.
+- Partial fog delete orphans when router/NATS teardown failed mid-flight — delete is async via platform reconcile **`Deleting`** phase.
+- Service provisioning races and lost hub ConfigMap updates under multi-Controller — serialized hub lock and DB-backed service reconcile tasks.
+- Dual writers to router microservice bridge config from fog create/update and service create/update/delete — single full-recompute path on fog reconcile.
+- SQLite startup lock contention on single-controller deployments — WAL + `busy_timeout` pragmas on connect, `withDbBusyRetry` on fog/service/NATS task claims, staggered reconcile-heavy job startup.
+- **`reconcileFog` transaction parameter** — removed unused `options` argument so worker-decorated calls receive the transaction correctly.
 
 ### Changed
 

diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-# Stage 1 — EdgeOps Console static SPA (Plan 11-1)
+# Stage 1 — EdgeOps Console static SPA 
 # ioFog overrides: EDGEOPS_CONSOLE_REPO=https://github.com/eclipse-iofog/edgeops-console
 #                   EDGEOPS_CONSOLE_FLAVOR=iofog
 # node:24-bookworm — pin manifest list digest for reproducible multi-arch builds

diff --git a/docs/architecture.md b/docs/architecture.md
@@ -137,8 +137,128 @@ Full request/response shapes: **`docs/swagger.yaml`** (agent paths).
 | Controller cleanup | `controller-cleanup-job.js` | Orphaned controller MS housekeeping |
 | Event cleanup | `event-cleanup-job.js` | Audit event retention |
 | NATS reconcile | `nats-reconcile-worker-job.js` | NATS operator sync |
+| Platform reconcile | `platform-reconcile-worker-job.js` | Fog + service platform claim/reconcile (one job, two queues) |
+| Fog platform sweep | `fog-platform-sweep-job.js` | Drift detection; re-enqueue stale fog/service tasks |
 | Stopped app status | `stopped-app-status-job.js` | Application state maintenance |
 
+### Platform reconcile (fog + service + resolver)
+
+Router/NATS **fog platform lifecycle**, **service endpoint provisioning**, and the existing **NATS resolver** layer are three separate reconcile queues — not fire-and-forget `setImmediate` blocks in `iofog-service.js` / `services-service.js`.
+
+```mermaid
+flowchart TB
+  subgraph api [Synchronous API]
+    FogAPI[POST/PATCH/DELETE /iofog]
+    SvcAPI[POST/PATCH/DELETE /services + yaml]
+    FogAPI --> FSpec[Upsert FogPlatformSpecs]
+    SvcAPI --> SDB[Write Services + tags]
+    FSpec --> FEnqueue[FogPlatformReconcileTasks]
+    SDB --> SEnqueue[ServicePlatformReconcileTasks]
+  end
+
+  subgraph worker [One job — any Controller replica]
+    ClaimF[claimNextFogTask]
+    ClaimS[claimNextServiceTask]
+    ReconcileF[FogPlatformService.reconcileFog]
+    ReconcileS[ServicePlatformService.reconcileService]
+    HubLock[Hub ConfigMap lock]
+    ClaimF --> ReconcileF
+    ClaimS --> HubLock
+    HubLock --> ReconcileS
+  end
+
+  subgraph runtime [Observed state]
+    Routers[(Routers)]
+    Nats[(NatsInstances)]
+    MS[System MS + secrets]
+    K8sSvc[K8s Service]
+    CM[ConfigMap iofog-router]
+  end
+
+  subgraph resolver [NATS resolver — unchanged]
+    NRT[NatsReconcileTasks]
+  end
+
+  FEnqueue --> ClaimF
+  SEnqueue --> ClaimS
+  ReconcileF --> Routers
+  ReconcileF --> Nats
+  ReconcileF --> MS
+  ReconcileS --> CM
+  ReconcileS --> K8sSvc
+  ReconcileS -->|fan-out service-changed| FEnqueue
+  ReconcileF -->|topology change| NRT
+```
+
+| Layer | Table | Worker | Purpose |
+|-------|-------|--------|---------|
+| **Fog platform** | `FogPlatformReconcileTasks` | Same job: `claimNextFogTask` | Router/NATS instances, PKI, system MS, **full recompute** of service-derived TCP bridges per fog |
+| **Service platform** | `ServicePlatformReconcileTasks` | Same job: `claimNextServiceTask` | Hub connector/listener, K8s Service, ConfigMap (DB lock), fan-out fog tasks on tag change |
+| **NATS resolver** | `NatsReconcileTasks` | `nats-reconcile-worker-job.js` | JWT bundles, account/user creds after app deploy |
+
+**Fog operator API:** `GET /iofog/{uuid}` includes optional **`platformStatus`** (`phase`, `generation`, `lastError`); list/get derive `routerMode`/`natsMode` from **`FogPlatformSpecs`** when runtime rows are pending. **`POST /iofog/{uuid}/reconcile`** for manual retry.
+
+**Service operator API:** JSON + YAML create/update/delete enqueue service reconcile; **`provisioningStatus=ready`** marks hub complete (K8s Service + hub ConfigMap); edge listeners converge via fog fan-out. **`POST /services/{name}/reconcile`** for manual retry.
+
+Full spec: [`.cursor/controllerv3.8/docs/15-fog-platform-reconcile.md`](../.cursor/controllerv3.8/docs/15-fog-platform-reconcile.md) · RFC R69–R79.
+
+---
+
+## WebSocket exec & log sessions
+
+Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens lifecycle, quotas, multi-replica HA relay, and observability — **without changing the Edgelet wire protocol**.
+
+```mermaid
+sequenceDiagram
+  participant U as User WS
+  participant C as Controller replica
+  participant DB as Database
+  participant Q as AMQP Router
+  participant A as Agent WS
+
+  Note over U,A: Exec (R80–R81, R84)
+  U->>C: WS exec (RBAC)
+  A->>C: WS agent/exec + execId
+  C->>C: Pair SessionManager
+  C->>Q: Enable agent-{execId} user-{execId}
+  U->>C: STDIN
+  C->>Q->>A: or direct WS same replica
+  A->>C: STDOUT
+  C->>Q->>U: relay
+  U-->>C: close
+  C->>DB: execEnabled=false INACTIVE
+
+  Note over U,A: Logs (R82–R83, R84)
+  U->>C: WS logs + tail params
+  C->>DB: PENDING sessionId
+  A->>C: WS agent/logs/:sessionId
+  A->>C: LOG_LINE
+  C->>Q->>U: logs-user-{sessionId}
+```
+
+| Topic | Normative value (RFC R80–R91) |
+|-------|-------------------------------|
+| Exec lifecycle | **exec_b** — WS close sets `execEnabled=false`; **1** user exec WS per microservice |
+| Exec timeouts | **60s** pending for agent; **8h** max active session |
+| Log concurrency | **3** user log WS per microservice (or per fog for node logs) |
+| Log limits | Tail max **5,000** lines; **120s** pending; **2h** idle |
+| Log content | Live relay only — no log line persistence; audit connect/disconnect |
+| HA relay | Cross-replica sessions **require** AMQP (`WebSocketQueueService`); same-replica may use direct WS; **fail fast** when router down |
+| Graceful drain | **30s** on SIGTERM / k8s `preStop` — CLOSE frames, queue cleanup, DB status update |
+| Security | Agent handlers validate fog token **before** message processing; **50** upgrades/min/IP; **100** active WS/IP; JWT in `?token=` (ingress log redaction required) |
+| Scale SLO | **500** concurrent WS per replica; **p99 pairing < 5s** |
+| Observability | OpenTelemetry: active/pending sessions, pairing latency, AMQP failures, router connectivity |
+
+**OTEL metric names (R87):** `ws_exec_sessions_active`, `ws_log_sessions_active`, `ws_pending_pairings`, `ws_pairing_duration_ms` (histogram), `ws_amqp_publish_errors`, `ws_router_connected` (gauge). Emitted when `ENABLE_TELEMETRY=true`; see `src/websocket/ws-metrics.js`.
+
+**HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`), `failFastOnRouterUnavailable` (default `true`). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
+
+**Core modules:** `src/websocket/server.js`, `session-manager.js`, `log-session-manager.js`, `src/services/websocket-queue-service.js`, `src/services/router-connection-service.js`.
+
+**Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica AMQP requirement, k8s preStop drain, load SLO probe.
+
+Full spec: [`.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md`](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · RFC R80–R91 · Edgelet contract: [edgelet-invariants.md §10](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
+
 ---
 
 ## Edgelet agent contract (summary)
@@ -197,7 +317,21 @@ For the full bilateral contract (including ControlPlane env vars and verificatio
 
 | Topic | v3.8 behavior |
 |-------|---------------|
-| **Database** | Greenfield v3.8.0 schema — **new install only** (no v3.7 migrator). Supports sqlite (dev), mysql, postgres (production/HA). |
+| **Database** | Greenfield v3.8.0 schema — **new install only** (no v3.7 migrator). Supports **sqlite** (single-controller production), **mysql**, and **postgres** (multi-replica / HA). |
+
+### SQLite single-node production
+
+Small deployments with **one Controller process** may use SQLite as the production database (embedded OIDC requires a single replica in this profile).
+
+| Topic | Behavior |
+|-------|----------|
+| **When to use** | Single Controller, no DB HA requirement, edge/small-cluster PoT |
+| **Concurrency** | WAL journal mode + `busy_timeout` pragmas on connect; connection pool size 1 |
+| **Background jobs** | Reconcile-heavy jobs start after a configurable delay (`settings.jobStartupDelaySeconds`, default 3s) and stagger by 500ms to avoid restart lock bursts |
+| **Task claims** | Fog/service/NATS reconcile task claims retry on `SQLITE_BUSY` (same retry budget as `TransactionDecorator`) |
+| **Persistence** | Mount a persistent volume for `controller_db.sqlite` and WAL sidecar files (`-wal`, `-shm`) |
+| **Backup** | Use SQLite backup API or copy DB + WAL files during a quiet window |
+| **HA path** | mysql/postgres + multiple Controller replicas — see [oidc-configuration.md](oidc-configuration.md) |
 | **Applications** | Table `Applications` (was `Flows`); API identity by **name** string. |
 | **Architectures** | Table `Architectures` (was `FogTypes`); `archId` 0–4. |
 | **PKI** | Central **default-router-local-ca** and **default-nats-local-ca** for all new agents; no per-agent local CAs on provision (greenfield — no v3.7 PKI migration job). See [pki.md](pki.md). |

diff --git a/docs/operations/ws-sessions.md b/docs/operations/ws-sessions.md
@@ -0,0 +1,143 @@
+# WebSocket exec & log sessions — operator guide
+
+**Audience:** Platform operators running Controller in production  
+
+---
+
+## Overview
+
+Controller exposes **interactive exec** and **log streaming** over WebSocket on the API port (default **51121**). Sessions pair an operator browser/CLI client (Bearer JWT) with an Edgelet agent (fog token). In multi-replica deployments, cross-replica relay requires the **Skupper-style AMQP router** microservice.
+
+---
+
+## HTTPS and authentication
+
+| Requirement | Detail |
+|-------------|--------|
+| **HTTPS-only WS** | Set `CONTROLLER_PUBLIC_URL` to `https://…` and terminate TLS at ingress or the Controller listener (`TLS_PATH_*`). WebSocket upgrades must use `wss://`. |
+| **User auth** | Bearer JWT via `Authorization` header or `?token=` query param (browser Console). RBAC: `execSessions`, `logs`, `systemExecSessions`, `systemLogs`. |
+| **Agent auth** | Fog token on `/api/v3/agent/exec/*` and `/api/v3/agent/logs/*` — OIDC does **not** apply to agent routes. |
+
+### Ingress log redaction (required)
+
+Browser clients pass JWT in the query string: `wss://controller.example.com/api/v3/microservices/{uuid}/logs?token=…`
+
+**Configure ingress / reverse proxy access logs to redact `token` query parameters.** Example nginx:
+
+```nginx
+log_format ws_redacted '$remote_addr - [$time_local] "$request" $status '
+                       '"$http_referer" "$http_user_agent"';
+# Use map or custom log filter to strip ?token=… before writing logs.
+```
+
+Without redaction, long-lived bearer tokens may appear in load balancer logs.
+
+---
+
+## Multi-replica HA
+
+| Setting | Default | Env |
+|---------|---------|-----|
+| Cross-replica requires AMQP | `true` | `WS_HA_CROSS_REPLICA_REQUIRES_AMQP` |
+| Fail fast when router down | `true` | `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE` |
+
+**Requirements:**
+
+1. Deploy the **router** system microservice and ensure Controller can reach AMQP (`RouterConnectionService`).
+2. Run **2+ Controller replicas** behind a load balancer with **sticky sessions optional** — cross-replica exec/log uses AMQP queues (`agent-{execId}`, `user-{execId}`, `logs-user-{sessionId}`).
+3. When the router is unavailable, new cross-replica sessions close with WebSocket code **1013** (`Router unavailable for cross-replica session`).
+
+Same-replica sessions may relay directly without AMQP when both user and agent land on the same pod.
+
+---
+
+## Graceful drain (SIGTERM / Kubernetes preStop)
+
+On shutdown, Controller drains WebSocket sessions for up to **`WS_DRAIN_TIMEOUT_MS`** (default **30s**):
+
+1. Reject new upgrades (`verifyClient` → draining).
+2. Close pending users with code **1001** (`Server draining`).
+3. Send CLOSE frames, clean exec/log session DB rows, tear down AMQP bridges.
+
+### Kubernetes manifest example
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller
+spec:
+  template:
+    spec:
+      terminationGracePeriodSeconds: 45
+      containers:
+        - name: controller
+          lifecycle:
+            preStop:
+              exec:
+                command:
+                  - /bin/sh
+                  - -c
+                  - sleep 5
+          env:
+            - name: WS_DRAIN_TIMEOUT_MS
+              value: "30000"
+```
+
+**Procedure (manual verification):**
+
+1. Open an exec or log session against a running pod.
+2. `kubectl delete pod <controller-pod> --grace-period=45`
+3. Confirm the client receives close code **1001** within ~30s and exec is disabled (`execEnabled=false` for exec sessions).
+4. Confirm replacement pod accepts new sessions.
+
+---
+
+## Scale SLO (R88)
+
+| Metric | Target |
+|--------|--------|
+| Concurrent WS per replica | **500** (`WS_REPLICA_MAX_CONCURRENT_WS`) |
+| p99 exec pairing latency | **< 5s** |
+
+Run the load probe locally:
+
+```bash
+nvm use 24
+node test/load/ws-pairing-load.js --pairs 500
+```
+
+For production validation, repeat against a staging cluster with real agent simulators and record p99 from Controller OTEL histogram `ws_pairing_duration_ms`.
+
+---
+
+## OTEL metrics
+
+Enable `ENABLE_TELEMETRY=true`. Key metrics (`src/websocket/ws-metrics.js`):
+
+| Metric | Type |
+|--------|------|
+| `ws_exec_sessions_active` | gauge |
+| `ws_log_sessions_active` | gauge |
+| `ws_pending_pairings` | gauge |
+| `ws_pairing_duration_ms` | histogram |
+| `ws_amqp_publish_errors` | counter |
+| `ws_router_connected` | gauge |
+
+---
+
+## Session limits (normative)
+
+| Session | Limit |
+|---------|-------|
+| Exec user WS per microservice | **1** |
+| Exec pending (user waits for agent) | **60s** |
+| Exec max duration | **8h** |
+| Log user WS per microservice/fog | **3** |
+| Log pending (user waits for agent) | **120s** |
+| Log idle | **2h** |
+| Log tail max lines | **5000** |
+| WS upgrades per IP per minute | **50** |
+| Active WS per IP | **100** |
+
+See [architecture.md](../architecture.md#websocket-exec--log-sessions) for protocol diagrams.