From b3ff2a03d9b0661bd1a0ca31d61ca30ae1c06dd9 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 16 Mar 2026 11:48:30 -0700 Subject: [PATCH 01/17] In progress, largely vibe coded, still needs to be finalized --- adrs/001-s3-credentials.md | 102 ++++++ adrs/002-runtimes.md | 86 +++++ adrs/003-rust.md | 62 ++++ adrs/004-sts.md | 176 +++++++++ adrs/005-authorization.md | 126 +++++++ adrs/006-outbound-storage.md | 88 +++++ adrs/007-middleware.md | 88 +++++ adrs/008-crate-architecture.md | 93 +++++ adrs/009-configuration.md | 105 ++++++ adrs/rfc-001.md | 638 +++++++++++++++++++++++++++++++++ 10 files changed, 1564 insertions(+) create mode 100644 adrs/001-s3-credentials.md create mode 100644 adrs/002-runtimes.md create mode 100644 adrs/003-rust.md create mode 100644 adrs/004-sts.md create mode 100644 adrs/005-authorization.md create mode 100644 adrs/006-outbound-storage.md create mode 100644 adrs/007-middleware.md create mode 100644 adrs/008-crate-architecture.md create mode 100644 adrs/009-configuration.md create mode 100644 adrs/rfc-001.md diff --git a/adrs/001-s3-credentials.md b/adrs/001-s3-credentials.md new file mode 100644 index 0000000..5def272 --- /dev/null +++ b/adrs/001-s3-credentials.md @@ -0,0 +1,102 @@ +# ADR-001: S3 API Compatibility and Temporary-Credentials-Only Credential Model + +**Status:** Draft +**Date:** 2026-03-14 +**RFC:** RFC-001 §4 + +--- + +## Context + +Source Cooperative exposes a data proxy that must be consumable by the broadest possible range of data engineering tooling without requiring Source-specific client libraries. The S3 API has become the de facto standard protocol for object storage access. The ecosystem of compatible tooling is vast: AWS SDKs in every major language, CLI tools (`aws s3`, `rclone`), data frameworks (DuckDB, Polars, PyArrow, fsspec, GDAL/VSI), orchestration systems (Airflow, Dagster, Prefect), and notebook environments all speak S3 natively. + +The current proxy implements S3 compatibility and issues long-lived static `Access Key ID` / `Secret Access Key` pairs per user. Long-lived static credentials are a persistent security liability: they are frequently stored in plaintext config files, are difficult to rotate, and have no ambient context about the caller's environment or intended scope. Several high-profile incidents in the Source Cooperative infrastructure (including a compromised IAM credential used to conduct an SES email campaign) underscore the operational risk of long-lived secrets. + +The industry has broadly moved toward short-lived, exchanged credentials via OIDC workload identity federation. AWS STS, GCP Workload Identity Federation, and Azure Federated Identity Credentials all use the same underlying pattern: a trusted identity token is exchanged for short-lived scoped credentials at a Security Token Service. This pattern eliminates stored secrets on the caller side and ensures credentials expire automatically. + +--- + +## Decision + +### S3 API Compatibility + +We implement the AWS Signature Version 4 (SigV4) HMAC request signing protocol. All S3-compatible clients sign requests using an `Authorization` header derived from an `Access Key ID` and `Secret Access Key`. The proxy verifies this signature on every incoming request. + +This is unchanged from the current proxy. S3 API compatibility is a non-negotiable requirement for ecosystem reach. + +### Temporary Credentials Only + +**We do not issue or support long-lived static `Access Key ID` / `Secret Access Key` pairs.** + +All SigV4 credentials issued by Source Cooperative are temporary session credentials — the same triplet shape that AWS STS issues: + +``` +AccessKeyId (e.g. "ASIA...") +SecretAccessKey (short-lived derived key) +SessionToken (signed JWT encoding identity, role, and expiry) +``` + +Callers obtain these credentials by exchanging a trusted identity token at the STS endpoint (`POST /.sts/assume-role-with-web-identity`) before making S3 API calls. + +### Session Token Design + +The `SessionToken` is a stateless signed JWT. Its payload contains: + +```json +{ + "user_id": "", + "role_id": "", + "access_key_id": "", + "secret_access_key": "", + "exp": "" +} +``` + +The proxy verifies incoming SigV4 requests by: + +1. Extracting the `AccessKeyId` from the `Authorization` header +2. Looking up the corresponding `SessionToken` — presented as the `X-Amz-Security-Token` header +3. Verifying the JWT signature against the proxy's public key +4. Checking `exp` has not passed +5. Reconstructing the expected SigV4 signature using the `SecretAccessKey` from the token payload and comparing it to the presented signature + +No external database lookup is required to verify a request. The token is self-contained. + +**Permissions are not encoded in the session token.** The token encodes identity and role only. Per-request permission resolution is handled by the authorisation layer (see ADR-005) by consulting the policy store at request time. This is the same model AWS uses: the STS token asserts role membership, and IAM evaluates the role's current policies live on each API call. + +### Accepted Trade-offs + +**Tokens cannot be revoked once issued.** A compromised session token remains valid until its `exp`. Short TTLs (15–60 minutes recommended) limit the blast radius. Immediate revocation is out of scope for this iteration. + +**Callers must perform a token exchange before making S3 API calls.** This is a one-time step per session. All major AWS SDKs handle STS-derived session credentials natively via the credential provider chain. Tooling that accepts `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables works without modification. + +**Documentation and CLI tooling must minimise the friction of the exchange step.** Users accustomed to copying a static key into a config file will encounter a new workflow. A `source login` CLI command and SDK credential provider helpers are planned to make the exchange step transparent. + +--- + +## Consequences + +**Benefits** + +- No long-lived credentials anywhere in the system. Credentials expire automatically. +- Full compatibility with the existing S3 tooling ecosystem — no client changes required. +- The session token is stateless and self-verifying — no credential store on the hot path. +- Short-lived credentials limit blast radius of any credential compromise. +- Composable with OIDC workload identity federation (see ADR-004) — the exchange step is the same regardless of the upstream identity source. + +**Costs / Risks** + +- Callers must perform a token exchange before first use. This is new friction compared to the current static key model. +- The `/.sts` exchange endpoint is on the critical path for session establishment. Its availability affects whether callers can obtain credentials. +- Session tokens cannot be revoked. A credential leaked mid-session remains valid until TTL expiry. +- S3 tooling that hardcodes static credential configuration (rather than using the SDK credential provider chain) may require workarounds. + +--- + +## Alternatives Considered + +**Long-lived static credentials (current model)** — rejected. Persistent security liability; does not compose with workload identity federation; difficult to audit or rotate at scale. + +**Short-lived credentials with a server-side revocation list** — considered. Would allow immediate invalidation of compromised credentials. Rejected for this iteration: adds a stateful dependency on the hot path of every request, increasing latency and operational complexity. Can be added in a future iteration if the threat model requires it. + +**Custom non-S3 protocol** — rejected. Would require Source-specific client libraries and break compatibility with the entire existing ecosystem of data tooling. \ No newline at end of file diff --git a/adrs/002-runtimes.md b/adrs/002-runtimes.md new file mode 100644 index 0000000..4bde41d --- /dev/null +++ b/adrs/002-runtimes.md @@ -0,0 +1,86 @@ +# ADR-002: Runtime — Cloudflare Workers (Primary) + Regional ECS (Secondary) + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §5 + +--- + +## Context + +Source Cooperative's data proxy serves users globally, but most upstream data resides in AWS `us-west-2`. Users far from that region experience significant latency. Replicating data to additional regions is cost-prohibitive. + +The proxy needs two deployment modes to serve distinct access patterns: + +1. **Global, latency-sensitive reads** — the majority of traffic. Users worldwide reading public datasets. These benefit from edge deployment close to the caller. +2. **High-throughput, in-region workflows** — data pipelines (Spark, Databricks, Polars) running in `us-west-2` reading large volumes from S3 in the same region. Routing this traffic through an edge node adds unnecessary hops and latency. + +The current proxy is a single ECS deployment. It handles both patterns, but serves neither optimally. + +--- + +## Decision + +### Cloudflare Workers (Primary) + +The primary deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. Workers deploy to Cloudflare's edge network (330+ locations worldwide) automatically. + +Key properties: + +- **Global distribution without operational overhead.** Requests are served from the location closest to the caller. Onward routing to upstream storage traverses the Cloudflare backbone rather than the public internet. +- **Effectively no cold start.** Workers use V8 isolates (not containers). Cloudflare's "Shard and Conquer" consistent hashing achieves a 99.99% warm request rate. +- **No Cloudflare-imposed egress fees.** Upstream object store egress fees still apply, but Cloudflare does not charge for bandwidth out of Workers. +- **No wall-clock timeout.** CPU time limits apply per invocation, but streaming large objects is not killed mid-response due to elapsed time. +- **Predictable, low cost.** $5/mo base, $0.30/M requests, $0.02/M CPU-ms; 10M requests + 30M CPU-ms included. +- **WASM compatibility.** Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`). + +### Regional ECS Deployments (Secondary) + +Traditional containerised Rust services deployed into specific cloud regions on demand. Intended for high-throughput, in-region workflows where: + +- Egress fees are zero or near-zero when traffic stays within the region +- Network throughput is higher and latency is lower than routing through an edge node +- The Workers path adds unnecessary hops + +Regional deployments share the same Rust core as the Workers deployment. The proxy logic, auth, and authz layers are identical; only the runtime adapter differs. + +### Shared STS and Credential Interoperability + +Each deployment target hosts its own STS endpoint at `/.sts`. Workers and all regional ECS deployments share the same signing key material, so session credentials issued by any target are valid across all targets. + +### Accepted Trade-offs + +**Regional access restriction is unresolved.** Regional proxies should only be accessible to in-region consumers. Candidate mechanisms include VPC-only endpoints, IP range allowlisting, region-scoped audience claims, and regional-specific session credentials. Each has tradeoffs around operational complexity and developer experience. See RFC-001 Open Question 1. + +**Two deployment targets increase operational surface.** The shared Rust core mitigates code divergence, but deployment, monitoring, and key management are duplicated. + +--- + +## Consequences + +**Benefits** + +- Global users experience lower latency without data replication +- In-region workflows avoid unnecessary edge hops and egress charges +- No Cloudflare egress fees for the majority of traffic +- Effectively zero cold start for the primary deployment target +- Shared core ensures behavioural consistency across targets + +**Costs / Risks** + +- Two deployment targets to build, test, deploy, and monitor +- WASM compilation constrains library choices for the shared core (no `std` features that don't work in WASM) +- Regional access restriction mechanism is unresolved +- Credential interoperability across targets requires shared key material and coordinated rotation + +--- + +## Alternatives Considered + +**Single ECS deployment (current model)** — rejected. Does not address global latency without data replication. No edge presence. + +**CDN in front of ECS** — considered. A traditional CDN (CloudFront, Cloudflare) can cache static responses, but the proxy's responses are not cacheable in a general-purpose CDN sense (authenticated, per-user). The proxy logic must run at the edge, not just caching. + +**Workers only (no regional ECS)** — considered. Simpler operationally, but penalises high-throughput in-region workflows with unnecessary hops and potentially higher latency for large data transfers within the same cloud region. + +**Lambda@Edge / CloudFront Functions** — considered. More limited runtime environment, tighter CPU and memory constraints, and AWS-specific. Workers offer a more capable and provider-neutral edge compute model. diff --git a/adrs/003-rust.md b/adrs/003-rust.md new file mode 100644 index 0000000..e4c8865 --- /dev/null +++ b/adrs/003-rust.md @@ -0,0 +1,62 @@ +# ADR-003: Rust as Implementation Language + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §6 + +--- + +## Context + +The re-architected proxy must compile to two targets: WebAssembly for Cloudflare Workers (ADR-002) and a native binary for regional ECS deployments. The language must support both targets from a single codebase. Performance matters for the ECS target (streaming large objects with tight latency requirements). The proxy handles security-sensitive operations: cryptographic signature verification, credential issuance, and access policy evaluation. + +The current proxy is written in Rust. The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known but is unsuitable for the WASM target. + +--- + +## Decision + +We continue with **Rust** as the implementation language for both deployment targets. + +### Rationale + +**WASM maturity.** Rust has the most mature and production-ready toolchain for compiling to WebAssembly. The `worker-rs` crate provides idiomatic bindings to the Cloudflare Workers runtime. This is a well-trodden path, not a bet on emerging capability. + +**Performance.** For the regional ECS deployment, raw throughput matters. Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that streams large objects with tight latency requirements. This was already proven by the current proxy. + +**Type system and correctness.** The proxy handles authentication tokens, credential issuance, cryptographic signature verification, and access policy evaluation. Rust's type system — and in particular its trait system — encodes invariants that would be runtime errors in other languages. This is increasingly valuable in a codebase where AI-assisted development is part of the workflow: a strong type system provides a correctness harness that catches generated code that compiles but violates domain constraints. + +**Trait-based extensibility.** The Rust trait system is central to the modularity goals described in ADR-008. Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. + +**Community familiarity.** Rust is the best fit given the actual pool of contributors. + +--- + +## Consequences + +**Benefits** + +- Single codebase compiles to both WASM and native targets +- Zero-cost abstractions and no GC pauses for high-throughput streaming +- Trait system enables the modular, community-extensible architecture +- Strong type system as a correctness harness for security-sensitive code +- Continuity with the existing proxy — no rewrite learning curve for current contributors + +**Costs / Risks** + +- Steeper learning curve for new contributors compared to Go or Python +- Longer compilation times than Go +- WASM target constrains which crates and `std` features can be used in the shared core +- Async ecosystem split (`tokio` for ECS, `worker-rs` primitives for Workers) requires careful abstraction + +--- + +## Alternatives Considered + +**Go** — considered. Strong WASM support is emerging but less mature than Rust's. Lacks the trait system needed for the modularity goals. GC pauses are a concern for high-throughput streaming. Fewer Rust contributors would need to learn a new language than Go contributors. + +**TypeScript (native Workers language)** — considered. First-class Workers support, but unsuitable for the ECS target's performance requirements. No type-level enforcement of security invariants comparable to Rust's ownership and trait system. + +**Python** — rejected. Does not compile to WASM. Runtime overhead incompatible with the regional proxy's performance goals. + +**C++** — rejected. Less community familiarity than Rust. Memory safety concerns for security-sensitive code. No comparable trait system for extensibility. diff --git a/adrs/004-sts.md b/adrs/004-sts.md new file mode 100644 index 0000000..51c43a1 --- /dev/null +++ b/adrs/004-sts.md @@ -0,0 +1,176 @@ +# ADR-004: Inbound Authentication — OIDC Federation, STS Exchange, and SC Credential Tokens + +**Status:** Draft +**Date:** 2026-03-14 +**RFC:** RFC-001 §7 +**Depends on:** ADR-001 + +--- + +## Context + +Source Cooperative's data proxy must authenticate a wide range of callers: + +- **CI/CD pipelines** (GitHub Actions, GitLab CI, Azure DevOps, Terraform Cloud) running data ingestion or validation jobs +- **Managed compute environments** (Databricks, AWS Lambda, GCP Cloud Run) running data processing workflows +- **Interactive developers** working locally via notebooks, terminals, or the Source Cooperative CLI +- **Unattended scripts and third-party tools** that cannot interactively obtain a session token +- **The Source Cooperative web application** (Next.js on Vercel) making requests on behalf of authenticated users and anonymous visitors + +ADR-001 establishes that all callers must obtain short-lived SigV4 session credentials before making S3 API calls. This ADR defines how those credentials are obtained — the design of the STS exchange endpoint and the supported identity sources. + +The industry standard for secretless workload authentication is OIDC workload identity federation: a workload presents a signed JWT issued by its platform's OIDC provider to a Security Token Service, which validates it and returns short-lived scoped credentials. AWS, GCP, Azure, GitHub Actions, GitLab CI, Vercel, and many others all support this pattern. + +The key insight is that any platform with a publicly reachable JWKS endpoint can be a trusted issuer. Trust is established by configuration (register the issuer URL and claim conditions), not by code. This means the STS is open to the entire OIDC ecosystem without modification. + +--- + +## Decision + +### STS Exchange Endpoint + +Both deployment targets (Cloudflare Workers and regional ECS) host an STS exchange endpoint at: + +``` +POST /.sts/assume-role-with-web-identity +``` + +Request body (form-encoded or JSON): +``` +identity_token= +``` + +Response (on success): +```json +{ + "AccessKeyId": "...", + "SecretAccessKey": "...", + "SessionToken": "...", + "Expiration": "" +} +``` + +The STS exchange flow: +1. Extract the `iss` claim from the presented JWT (without verifying signature yet) +2. Look up the registered issuer configuration for that `iss` value +3. Fetch the issuer's JWKS (from cache if available) and verify the JWT signature +4. Evaluate any registered claim conditions for this issuer (e.g. `repository == "source-cooperative/data-pipeline"`) +5. Map the verified identity to an internal `role_id` and `user_id` +6. Issue and return a signed session token (see ADR-001 for token structure) + +Trust is established per-issuer by configuration: +- Issuer URL (must match `iss` claim) +- JWKS endpoint URL (defaults to `/.well-known/jwks.json` per OIDC discovery) +- Claim conditions (optional; constrain which tokens from this issuer are accepted) +- Role mapping (which `role_id` to assign to matching tokens) + +**No code changes are required to add a new trusted issuer.** It is a configuration operation. + +### Trusted Issuer Registry — Well-Known Issuers + +The following issuers are relevant to data engineering workflows and are the primary documentation and support targets. Any issuer with a publicly reachable JWKS endpoint can be registered. + +**CI/CD Platforms** + +| Platform | Issuer URL | Key Claims for Conditions | +| -------------- | --------------------------------------------- | ------------------------------------------------------ | +| GitHub Actions | `https://token.actions.githubusercontent.com` | `repository`, `ref`, `environment`, `job_workflow_ref` | +| GitLab CI/CD | `https://gitlab.com/` | `project_path`, `ref_type`, `environment` | +| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | +| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | +| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | + +**Managed Compute and Data Platforms** + +| Platform | Token Source | Notes | +| --------------------------- | ------------------------------------ | ----------------------------------------------- | +| AWS (EC2, ECS, Lambda) | Instance/task metadata service | IAM role identity token | +| GCP (Cloud Run, GKE) | Metadata server | Service account identity token, audience-scoped | +| Azure (AKS, Container Apps) | Managed Identity / Entra ID | Federated credential | +| Databricks Jobs | Platform SDK / environment injection | Workspace, job, cluster identity | + +**Claim Conditions** + +Claim conditions prevent overly broad trust grants. For example, a GitHub Actions issuer registration should always include at minimum a `repository` claim condition to prevent any GitHub Actions workflow from obtaining Source Cooperative credentials. Recommended minimum conditions per issuer should be documented. + +### Identity Source: Source Cooperative Auth (`auth.source.coop`) + +The Source Cooperative auth system is Ory-based and issues standard OIDC access tokens. `auth.source.coop` is registered as a trusted issuer with the following mapping: +- Verified token → `role_id: "authenticated_user"`, `user_id` extracted from `sub` claim +- Admin users: identified by an Ory group membership claim → `role_id: "admin"` + +This path is appropriate for: +- Interactive local development (developer exchanges a browser session token) +- CLI `source login` flow (browser device flow → Ory token → STS exchange) +- Next.js client-side exchange for authenticated web users + +### Identity Source: SC Credential Tokens + +For unattended workflows that lack an ambient OIDC token, users may generate a **Source Cooperative Credential Token** from the platform UI or API. + +SC Credential Tokens are: +- Signed JWTs issued by Source Cooperative using the same private key infrastructure as the outbound OIDC tokens (see ADR-006) +- Validated via the proxy's published JWKS — Source Cooperative is registered as a trusted issuer of its own credential tokens +- Scoped at mint time: the JWT encodes `user_id`, permitted datasets/collections, and the `role_id` to assume +- Subject to a mandatory maximum TTL enforced at exchange time (recommended: 90 days) +- **Not revocable** — consistent with the stateless session token design in ADR-001 + +SC Credential Tokens flow through the same `/.sts` exchange endpoint as all other issuers. No special code path. + +The user stores the SC Credential Token in their workflow's secret manager (GitHub Actions secrets, a CI environment variable, etc.). At runtime, the workflow presents it to `/.sts` and receives short-lived SigV4 session credentials. The stored token is long-lived; the operational credentials are always short-lived. + +**Comparison of identity sources:** + +| Source | Suited for | Stored secret? | SigV4 credential TTL | +| --------------------------------------- | ----------------------------------------- | ------------------ | -------------------- | +| Ambient OIDC (GitHub, Databricks, etc.) | CI/CD, managed compute | No | 15–60 min | +| `auth.source.coop` / Ory | Interactive local dev, CLI | No (session-based) | 15–60 min | +| SC Credential Token | Unattended pipelines without ambient OIDC | Yes (token, ≤90d) | 15–60 min | + +### Next.js and Front-End Authentication + +**Authenticated users:** The browser holds the Ory session token. The client exchanges it directly with `/.sts` (client-side) to obtain SigV4 session credentials. S3 API calls are made directly from the browser to the proxy. Access is recorded in proxy metrics under the user's own identity. + +**Anonymous visitors:** The Next.js server uses its Vercel OIDC token (`VERCEL_OIDC_TOKEN`, available in Vercel server-side contexts) to exchange for a `public-read`-scoped `anonymous` role credential. This credential is used for requests on behalf of anonymous visitors. All anonymous traffic flows through the proxy's full middleware stack — rate limiting and access metrics apply. + +**Admin users:** Same flow as authenticated users. Admin role is granted at STS exchange time based on an Ory group membership claim. + +### CLI and SDK Support + +**`source` CLI:** +- `source login` — browser-based device flow via `auth.source.coop`, exchanges resulting Ory token at `/.sts`, writes SigV4 session credentials to `~/.aws/credentials` or exports as environment variables +- `source credentials export` — re-exchanges and outputs credentials in a specified format + +**Python SDK:** +- A credential provider compatible with `boto3`'s credential provider chain and `fsspec`/`s3fs` +- Wraps the `/.sts` exchange transparently; handles token refresh before expiry + +--- + +## Consequences + +**Benefits** +- No stored secrets for CI/CD and managed compute workflows that have ambient OIDC tokens +- Open to any OIDC-compliant issuer without code changes +- All credential paths converge on the same `/.sts` endpoint — one validation and issuance code path +- SC Credential Tokens reuse the outbound OIDC key infrastructure — no separate signing system +- Short-lived operational credentials regardless of how the caller authenticated + +**Costs / Risks** +- The `/.sts` endpoint is on the critical path for session establishment. Its availability and latency directly affect all new sessions. +- JWKS fetching and caching must be robust — a stale or unavailable JWKS causes all exchange attempts for that issuer to fail. +- Claim conditions must be carefully configured per issuer. Misconfigured conditions (too broad) are a privilege escalation vector. +- SC Credential Tokens reintroduce stored secrets for the unattended pipeline use case. Their 90-day TTL and narrow scope partially mitigate this; they cannot be revoked if compromised before expiry. +- Adding a new issuer requires operator access to register it. There is no self-service issuer registration (by design — unreviewed issuers are a security risk). + +--- + +## Alternatives Considered + +**Fixed allowlist of supported OIDC issuers (code-driven)** — rejected. Would require code changes to add new platforms, limiting adoption by teams using less common tooling. Configuration-driven trust is strictly more flexible. + +**Long-lived API keys instead of SC Credential Tokens** — considered. SC Credential Tokens are preferred because they are structured JWTs with embedded scope and expiry, flow through the same validation path as ambient OIDC tokens, and carry a hard expiry enforced at exchange time. A raw API key would require a separate validation code path and a database lookup on every exchange. + +**RFC 8693 token exchange ("act-as") for Next.js server-side requests** — considered for the case where Next.js makes S3 requests on behalf of a specific authenticated user server-side. Rejected for initial implementation: passing the user's Ory token through client-side is simpler and achieves the same access metrics goal. RFC 8693 is worth revisiting if the audit log needs to distinguish "user accessed directly" from "Next.js accessed on user's behalf." + +**Session token revocation list** — considered. Would allow immediate invalidation of SC Credential Tokens and session tokens. Rejected for this iteration: adds a stateful dependency to the hot path of every request. Can be added later if the threat model requires it. diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md new file mode 100644 index 0000000..e0fa7d1 --- /dev/null +++ b/adrs/005-authorization.md @@ -0,0 +1,126 @@ +# ADR-005: Authorization Model — Dynamic Per-Request Policy Resolution + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §8 +**Depends on:** ADR-001, ADR-004 + +--- + +## Context + +ADR-001 establishes that session tokens are stateless JWTs encoding identity and role, but **not** permissions. This ADR defines how permissions are resolved at request time. + +Two properties drive the design: + +1. **Permissions are dynamic.** A user who creates a new organisation or dataset should be able to access it immediately. Encoding permissions in the session token would freeze them at exchange time, requiring re-exchange to reflect changes. + +2. **The role is a ceiling; user permissions are the grants.** The role answers "what classes of action are permitted for this identity type?" The per-user permission lookup answers "which specific resources can this identity access?" The proxy enforces the intersection. + +This mirrors AWS IAM: a session token asserts role membership, and the role's current policies are evaluated live on each API call. + +--- + +## Decision + +### Identity Model + +The session token carries three fields relevant to authorization: + +- `user_id` — stable identifier for the authenticated principal +- `role_id` — one of: `anonymous`, `authenticated_user`, `admin` +- `exp` — token expiry; checked before any policy evaluation + +### Role Definitions + +**`anonymous`** +- Permitted action classes: read-only (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`) +- Role-level filter: only buckets flagged `public = true` are visible and accessible +- No user permission lookup — role filter is the only guard + +**`authenticated_user`** +- Permitted action classes: read and write (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`, `ListObjects`, `ListBuckets`, `CreateBucket`) +- Role-level filter: none — user permission lookup determines which resources are accessible +- User permission lookup is always performed for non-public resources + +**`admin`** +- Permitted action classes: all +- Role-level filter: none +- User permission lookup: **skipped** — admin role has unconditional access to all resources +- Admin role assumption is gated on strong identity claims (e.g. Ory group membership) to prevent accidental privilege escalation + +### Per-Request Resolution Strategy + +Authorization proceeds in at most three steps, with early exits to minimise unnecessary lookups: + +**Step 1 — Role action check** +Does this role permit the requested action class? If the role is `anonymous` and the request is a `PutObject`, deny immediately. This is pure in-memory logic against the role definition — no lookup required. + +**Step 2 — Public resource early exit** +For read operations only: is the requested bucket flagged `public = true` in the policy store? If yes, and the role permits reads, permit immediately without a user permission lookup. This covers the majority of Source Cooperative traffic (public dataset access) and avoids a per-user lookup for every read of public data. + +**Step 3 — User permission lookup** +For non-public resources or write operations: fetch the user's permissions from the policy store and evaluate them against the requested resource. This reflects current organisation membership, dataset ownership, and explicit grants. + +### Operation-Specific Behaviour + +**Single-resource operations (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`)** +After the role check and public early exit, a point lookup: does `(user_id, bucket_id)` resolve to an access grant? If the grant includes prefix restrictions, those are enforced against the requested object key. + +**`ListBuckets`** +The proxy constructs this response entirely from the policy store — the upstream is never called. Anonymous users see `public = true` buckets; authenticated users see all buckets they have grants for; admins see all buckets. + +**`ListObjects` (within a bucket)** +After the role check, public early exit, and user permission lookup: if the grant includes a key prefix restriction, it is passed as a filter to the upstream `ListObjects` call so the upstream enforces the boundary. + +### Cache Strategy + +All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container). + +| Lookup | Cache Key | TTL | +|---|---|---| +| Role definition | `role_id` | In-memory constant | +| Bucket public flag | `bucket_id` | 60–300s | +| Single-resource user grant | `(user_id, bucket_id)` | 30–60s | +| User's full bucket list (`ListBuckets`) | `(user_id, role_id)` | 5–10s | + +The short TTL on the full bucket list ensures that a user who creates a new dataset sees the change within seconds. For Workers, cache is per-isolate and not shared across edge nodes; Workers KV is available as a shared tier if needed. + +### Unresolved: Grant Schema + +The exact schema of user access grants is unresolved. Open questions include: + +- Whether grants are bucket-level only or support sub-bucket prefix granularity +- Whether grants are additive (allow-only) or support explicit denies +- How organisation membership is modelled — derived grants from membership, or explicit per-bucket grants per member + +These questions are tracked in RFC-001 Open Question 7. + +--- + +## Consequences + +**Benefits** + +- Permissions reflect current state — no re-exchange required after creating a new dataset or joining an organisation +- The majority of traffic (public dataset reads) resolves with no user-specific lookup +- Admin bypass eliminates unnecessary lookups for administrative operations +- Cache TTLs are tuned per-operation to balance freshness and performance +- The model is familiar to anyone who knows AWS IAM + +**Costs / Risks** + +- Every non-public authenticated request requires a policy store lookup (mitigated by caching) +- The policy store is on the hot path — its availability affects request latency for cache misses +- Per-isolate caching in Workers means cache is not shared across edge nodes (cold isolate = cache miss) +- Grant schema is unresolved — the implementation cannot begin until the schema is defined + +--- + +## Alternatives Considered + +**Encode permissions in the session token** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens to see permission changes. Unacceptable for a platform where users create datasets and join organisations dynamically. + +**Centralised permission cache (Redis / Workers KV as primary)** — considered. Would share cache across isolates and containers. Rejected as the primary tier: adds a network hop to every cache read. Per-isolate caching with optional Workers KV as a secondary tier is preferred. + +**Explicit deny support in grants** — deferred. Additive (allow-only) grants are simpler to reason about and sufficient for the initial use cases. Explicit denies can be added later if the access control model requires it. diff --git a/adrs/006-outbound-storage.md b/adrs/006-outbound-storage.md new file mode 100644 index 0000000..0779b30 --- /dev/null +++ b/adrs/006-outbound-storage.md @@ -0,0 +1,88 @@ +# ADR-006: Outbound Connectivity — OIDC Issuer Model and `object_store` Adoption + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §9 +**Depends on:** ADR-002 + +--- + +## Context + +When the proxy receives an authenticated, authorised request, it must retrieve or write the underlying object from an upstream storage backend (S3, GCS, Azure Blob, R2, etc.). This outbound connection must itself be authenticated, without embedding long-lived cloud credentials in the proxy service. + +The current proxy implements per-backend adapters manually — a separate integration for each cloud storage provider, with bespoke error mapping from each provider's client library. This is maintenance-intensive and creates an ongoing gap as new backends are added or existing client APIs change. + +Additionally, Source Cooperative intends to support **data providers** who register their own upstream storage with the platform. The proxy fronts their buckets with auth, authz, rate limiting, and metering. + +--- + +## Decision + +### `object_store` as Unified Storage Abstraction + +The [`object_store`](https://crates.io/crates/object_store) crate replaces all manual per-backend adapters. `object_store` provides a single async trait (`ObjectStore`) with implementations for S3, GCS, Azure Blob, R2, HTTP, and local filesystem. + +This eliminates backend-specific client code and error mapping from the proxy codebase. New storage backends supported by `object_store` become available without proxy changes. + +### Outbound Authentication — OIDC Token Issuance (Preferred) + +Source Cooperative operates as an OIDC identity provider, publishing: +- `/.well-known/openid-configuration` — OIDC discovery document +- A JWKS endpoint — public keys for verifying tokens issued by the proxy + +Upstream cloud providers (AWS, GCP, Azure) register Source Cooperative as a trusted external identity provider via their native workload identity federation mechanisms. The proxy generates short-lived, audience-scoped JWTs and exchanges them for cloud credentials at each provider's STS. + +This model means: +- No long-lived cloud credentials are stored in the proxy +- Credentials are ephemeral +- The trust relationship is declarative and auditable +- Key rotation at the proxy level propagates automatically without reconfiguring upstream providers + +### Outbound Authentication — Stored Secrets (Fallback) + +For upstream providers or storage systems that do not support OIDC workload identity federation, credentials may be stored as encrypted secrets and injected into the proxy's configuration at startup. + +This is a fallback, not the preferred path. + +### Data Provider Hosting + +Data providers register their upstream storage (their own S3 bucket, GCS bucket, etc.) with Source Cooperative. The proxy serves as an access control, metering, and distribution layer in front of their data. + +Data providers get: +- **Cost control** — rate limiting, metering, and access thresholds prevent runaway egress costs +- **Access control** — fine-grained role and policy configuration +- **Exposure** — data is discoverable via the Source Cooperative platform and UI +- **Outbound auth flexibility** — the provider's own cloud credentials (or OIDC trust relationship) are used for the proxy's outbound connection + +### Unresolved: Provider Credential Operations + +For provider-hosted datasets where the provider's cloud does not support OIDC federation, the operational model for storing and rotating credentials securely is unresolved. Open questions include per-provider isolation and the trust boundary (what can Source Cooperative access in a provider's backend). See RFC-001 Open Question 5. + +--- + +## Consequences + +**Benefits** + +- Backend-specific client code and error mapping eliminated from the proxy codebase +- New `object_store` backends available to the proxy without changes +- Preferred outbound auth model uses no long-lived credentials +- Data providers can register their own storage and benefit from Source Cooperative's access control and distribution layer + +**Costs / Risks** + +- `object_store` must compile to `wasm32-unknown-unknown` for the Workers target — any features that don't work in WASM must be avoided or patched +- The OIDC issuer model requires upstream cloud providers to register Source Cooperative as a trusted IdP — this is a per-provider setup step +- Fallback stored secrets reintroduce long-lived credentials for providers that lack OIDC federation support +- Provider credential isolation and rotation model is unresolved + +--- + +## Alternatives Considered + +**Manual per-backend adapters (current model)** — rejected. Maintenance-intensive, creates ongoing integration gaps, and does not scale with new backends. + +**Provider-managed proxy instances** — considered. Each data provider runs their own proxy instance with their own credentials. Rejected: fragments the platform, complicates access control, and defeats the purpose of a unified distribution layer. + +**Proxy stores all upstream credentials in a secrets manager (e.g. AWS Secrets Manager)** — considered as the primary model rather than fallback. Rejected in favour of OIDC: secrets managers still store long-lived credentials that must be rotated. OIDC federation eliminates stored secrets entirely for providers that support it. diff --git a/adrs/007-middleware.md b/adrs/007-middleware.md new file mode 100644 index 0000000..8f38182 --- /dev/null +++ b/adrs/007-middleware.md @@ -0,0 +1,88 @@ +# ADR-007: Middleware Architecture — Rate Limiting, Metering, and Billing Hooks + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §10 +**Depends on:** ADR-005, ADR-008 + +--- + +## Context + +A general-purpose data proxy needs behaviours beyond authentication and object retrieval. Source Cooperative specifically requires: + +- **Rate limiting** — fine-grained and dynamic: different limits per bucket, per user, per organisation, or per role +- **Data metering** — tracking cumulative data transfer per identity or dataset, and enforcing access thresholds (e.g. denying access once a monthly quota is reached) +- **Usage tracking and billing hooks** — recording access events with enough fidelity to support downstream billing +- **Audit logging** — a complete, tamper-resistant record of who accessed what and when + +These concerns are cross-cutting: they apply to every request regardless of the specific storage backend or dataset, but their configuration and behaviour differ across deployments and use cases. Provider-hosted datasets may carry additional metering and quota requirements beyond what Source Cooperative's own datasets need. + +--- + +## Decision + +### Middleware Stack Pattern + +Cross-cutting concerns are implemented as a **composable middleware stack** wrapping the core request handler. Each middleware layer: + +- Receives the request context (resolved identity, role, resource, action) and may modify or enrich it +- May short-circuit the request with a denial response (e.g. quota exceeded, rate limit hit) +- May record an event (e.g. to a metering store or audit log) +- Passes the request to the next layer if permitted + +### Middleware as Rust Traits + +Middleware components are defined as Rust traits, making them first-class extension points. Source Cooperative ships standard implementations; operators can add their own without forking the core (see ADR-008). + +### Configuration Scope + +The middleware stack is configured per-deployment and potentially per-dataset. A dataset with no billing requirements carries a lightweight stack; a provider-hosted dataset with metered access carries additional quota and event-recording middleware. + +### Standard Middleware (Planned) + +| Middleware | Behaviour | +|---|---| +| Rate limiter | Per-identity or per-bucket request rate enforcement, configurable limits | +| Quota enforcer | Cumulative data transfer tracking; deny on threshold exceeded | +| Usage recorder | Structured event emission per request (bytes transferred, identity, resource, latency) | +| Audit logger | Tamper-evident request log for compliance and forensics | +| Billing emitter | Usage event publication to a configurable billing backend | + +### Unresolved + +The following details require further design: + +- **Middleware trait interface** — the exact trait signature, including how request context is threaded and how middleware ordering is enforced +- **Per-dataset configuration** — how middleware stacks are expressed per-deployment and per-bucket +- **Event schema** — the structured format for usage recording and billing events +- **Event backend** — the initial target for event emission (Kinesis stream, S3/R2 log, webhook, or other). See RFC-001 Open Question 6 +- **Middleware ordering** — whether order-dependent behaviours are made explicit or left to the operator + +--- + +## Consequences + +**Benefits** + +- Cross-cutting concerns are composable and configurable, not hardcoded +- New middleware can be contributed by the community without forking the core +- Per-dataset middleware stacks support the data provider hosting model +- The trait-based design enforces a consistent interface across all middleware + +**Costs / Risks** + +- Middleware on the hot path adds per-request overhead (mitigated by keeping middleware lightweight) +- Per-dataset middleware configuration adds operational complexity +- The middleware trait interface, event schema, and event backend are all unresolved — implementation cannot begin until these are defined +- Middleware ordering can introduce subtle bugs if order-dependent behaviours are not made explicit + +--- + +## Alternatives Considered + +**Hardcoded middleware in the core proxy** — rejected. Does not support the modularity and community-reuse goals. Provider-hosted datasets need different middleware stacks than Source Cooperative's own datasets. + +**Sidecar/external middleware (e.g. Envoy filters)** — considered. Offloads middleware to a separate process. Rejected: does not work in the Workers deployment target (no sidecar model), and adds latency from inter-process communication. + +**Plugin system (dynamic loading)** — considered. Would allow middleware to be loaded at runtime. Rejected for the Workers target: WASM does not support dynamic library loading. Rust traits with static dispatch are the natural fit for both targets. diff --git a/adrs/008-crate-architecture.md b/adrs/008-crate-architecture.md new file mode 100644 index 0000000..547dd3a --- /dev/null +++ b/adrs/008-crate-architecture.md @@ -0,0 +1,93 @@ +# ADR-008: Modular Crate Architecture and Community Reuse Model + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §11 + +--- + +## Context + +The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. It is difficult for external operators to deploy a version of the proxy for their own datasets, and equally difficult for contributors to improve the proxy in ways that are reusable outside Source Cooperative's deployment. + +The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration is a thin layer on top. + +This requires a clean separation between the general-purpose proxy framework and Source Cooperative-specific concerns. All Source Cooperative-specific behaviour must be expressed through the same trait interfaces that any other operator would use. + +--- + +## Decision + +### Crate Structure + +The proxy is structured as a set of Rust crates with well-defined trait boundaries between layers: + +| Crate | Responsibility | SC-specific? | +|---|---|---| +| `proxy-core` | Request routing, SigV4 verification, session credential management, middleware stack execution | No | +| `proxy-auth` | STS exchange logic, OIDC issuer registry, JWT validation, SC Credential Token minting | No — issuer list is configuration | +| `proxy-authz` | Role resolution, per-request policy evaluation, policy store interface trait | No — store backend is pluggable | +| `proxy-storage` | `object_store`-based backend abstraction | No | +| `proxy-middleware` | Middleware trait definition and standard implementations (rate limiter, quota enforcer, usage recorder, etc.) | No | +| `proxy-workers` | Cloudflare Workers runtime adapter, WASM build target | No | +| `proxy-ecs` | Traditional server runtime adapter, Hyper/Tokio based | No | + +**Nothing Source Cooperative-specific lives in the core crates.** An operator building their own proxy instantiates the core with their own implementations of the configuration traits — providing their own backend resolver, role mapping, middleware stack — without forking any crate. + +Source Cooperative's own deployment is the reference implementation of this pattern. + +### Trait-Based Extension Points + +Each layer defines traits that downstream operators implement: + +- **Auth:** issuer registry, claim condition evaluator, role mapper +- **Authz:** policy store, grant resolver +- **Storage:** backend resolver (maps bucket ID to `object_store` configuration) +- **Middleware:** middleware trait for custom cross-cutting concerns +- **Configuration:** configuration source trait for deployment-specific settings + +### Publication and Licensing + +Core crates are intended for publication to `crates.io` under a permissive licence. + +### Unresolved: Governance + +The following governance questions are unresolved: + +- Crate naming conventions +- Licence choice (MIT, Apache-2.0, or dual) +- API stability guarantees (semver policy, MSRV policy) +- Whether community-contributed crates live in the same repository, a separate organisation, or are fully external +- What "supported" means for community-contributed middleware or backends +- Contribution model and review process + +These are tracked in RFC-001 Open Question 8. + +--- + +## Consequences + +**Benefits** + +- Community members can build their own data proxies on the same foundation +- Contributions to the core (new middleware, new storage backends, auth improvements) benefit all deployments +- Source Cooperative's infrastructure demonstrates the framework's capabilities, aiding adoption +- Clean trait boundaries prevent Source Cooperative-specific concerns from leaking into the framework +- No forking required for custom deployments + +**Costs / Risks** + +- Maintaining trait stability across crate versions requires discipline and a clear semver policy +- Multiple crates increase the build and release coordination overhead +- Trait boundaries must be designed carefully upfront — changing a public trait is a breaking change +- Community governance and contribution model are unresolved + +--- + +## Alternatives Considered + +**Monolithic crate with feature flags** — considered. Simpler build, but makes it difficult for operators to depend on only the parts they need. Feature flags don't provide the same clean separation as separate crates with trait boundaries. + +**Fork-based customisation** — rejected. The current model. Leads to divergent forks that don't benefit from upstream improvements. Trait-based extension is strictly preferable. + +**Configuration file instead of trait implementations** — considered. Would allow operators to customise behaviour via YAML/TOML without writing Rust. Rejected: insufficient expressiveness for the range of customisation needed (custom auth flows, custom middleware, custom policy stores). Configuration can complement traits but cannot replace them. diff --git a/adrs/009-configuration.md b/adrs/009-configuration.md new file mode 100644 index 0000000..f0de2a8 --- /dev/null +++ b/adrs/009-configuration.md @@ -0,0 +1,105 @@ +# ADR-009: Configuration Layer — Policy Store Implementation and Caching Strategy + +**Status:** Pending +**Date:** 2026-03-14 +**RFC:** RFC-001 §12 +**Depends on:** ADR-005 + +--- + +## Context + +The authorization model (ADR-005) requires per-request lookups against a policy store for every non-public authenticated request. This is not optional — it is what enables dynamic permissions to reflect changes (new organisations, new dataset grants) in near real-time. Unlike a design that encodes permissions in the session token, this design explicitly trades token self-sufficiency for permission freshness. + +This constraint means the policy store is on the **hot path** of every authenticated request to a non-public resource. The question is not *whether* the proxy needs a policy store at request time, but *how* that access is implemented with acceptable latency and availability. + +--- + +## Decision + +### Access Patterns + +The proxy's configuration access has two distinct profiles: + +**High-frequency, latency-sensitive (per-request)** +- Bucket public flag lookup — `bucket_id -> {public, backend_config}` +- User grant lookup — `(user_id, bucket_id) -> {granted, prefix_restrictions}` +- User bucket list — `user_id -> [bucket_ids]` + +These must complete in single-digit milliseconds. In-process caching absorbs most of the load; the underlying lookup must be fast for cache misses. + +**Low-frequency, management (background)** +- Issuer JWKS refresh +- Role definition updates +- Provider credential rotation + +These are not on the request hot path and can tolerate higher latency. + +### Implementation Options (Unresolved) + +The implementation choice between the following options is unresolved and is the primary focus of RFC review: + +**Option A — REST API intermediary with aggressive caching** + +The proxy calls the existing Source Cooperative API for configuration lookups, wrapped in multi-layer caching: in-process (per-isolate or per-container) with short TTL, backed by Workers KV or ElastiCache as a shared distributed cache tier. + +*Advantages:* The Next.js application remains the schema owner; the proxy does not need direct database credentials; the API can enforce schema constraints. +*Risks:* The REST API is an availability dependency on the hot path. A cache miss on a cold Workers isolate hitting a degraded API directly impacts request latency. + +**Option B — Direct DynamoDB access** + +The proxy connects directly to DynamoDB tables for configuration lookups. In-process caching still applies. + +*Advantages:* DynamoDB read latency (single-digit milliseconds) is appropriate for the hot path; eliminates availability coupling to the Next.js application. +*Risks:* Two systems (proxy and Next.js) accessing the same DynamoDB tables creates a schema governance problem. DynamoDB's schemaless nature means there is no DDL to enforce consistency — schema drift between consumers is possible and difficult to detect until runtime failure. + +**Option C — Proxy as data model authority** + +The proxy owns and is the sole writer of the policy store schema. The Next.js application reads policy data through the proxy's API. + +*Advantages:* Single schema owner eliminates drift risk. +*Risks:* Expands the proxy's scope; requires refactoring the Next.js application; tightly couples front-end and proxy deployment cycles. + +**Hybrid option** — Direct DynamoDB for high-frequency per-request lookups (bucket flags, user grants); REST API for management operations (issuer registration, role updates). + +### Workers Caching Stack + +For the Workers deployment: + +- **In-process cache** — per-isolate, not shared across edge nodes, with TTLs from ADR-005 +- **Workers KV** — eventually consistent, globally distributed key-value store; serves as a shared cache tier that survives isolate recycling + +For access control decisions, eventual consistency is generally acceptable — a grant created seconds ago but not yet visible in KV is a minor inconvenience, not a security failure. + +### Unresolved + +- The implementation choice between Options A, B, C, and the hybrid is the primary open question. See RFC-001 Open Question 2. +- The full caching stack for Workers (which lookups use Workers KV vs. in-process only, cache warming strategy for cold isolates) requires further design. + +--- + +## Consequences + +**Benefits** + +- Per-request policy resolution enables dynamic permissions without token re-exchange +- In-process caching absorbs the majority of lookup load +- Workers KV provides a shared cache tier for the edge deployment +- The configuration layer is behind a trait interface, allowing different implementations per deployment + +**Costs / Risks** + +- The policy store is a single point of failure for authenticated requests to non-public resources +- Cache misses on cold Workers isolates add latency to the first request +- Schema governance between the proxy and Next.js application is a risk regardless of implementation choice +- The implementation decision is blocked pending team discussion + +--- + +## Alternatives Considered + +**Encode permissions in the session token (no policy store on hot path)** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens after any permission change. See ADR-005. + +**Global strongly-consistent cache (e.g. Durable Objects)** — considered. Would eliminate eventual-consistency concerns. Rejected: Durable Objects are single-region, adding latency for global edge requests. Eventual consistency is acceptable for the access control use case. + +**Push-based cache invalidation** — considered. The policy store pushes updates to Workers KV when grants change, rather than relying on TTL-based expiry. Worth exploring as an optimisation but adds operational complexity. Deferred. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md new file mode 100644 index 0000000..f756660 --- /dev/null +++ b/adrs/rfc-001.md @@ -0,0 +1,638 @@ +# RFC-001: Source Cooperative Data Proxy Re-Architecture + +**Status:** Draft — Request for Comment +**Date:** 2026-03-14 +**Authors:** @alukach +**Replaces:** Current data proxy (ECS, Rust, long-lived credentials) +**Version:** 3 — see [Changelog](#changelog) for changes from v2 + +--- + +## Purpose + +This document describes the proposed re-architecture of the Source Cooperative data proxy. It is written to give contributors, maintainers, and stakeholders a complete picture of the system design — including the reasoning behind each major choice — and to invite critique before decisions are ratified. Open questions are explicitly called out throughout. + +This is not a final specification. It is the basis for a conversation. Decisions that emerge from review will be captured in individual Architecture Decision Records (ADRs) referenced in the [Decision Index](#decision-index). + +--- + +## Table of Contents + +1. [Background](#1-background) +2. [Goals and Non-Goals](#2-goals-and-non-goals) +3. [System Overview](#3-system-overview) +4. [S3 API Compatibility](#4-s3-api-compatibility) +5. [Runtime and Deployment](#5-runtime-and-deployment) +6. [Implementation Language — Rust](#6-implementation-language--rust) +7. [Inbound Authentication](#7-inbound-authentication) +8. [Authorization](#8-authorization) +9. [Outbound Connectivity](#9-outbound-connectivity) +10. [Extensibility — Middleware and Metering](#10-extensibility--middleware-and-metering) +11. [Modular Architecture and Community Reuse](#11-modular-architecture-and-community-reuse) +12. [Configuration and Data Layer](#12-configuration-and-data-layer) +13. [Open Questions](#13-open-questions) +14. [Decision Index](#14-decision-index) + +--- + +## 1. Background + +Source Cooperative is a platform for hosting and distributing geospatial and scientific datasets. The data proxy is the component responsible for serving those datasets to end users and automated systems — translating authenticated requests into reads (and writes) against object storage backends across multiple cloud providers. + +### Current System + +The current proxy is a Rust service deployed to AWS ECS. It: + +- Exposes an S3-compatible API +- Authenticates callers using long-lived static access key / secret key pairs issued per user +- Calls an external REST API (built into the Source Cooperative Next.js frontend) to resolve configuration, which in turn reads from DynamoDB +- Implements per-backend cloud storage adapters manually, mapping each backend client's error surface to internal error types +- Has Source Cooperative's data model and backend structure baked in, with minimal support for external operators or community extension + +### Motivation for Re-Architecture + +Several pressures have converged to make a re-architecture worthwhile rather than incremental: + +- **Credential model:** Long-lived static credentials are a persistent security liability. The industry has largely moved to short-lived, exchanged credentials via OIDC workload identity federation — a pattern that Source Cooperative should both support and exemplify. +- **Global performance:** Users far from `us-west-2` (where most data resides) experience significant latency. Mirroring data to additional regions is cost-prohibitive. A CDN-native deployment can route traffic more efficiently without duplicating storage. +- **Extensibility:** The current proxy is difficult to reuse or extend. The broader community — data providers, researchers, institutions — would benefit from a proxy they could deploy and adapt. This requires a clean separation between the Source Cooperative-specific layer and the general-purpose framework beneath it. +- **Operational scope:** We anticipate supporting data providers who bring their own datasets and infrastructure, using Source Cooperative as a distribution and access-control layer. The current architecture cannot accommodate this model cleanly. + +--- + +## 2. Goals and Non-Goals + +### Goals + +- Full S3 API compatibility, consumable by existing S3 SDKs and tooling without modification +- Short-lived, scoped credentials only — no long-lived static keys +- A globally distributed deployment model that improves latency for international users without data replication costs +- Support for any standards-compliant OIDC identity provider as an authentication source +- An authorization model expressive enough to support per-dataset, per-user, and per-organisation access control, rate limiting, metering, and billing hooks +- A modular, trait-based Rust implementation that the community can depend on, extend, and contribute to +- First-class support for data providers hosting their own datasets through Source Cooperative's access control and distribution layer +- Support for all object storage backends provided by the `object_store` crate, including AWS S3, GCS, Azure Blob Storage, Cloudflare R2, and HTTP + +### Non-Goals + +- Replacing or re-implementing upstream object storage (we proxy to existing cloud storage backends) +- Supporting storage backends not covered by the `object_store` crate in this iteration +- Building a general-purpose CDN or storage product + +--- + +## 3. System Overview + +The re-architected proxy consists of two complementary deployment targets that share a common Rust core: + +**Cloudflare Workers (primary, global)** +A WASM-compiled Rust service deployed across Cloudflare's global edge network. Handles the majority of traffic. Requests are routed through the Cloudflare network to upstream object storage, reducing latency for users far from origin storage regions. Suited for read-heavy, latency-sensitive workloads. + +**Regional ECS deployments (secondary, on-demand)** +Traditional containerised Rust services deployed into specific cloud regions on demand. Intended for high-throughput, in-region workflows — for example, a Databricks cluster in `us-west-2` reading large volumes of data from an S3 bucket in the same region, where egress fees and network hops are a concern. Access to regional deployments should be restricted to in-region consumers (see [Open Questions](#13-open-questions)). + +Each deployment target hosts its own STS endpoint at `/.sts`. The Workers deployment and all regional ECS deployments share the same signing key material, so session credentials issued by any target are valid across all targets. Whether credential interoperability across targets is a required property — versus each target issuing and verifying its own credentials independently — is an open question worth discussing. + +``` + ┌──────────────────────────────────────────┐ + │ Caller / Workflow │ + └──────────────┬───────────────────────────┘ + │ 1. Present OIDC token + │ to /.sts on either target + ▼ + ┌─────────────────────────┐ ┌──────────────────────────┐ + │ Cloudflare Workers │ │ Regional ECS Proxy │ + │ /.sts + proxy │ │ /.sts + proxy │ + └────────────┬────────────┘ └─────────────┬────────────┘ + │ 2. Short-lived SigV4 creds │ + │ (shared keys; interoperable)│ + │ 3. Authenticated S3 request │ + ▼ ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ Upstream Object Storage (S3, GCS, R2, Azure, …) │ + └──────────────────────────────────────────────────────────────┘ +``` + +--- + +## 4. S3 API Compatibility + +### Why S3 + +The S3 API has become the de facto standard protocol for object storage access. The ecosystem of compatible tooling is vast: AWS SDKs in every major language, CLI tools (`aws s3`, `rclone`), data frameworks (DuckDB, Polars, PyArrow, fsspec, GDAL/VSI), orchestration systems (Airflow, Dagster, Prefect), and notebook environments all speak S3 natively. + +By exposing an S3-compatible surface, Source Cooperative becomes immediately accessible to this entire ecosystem without requiring callers to install or learn Source-specific client libraries. This was true of the current proxy and remains true of the re-architecture. + +### Credential Model Change + +The standard SigV4 model assumes long-lived static `Access Key ID` / `Secret Access Key` pairs. We are making a deliberate departure: **Source Cooperative will not issue or accept long-lived static credentials.** All SigV4 credentials are temporary session credentials — the same triplet shape (`AccessKeyId`, `SecretAccessKey`, `SessionToken`) that AWS STS issues — with a bounded TTL. + +Callers must exchange a trusted identity token for session credentials before making S3 API calls. This is a one-time step per session that all major AWS SDKs support natively via the credential provider chain. For data tooling that accepts credentials via environment variable (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`), the experience is identical to using STS-derived credentials against AWS itself. + +This choice is a meaningful friction increase for users accustomed to copying a static key into a config file. That friction is intentional: it pushes users toward credential patterns that are auditable, short-lived, and composable with modern workload identity systems. Documentation and CLI tooling should minimise the practical burden of the exchange step. + +### SigV4 Verification and Stateless Session Tokens + +Incoming requests carry a SigV4 `Authorization` header. The proxy verifies the signature using the `SecretAccessKey` embedded within the `SessionToken` itself. + +Session tokens are **stateless signed JWTs**. The token payload encodes the minimum information needed to identify the caller and verify the signature: `user_id`, `role_id`, `AccessKeyId`, `SecretAccessKey`, and `exp`. Crucially, the token does **not** encode the caller's full permission set — permissions are dynamic and are resolved per-request from the configuration layer (see §8). This mirrors how AWS STS works: the token asserts role membership, and the role's current policies are evaluated live on each API call. + +The proxy validates the JWT signature against its own public key and enforces the `exp` claim. No external store lookup is required to verify the token itself. The subsequent per-request permission resolution is a separate step, covered in §8. + +This design has a deliberate trade-off: **session tokens cannot be revoked once issued.** A compromised token remains valid until its `exp`. Short TTLs (e.g. 15–60 minutes) limit the blast radius. Immediate revocation is explicitly out of scope for this iteration. + +--- + +## 5. Runtime and Deployment + +### Cloudflare Workers (Primary) + +The primary deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. This choice is motivated by several properties of the Workers platform: + +**Global distribution without operational overhead.**[^1] Workers deploy to Cloudflare's edge network (330+ locations worldwide) automatically. Requests are served from the location closest to the caller, and onward routing to upstream object storage traverses the Cloudflare backbone[^2] rather than the public internet. For users in Europe, Asia-Pacific, or South America accessing data stored in `us-west-2`, this meaningfully reduces latency without requiring us to replicate the underlying data. + +**Effectively no cold start.**[^3][^4] Workers use the V8 isolate model rather than container-based execution. Historically, Cloudflare pre-warmed Workers during the TLS handshake of incoming requests — because isolates initialised in single-digit milliseconds, the cold start could be entirely hidden within handshake latency. As Workers have grown to support larger, more complex applications, Cloudflare introduced a "Shard and Conquer" technique using consistent hashing to coalesce traffic onto warm instances, achieving a sustained warm request rate of 99.99%.[^5] In practice, callers should expect effectively zero cold start latency for a proxy workload. + +**No Cloudflare-imposed egress fees.** Cloudflare does not charge for bandwidth egress from Workers, regardless of response payload size. Note that egress fees charged by upstream object storage providers (AWS S3, GCS, etc.) still apply — Cloudflare's no-egress policy eliminates the additional layer of transfer charges that would otherwise be imposed by the compute platform itself. + +**No wall clock restrictions.** Unlike some serverless platforms, Cloudflare Workers do not impose a wall-clock timeout on in-flight requests. CPU time limits apply per invocation (configurable up to 5 minutes on paid plans), but streaming large objects through the proxy does not risk being killed mid-response due to elapsed time. + +**Predictable, low cost.** The Workers paid plan charges $0.30 per million requests and $0.02 per million CPU milliseconds, with a $5/month base subscription that includes 10 million requests and 30 million CPU milliseconds. For a streaming proxy workload with low per-request CPU time, this is highly favourable. There are no additional charges for data transfer through Cloudflare. + +**WASM compatibility.** The Workers runtime supports WASM natively. Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`), making the Workers target a natural fit for a Rust-implemented proxy. + +[^1]: Cloudflare operates a global anycast network across 330+ cities. See [Cloudflare Network](https://www.cloudflare.com/network/). +[^2]: Cloudflare's backbone is a private network interconnecting its data centres, used to route traffic between edge nodes and origin servers without traversing the public internet. See [Cloudflare Network Interconnect](https://www.cloudflare.com/network-interconnect/). +[^3]: Cloudflare blog: [Eliminating Cold Starts with Cloudflare Workers](https://blog.cloudflare.com/eliminating-cold-starts-with-cloudflare-workers/) — describes the original TLS handshake pre-warming technique. +[^4]: Cloudflare Workers docs: [How Workers Works](https://developers.cloudflare.com/workers/reference/how-workers-works/) — isolates start ~100× faster than a Node.js process in a container. +[^5]: Cloudflare blog: [Eliminating Cold Starts 2: Shard and Conquer](https://blog.cloudflare.com/eliminating-cold-starts-2-shard-and-conquer/) — describes the consistent hashing technique that achieves a 99.99% warm request rate. + +### Regional ECS Deployments (Secondary) + +Regional ECS deployments will be made available on demand for in-region, high-throughput workflows. The motivating case is a data pipeline running inside the same cloud region as its source data — for example, a Spark or Databricks job in `us-west-2` reading terabytes of data from S3 in the same region. In this scenario: + +- Egress fees are zero or near-zero when traffic stays within the region +- Network throughput is higher and latency is lower than routing through an edge node +- The Cloudflare Workers path adds unnecessary hops + +Regional deployments share the same Rust core as the Workers deployment. The deployment wrapper and runtime adapter differ; the proxy logic, auth, and authz layers are identical. + +**Access restriction.** Regional proxies should only be accessible to in-region consumers — allowing general internet access would route traffic that should go to the edge through a less optimal path and would undercut the cost model. + +> [!NOTE] +> **Open Question:** How do we restrict regional proxy access to in-region consumers? Options include: VPC-only exposure (no public endpoint), IP allowlisting by cloud provider IP ranges, requiring a region-specific audience claim in the STS exchange, or issuing regional-specific session credentials that are only accepted by the corresponding regional proxy. Each has tradeoffs around operational complexity and developer experience. See [Open Questions](#13-open-questions). + +### Deployment Topology + +> [!NOTE] +> **TODO:** Diagram the full deployment topology, including how the STS exchange endpoint is deployed across both Workers and ECS targets, how JWKS endpoints are cached at the edge, and how regional ECS instances are provisioned and registered. + +--- + +## 6. Implementation Language — Rust + +We are continuing with Rust as the implementation language for both the Workers and ECS deployments. The reasoning is as follows: + +**WASM maturity.** Rust has the most mature and production-ready toolchain for compiling to WebAssembly of any systems language. The `worker-rs` crate provides idiomatic bindings to the Cloudflare Workers runtime. This is not a bet on an emerging capability — it is a well-trodden path. + +**Performance.** For the regional ECS deployment, raw throughput matters. Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that may stream large objects with tight latency requirements. This was already proven by the current proxy. + +**Type system and correctness.** The proxy handles authentication tokens, credential issuance, cryptographic signature verification, and access policy evaluation. Rust's type system — and in particular its trait system — makes it practical to encode invariants that would be runtime errors in other languages. This is increasingly valuable in a codebase where AI-assisted development is part of the workflow: a strong type system provides a correctness harness that catches generated code that compiles but violates domain constraints. + +**Community familiarity.** The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known, but is not suitable for the WASM target and carries runtime overhead incompatible with the regional proxy's performance goals. Rust is the best fit given the actual pool of contributors. + +**Trait-based extensibility.** The Rust trait system is central to the modularity goals described in [Section 11](#11-modular-architecture-and-community-reuse). Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. This is difficult to achieve cleanly in languages without a comparable abstraction. + +--- + +## 7. Inbound Authentication + +### Design Principle + +Source Cooperative will not issue or accept long-lived static credentials. All authentication flows terminate in short-lived SigV4 session credentials obtained through a token exchange. The exchange endpoint at `/.sts` acts as a Security Token Service that accepts JWTs from trusted OIDC identity providers and returns temporary credentials. + +### STS Token Exchange + +The STS endpoint accepts a signed JWT from any registered trusted issuer and returns a session credential triplet: + +``` +POST /.sts/assume-role-with-web-identity +→ { AccessKeyId, SecretAccessKey, SessionToken, Expiration } +``` + +The STS: +1. Resolves the issuer from the JWT `iss` claim +2. Fetches and caches the issuer's JWKS to verify the signature +3. Evaluates claim-based conditions against the registered role mapping +4. Issues a short-lived stateless session token (see §4) containing `user_id`, `role_id`, and `exp` + +Trust is established per-issuer by configuration: register the issuer URL, JWKS endpoint, and claim conditions. No code changes are required to add a new issuer. + +### Supported Identity Sources + +#### Ambient OIDC — CI/CD and Managed Compute + +Workflows running in environments that provide ambient OIDC tokens can exchange them directly. These environments require no stored secrets: + +| Platform | OIDC Issuer | Distinguishing Claims | +| --------------------------- | ---------------------------------------- | ----------------------------------------------- | +| GitHub Actions | `token.actions.githubusercontent.com` | `repository`, `ref`, `environment` | +| GitLab CI/CD | `https://gitlab.com/` | `project_path`, `ref_type`, `environment` | +| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | +| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | +| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | +| AWS (EC2, ECS, Lambda) | Instance/task metadata service | IAM role identity | +| GCP (Cloud Run, GKE) | Metadata server | Service account, audience-scoped | +| Azure (AKS, Container Apps) | Managed Identity / Entra ID | Federated credential | +| Databricks Jobs | Platform SDK / env injection | Workspace, job, cluster | + +This list is illustrative, not exhaustive. Any issuer with a publicly reachable JWKS endpoint can be registered. + +#### Source Cooperative Auth (`auth.source.coop`) + +Users authenticated interactively via Source Cooperative's Ory-based auth system may present their Ory access token to the STS exchange endpoint. The STS validates it against Ory's JWKS (`auth.source.coop`) and issues session credentials. This is appropriate for: + +- Interactive local development +- Ad-hoc data access from a notebook or terminal +- CLI tooling that performs a browser-based login flow + +It is not suited to unattended pipelines, as it requires an active user session. + +#### Source Cooperative Credential Tokens + +For unattended workflows that lack ambient OIDC tokens — local scripts, third-party orchestrators, environments not listed above — users may generate a **Source Cooperative Credential Token** from the platform UI or API. + +These tokens are: + +- Signed JWTs issued by Source Cooperative, using the same private key infrastructure as our outbound OIDC tokens +- Validated via our published JWKS through the same STS exchange path as all other inbound tokens +- Scoped at mint time: the JWT payload encodes user identity, permitted datasets/collections, and the internal role to assume +- Subject to a mandatory maximum TTL (e.g. 90 days) enforced at exchange time +- **Not revocable** — consistent with the stateless session token model. A compromised credential token remains exchangeable until its TTL expires. Short TTLs and narrow scope limit the blast radius. + +The user stores the token in their workflow's secret manager. At runtime, the workflow exchanges it for short-lived session credentials via the standard `/.sts` endpoint — the same flow used by all other issuers. + +| Mechanism | Suited for | Stored secret? | Credential TTL | +| --------------------------------------- | ---------------------- | ------------------ | -------------- | +| Ambient OIDC (GitHub, Databricks, etc.) | CI/CD, managed compute | No | Minutes | +| `auth.source.coop` / Ory token | Interactive local dev | No (session-based) | Minutes | +| SC Credential Token | Unattended pipelines | Yes (token, ≤90d) | Minutes | + +### Next.js and Front-End Authentication + +The Source Cooperative web application (Next.js, deployed on Vercel) requires access to the proxy for two distinct caller types: authenticated users and anonymous visitors. + +**Authenticated users — client-side STS exchange** + +When a user is logged in, the browser holds an Ory session token from `auth.source.coop`. Rather than routing proxy requests through the Next.js server, the client exchanges the Ory token directly with `/.sts` to obtain short-lived session credentials. All subsequent S3 API calls are made directly from the browser to the proxy using those credentials. + +This approach is preferred because: +- Access is recorded in proxy metrics under the user's own identity, not aggregated under a server-side service account +- The Next.js server does not handle S3 credentials at all — it remains stateless with respect to data access +- The user's current permissions are always reflected (see §8 — permissions are resolved per-request, not embedded in the token) + +**Anonymous visitors — Vercel OIDC service identity** + +Anonymous visitors have no Ory token. Since the Next.js application is deployed on Vercel, it has access to a Vercel-issued OIDC token (`VERCEL_OIDC_TOKEN`) in server-side contexts. The Next.js server exchanges this token at `/.sts` for a session credential bound to a `public-read` role. This credential can be embedded in the rendered page or used server-side to proxy requests on behalf of anonymous visitors. + +The `public-read` role is constrained to read-only operations on buckets flagged as publicly accessible. All anonymous traffic still flows through the proxy's full auth/authz/middleware stack, meaning rate limiting and access metrics apply even to unauthenticated requests. + +**Admin users** + +Admin users authenticate the same way as standard users (Ory token → STS exchange). Their `user_id` maps to the `admin` role at exchange time. The admin role bypasses resource-level permission checks — see §8. + +### CLI and SDK Support + +The credential exchange step should be invisible for common workflows. We intend to provide: + +- A **Source Cooperative CLI** that supports `source login` (browser-based device flow via `auth.source.coop`) and `source credentials export` (writes STS-derived session credentials to environment or AWS credentials file format, consumable by any S3 SDK or tool) +- An **SDK helper** (initially Python, given the data engineering audience) that implements a credential provider wrapping the STS exchange, compatible with `boto3`'s credential provider chain and `fsspec`/`s3fs` configuration + +> [!NOTE] +> **TODO:** Define the CLI command surface and SDK packaging strategy. Determine whether the CLI performs the STS exchange itself or delegates to the AWS CLI credential provider chain. + +--- + +## 8. Authorization + +### Design Principles + +Two properties drive the authorization design: + +1. **Permissions are dynamic.** A user who creates a new organisation or dataset should be able to access it immediately. Permissions cannot be frozen into the session token at exchange time — they must be resolved from the policy store on each request, with short-lived caching to reduce latency. + +2. **The role is a ceiling; user permissions are the grants.** The role answers "what classes of action are permitted for this identity type?" The per-user permission lookup answers "which specific resources can this identity access?" The proxy enforces the intersection. These are separate lookups with different cache characteristics. + +This mirrors how AWS IAM works: a session token asserts role membership, and the role's current policies are evaluated live on each API call against the IAM policy store. + +### Identity Model + +The session token (§4) carries three fields relevant to authorization: + +- `user_id` — stable identifier for the authenticated principal +- `role_id` — one of: `anonymous`, `authenticated_user`, `admin` +- `exp` — token expiry; checked before any policy evaluation + +### Role Definitions + +**`anonymous`** +- Permitted action classes: read-only (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`) +- Role-level filter: only buckets flagged `public = true` are visible and accessible +- No user permission lookup — role filter is the only guard + +**`authenticated_user`** +- Permitted action classes: read and write (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`, `ListObjects`, `ListBuckets`, `CreateBucket`) +- Role-level filter: none — user permission lookup determines which resources are accessible +- User permission lookup is always performed for non-public resources + +**`admin`** +- Permitted action classes: all +- Role-level filter: none +- User permission lookup: **skipped** — admin role has unconditional access to all resources +- Admin role assumption should be gated on strong identity claims (e.g. a specific Ory group membership claim) to prevent accidental privilege escalation + +### Per-Request Resolution Strategy + +Authorization resolution proceeds in at most three steps, with early exits to minimise unnecessary lookups: + +**Step 1 — Role action check** +Does this role permit the requested action class? If the role is `anonymous` and the request is a `PutObject`, deny immediately. This check is pure in-memory logic against the role definition — no lookup required. + +**Step 2 — Public resource early exit** +For read operations only: is the requested bucket flagged `public = true` in the policy store? If yes, and the role permits reads, permit immediately without a user permission lookup. This early exit covers the majority of Source Cooperative traffic (public dataset access) and avoids a per-user lookup for every anonymous or authenticated read of public data. The `public` flag is cached aggressively (60–300 seconds) since it changes rarely. + +**Step 3 — User permission lookup** +For non-public resources, or write operations: fetch the user's permissions from the policy store and evaluate them against the requested resource. This is the dynamic lookup that reflects current membership in organisations, ownership of datasets, and any explicit grants. + +### Operation-Specific Behaviour + +**Single-resource operations (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`)** +After the role check and public early exit, a single point lookup is performed: does `(user_id, bucket_id)` resolve to an access grant? If the grant exists and includes any prefix restrictions, those are enforced against the requested object key. Deny if no grant exists. + +**`ListBuckets`** +This operation cannot be delegated to the upstream — the upstream would return all buckets in the cloud account, not the user's logical buckets. The proxy must construct the `ListBuckets` response entirely from the policy store: + +1. Role check: does the role permit `ListBuckets`? +2. If `anonymous`: fetch all buckets with `public = true` from the policy store +3. If `authenticated_user`: fetch all bucket IDs the user has an access grant for; apply role filter +4. If `admin`: return all buckets +5. Construct and return the response — no upstream call is made + +Because `ListBuckets` requires a full enumeration of the user's accessible buckets, it is the most expensive and freshness-sensitive lookup. A short cache TTL (5–10 seconds) is warranted here — this is the operation most likely to reflect a user's recent creation of a new organisation or dataset. + +**`ListObjects` (within a bucket)** +1. Role check: does the role permit `ListObjects`? +2. Public early exit: is the bucket public? +3. User permission lookup: does `(user_id, bucket_id)` have an access grant? +4. If the grant includes a key prefix restriction, pass it as a filter to the upstream `ListObjects` call so the upstream enforces the boundary, rather than filtering the result set after the fact + +### Cache Strategy + +All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container instance). Cache keys and TTLs: + +| Lookup | Cache Key | TTL | +| --------------------------------------- | ---------------------- | ------------------------------------------------ | +| Role definition | `role_id` | In-memory constant — role definitions are static | +| Bucket public flag | `bucket_id` | 60–300s | +| Single-resource user grant | `(user_id, bucket_id)` | 30–60s | +| User's full bucket list (`ListBuckets`) | `(user_id, role_id)` | 5–10s | + +The short TTL on the full bucket list (5–10 seconds) ensures that a user who creates a new dataset or joins an organisation sees that change reflected in `ListBuckets` within a few seconds — acceptable for a UI interaction, where the user would typically navigate to the new resource after creating it. + +For the Workers deployment, cache is per-isolate and is not shared across edge nodes. This is acceptable: the worst case is a single additional policy store lookup per edge node per cache interval, not a cache miss storm. Workers KV is available for a globally consistent cache tier if per-isolate TTLs prove insufficient. + +> [!NOTE] +> **TODO:** Define the exact schema of user access grants in the policy store — whether grants are bucket-level only or support sub-bucket prefix granularity; whether grants are additive (explicit allows) only or also support explicit denies; and how organisation membership is modelled (membership → derived grants, or explicit per-bucket grants per member). + +--- + +## 9. Outbound Connectivity + +### Design Principle + +When the proxy receives an authenticated, authorised request, it must retrieve or write the underlying object from an upstream storage backend (S3, GCS, Azure Blob, R2, etc.). This connection to upstream storage must itself be authenticated, without embedding long-lived cloud credentials in the proxy service. + +### Current Approach and Its Limitations + +The current proxy implements per-backend adapters manually — a separate integration for each cloud storage provider, with bespoke error mapping from each provider's client library to Source Cooperative's internal error types. This is maintenance-intensive and creates an ongoing gap as new backends are added or existing client APIs change. + +### `object_store` Adoption + +The re-architecture adopts the [`object_store`](https://crates.io/crates/object_store) crate as the unified abstraction layer for upstream storage access. `object_store` provides a single async trait (`ObjectStore`) with implementations for S3, GCS, Azure Blob, R2, HTTP, and local filesystem. By building on this abstraction: + +- Backend-specific client code and error mapping is eliminated from the proxy codebase +- New storage backends supported by `object_store` become available to the proxy without proxy changes +- The community can contribute additional `object_store` implementations that the proxy can consume + +### Outbound Authentication + +Two mechanisms for authenticating to upstream backends are supported: + +**OIDC token issuance (preferred)** + +Source Cooperative operates as an OIDC identity provider, publishing a discovery document (`/.well-known/openid-configuration`) and JWKS endpoint. Upstream cloud providers (AWS, GCP, Azure) can register Source Cooperative as a trusted external identity provider via their native workload identity federation mechanisms. The proxy then generates short-lived, audience-scoped JWTs and exchanges them for cloud credentials at each provider's STS — no long-lived cloud credentials are stored in the proxy. + +This is the preferred model: credentials are ephemeral, the trust relationship is declarative and auditable, and key rotation at the proxy level propagates automatically without reconfiguring upstream providers. + +**Stored credential secrets (fallback)** + +For upstream providers or storage systems that do not support OIDC workload identity federation, credentials may be stored as encrypted secrets and injected into the proxy's configuration at startup. This is a fallback, not the preferred path, and should be documented as such. + +### Data Provider Hosting + +Beyond serving Source Cooperative's own managed datasets, the proxy is intended to support **data providers** who bring their own storage infrastructure. In this model, a data provider registers their upstream storage (their own S3 bucket, GCS bucket, etc.) with Source Cooperative, and the proxy serves as an access control, metering, and distribution layer in front of their data. + +This model offers data providers: + +- **Cost control:** Rate limiting, metering, and access thresholds enforced by the proxy prevent runaway egress or compute costs on the provider's underlying storage +- **Access control:** Fine-grained role and policy configuration determines who can access which datasets under what conditions +- **Exposure:** Data is discoverable and accessible via the Source Cooperative platform and UI, without the provider having to build their own access layer +- **Outbound auth flexibility:** The provider's own cloud credentials (or OIDC trust relationship) are used for the proxy's outbound connection to their storage — the provider retains ownership of their backend + +> [!NOTE] +> **TODO:** Define the operational model for data provider onboarding. Clarify how outbound credentials for provider-hosted storage are stored and scoped — whether they are isolated per-provider or share infrastructure with Source Cooperative's own backend credentials. Define the trust boundary: what can Source Cooperative see or access in a provider's backend? + +--- + +## 10. Extensibility — Middleware and Metering + +### Motivation + +A general-purpose data proxy needs to support behaviours beyond simple authentication and object retrieval. Source Cooperative specifically requires: + +- **Rate limiting** that is fine-grained and dynamic: different limits per bucket, per user, per organisation, or per role — not a single global rate +- **Data metering:** tracking cumulative data transfer per identity or dataset, and enforcing access thresholds (e.g. denying access once a monthly quota is reached) +- **Usage tracking and billing hooks:** recording access events with enough fidelity to support downstream billing — charging data providers, charging consumers, or both +- **Audit logging:** a complete, tamper-resistant record of who accessed what and when + +These concerns are cross-cutting: they apply to every request regardless of the specific storage backend or dataset, but their configuration and behaviour differ across deployments and use cases. + +### Middleware Pattern + +These concerns are implemented as a **middleware stack** composing around the core request handler. Each middleware layer: + +- Receives the request context (resolved identity, role, resource, action) and may modify or enrich it +- May short-circuit the request with a denial response (e.g. quota exceeded, rate limit hit) +- May record an event (e.g. to a metering store or audit log) +- Passes the request to the next layer if permitted + +The middleware stack is configured per-deployment and potentially per-dataset. A dataset with no billing requirements carries a lightweight stack; a provider-hosted dataset with metered access carries additional quota-check and event-recording middleware. + +Middleware components are defined as Rust traits, making them first-class extension points for community contributions. Source Cooperative ships a set of standard middleware implementations; operators can add their own without forking the core. + +### Standard Middleware (Planned) + +| Middleware | Behaviour | +| --------------- | -------------------------------------------------------------------------------------- | +| Rate limiter | Per-identity or per-bucket request rate enforcement, configurable limits | +| Quota enforcer | Cumulative data transfer tracking; deny on threshold exceeded | +| Usage recorder | Structured event emission per request (bytes transferred, identity, resource, latency) | +| Audit logger | Tamper-evident request log for compliance and forensics | +| Billing emitter | Usage event publication to a configurable billing backend | + +> [!NOTE] +> **TODO:** Define the middleware trait interface in detail. Specify how middleware configuration is expressed (per-deployment, per-bucket, per-role). Determine the event schema for usage recording and billing emission, and which backend systems are targeted initially (e.g. a Kinesis stream, a webhook, a DynamoDB table). Consider how middleware ordering is enforced and whether order-dependent behaviours are made explicit. + +--- + +## 11. Modular Architecture and Community Reuse + +### Motivation + +The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. It is difficult for external operators to deploy a version of the proxy for their own datasets, and equally difficult for contributors to improve the proxy in ways that are reusable outside Source Cooperative's own deployment. + +The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration of that framework is a thin layer on top. + +### Design Approach + +The core proxy is structured as a set of Rust crates with well-defined trait boundaries between layers: + +- **`proxy-core`** — request routing, SigV4 verification, session credential management, middleware stack execution. No Source Cooperative specifics. +- **`proxy-auth`** — STS exchange logic, OIDC issuer registry, JWT validation, SC Credential Token minting. Configurable via traits; the Source Cooperative issuer list is one configuration. +- **`proxy-authz`** — role resolution, per-request policy evaluation, policy store interface trait. The IAM-inspired evaluation logic is the default; the store backend is pluggable. +- **`proxy-storage`** — `object_store`-based backend abstraction. New backends are `object_store` implementations. +- **`proxy-middleware`** — middleware trait definition and standard implementations (rate limiter, quota enforcer, usage recorder, etc.) +- **`proxy-workers`** — Cloudflare Workers runtime adapter, WASM build target +- **`proxy-ecs`** — traditional server runtime adapter, Hyper/Tokio based + +An operator building their own proxy instantiates the core with their own implementations of the configuration traits — providing their own backend resolver, their own role mapping, their own middleware stack — without forking any of the above crates. + +Source Cooperative's own deployment is the reference implementation of this pattern. + +### Community Contribution Model + +By publishing the core crates to `crates.io` under a permissive licence: + +- Community members can build their own data proxies on the same foundation +- Contributions to the core (new middleware, new storage backends, auth improvements) benefit all deployments +- Source Cooperative's own infrastructure becomes a demonstration of the framework's capabilities, which aids adoption + +> [!NOTE] +> **TODO:** Finalise crate naming, licensing, and governance model. Determine what "supported" means for community-contributed crates — whether they live in the same repository, a separate organisation, or are fully external. Define the public API stability guarantees for each crate. + +--- + +## 12. Configuration and Data Layer + +### Constraint: The Policy Store Is on the Hot Path + +The authorization model described in §8 requires the proxy to perform per-request lookups against a policy store for every non-public authenticated request. This is not optional — it is what enables dynamic permissions to reflect changes (new organisations, new dataset grants) in near real-time. Unlike the previous design (which attempted to encode permissions in the session token), this design explicitly trades token self-sufficiency for permission freshness. + +This constraint changes the framing of the configuration layer question. The question is no longer *whether* the proxy needs access to a policy store at request time — it does — but rather *how* that access is implemented with acceptable latency and availability. + +### Access Patterns + +The proxy's access to the configuration layer has two distinct profiles: + +**High-frequency, latency-sensitive (per-request)** +- Bucket public flag lookup — `bucket_id → {public, backend_config}` +- User grant lookup — `(user_id, bucket_id) → {granted, prefix_restrictions}` +- User bucket list — `user_id → [bucket_ids]` + +These must complete in single-digit milliseconds. In-process caching (§8) absorbs most of the load; the underlying lookup needs to be fast for cache misses. + +**Low-frequency, management (background)** +- Issuer JWKS refresh +- Role definition updates +- Provider credential rotation + +These are not on the request hot path and can tolerate higher latency. + +### Implementation Options + +**Option A — REST API intermediary with aggressive caching** + +The proxy calls the existing Source Cooperative API for configuration lookups, but wraps every call in a multi-layer cache: in-process (per-isolate or per-container) with short TTL, backed by Workers KV or ElastiCache for a shared distributed cache tier. + +*Advantages:* The Next.js application remains the schema owner; the proxy does not need direct database credentials; the API can enforce schema constraints before data reaches the proxy. +*Risks:* The REST API is an availability dependency on the hot path. Even with caching, a cache miss on a cold Workers isolate hitting a degraded API will directly impact request latency. The API must be engineered for the proxy's read performance requirements, not just the application's UX needs. + +**Option B — Direct DynamoDB access** + +The proxy connects directly to DynamoDB tables for configuration lookups, eliminating the REST API hop. In-process caching still applies. + +*Advantages:* DynamoDB read latency (single-digit milliseconds) is appropriate for the hot path; eliminates the availability coupling to the Next.js application. +*Risks:* Two systems (proxy and Next.js application) accessing the same DynamoDB tables creates a schema governance problem. DynamoDB's schemaless nature means there is no DDL to enforce consistency — schema drift between consumers is possible and difficult to detect until it causes a runtime failure. + +**Option C — Proxy as data model authority** + +The proxy owns and is the sole writer of the policy store schema. The Next.js application reads policy data through the proxy's API rather than directly from DynamoDB. + +*Advantages:* Single schema owner eliminates drift risk; the proxy API becomes the contract. +*Risks:* Significantly expands the proxy's scope of responsibility; requires refactoring the Next.js application's direct DynamoDB access; tightly couples front-end and proxy deployment cycles. + +> [!NOTE] +> **Open Question:** Options A and B are the most practical near-term choices. Option A is lower risk but introduces an availability dependency on the hot path; Option B is faster and more resilient but creates a schema governance problem with DynamoDB. A hybrid is possible — direct DynamoDB for the high-frequency per-request lookups (bucket flags, user grants), REST API for management operations (issuer registration, role updates) — but adds operational complexity. What is the team's appetite for managing DynamoDB schema consistency without a schema enforcement layer? + +### Configuration in Workers + +Cloudflare Workers have access to Workers KV (eventually consistent, globally distributed key-value store) and Durable Objects (strongly consistent, single-region). For the Workers deployment, the in-process cache described in §8 is per-isolate and not shared. Workers KV provides a shared distributed cache tier for policy data that survives isolate recycling and is consistent across edge nodes within its eventual-consistency window (typically seconds). + +For access control decisions, eventual consistency is generally acceptable — a grant that was created 2 seconds ago but not yet visible in KV is a minor inconvenience, not a security failure. Revocation is a different matter (not in scope for this iteration per §4). + +> [!NOTE] +> **TODO:** Design the full caching stack for the Workers deployment: in-process TTLs, Workers KV usage for shared policy cache, and the propagation path from the authoritative policy store to the edge. Specify which lookups require Workers KV vs. in-process only, and define the cache warming strategy for cold isolate starts. + +--- + +## 13. Open Questions + +The following questions are unresolved and are the primary focus of this RFC review. Answers will be captured in the ADRs listed in [Section 14](#14-decision-index). + +1. **Regional proxy access restriction.** How do we ensure regional ECS proxy deployments are only accessible to in-region consumers? VPC-only endpoints, IP range allowlisting, region-scoped session credentials, and audience claims are candidate mechanisms. What are the operational tradeoffs? + +2. **Configuration store implementation.** Given that the policy store is definitively on the hot path, should we use the REST API with aggressive caching (lower risk, availability dependency) or direct DynamoDB access (faster, schema governance risk)? A hybrid approach is also possible. What schema enforcement mechanisms can mitigate the DynamoDB drift risk in Option B? + +3. **STS endpoint deployment.** Both Workers and ECS targets host `/.sts`. How is the JWKS cache managed consistently across the edge? Is there a risk of JWKS cache inconsistency between edge nodes causing transient validation failures? + +4. **Credential interoperability across targets.** Is it a requirement that session credentials issued by the Workers STS be valid against a regional ECS proxy and vice versa? If so, what operational procedures govern shared key material and rotation across deployment targets? + +5. **Outbound OIDC vs. stored secrets — default and fallback.** For provider-hosted datasets where the provider's cloud does not support OIDC federation, what is the operational model for storing and rotating credentials securely? + +6. **Middleware event backend.** What is the initial target for usage recording and billing event emission? A push-based stream (Kinesis, Pub/Sub), a pull-based log (S3, R2), a webhook, or something else? + +7. **Policy language and grant schema.** Are grants bucket-level only, or do they support sub-bucket prefix granularity? Are explicit denies supported, or is the model additive (allow-only)? How is organisation membership modelled — derived grants from membership, or explicit per-bucket grants per member? + +8. **Crate governance.** What is the publication, maintenance, and contribution model for the core crates on `crates.io`? + +--- + +## 14. Decision Index + +The following ADRs will be produced as decisions are ratified through this RFC process. Links will be added as documents are published. + +| ADR | Decision | Status | +| ------- | ---------------------------------------------------------------------- | ------- | +| ADR-001 | S3 API compatibility and temporary-credentials-only model | Draft | +| ADR-002 | Runtime: Cloudflare Workers + regional ECS strategy | Pending | +| ADR-003 | Rust as implementation language | Pending | +| ADR-004 | Inbound authentication — OIDC federation, STS, SC Credential Tokens | Draft | +| ADR-005 | Authorization model — dynamic per-request policy resolution | Pending | +| ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | Pending | +| ADR-007 | Middleware architecture — rate limiting, metering, billing hooks | Pending | +| ADR-008 | Modular crate architecture and community reuse model | Pending | +| ADR-009 | Configuration layer — policy store implementation and caching strategy | Pending | + +--- + +*This RFC is open for comment. Please raise questions, objections, and alternative proposals against the open questions in Section 13 and the design decisions throughout. The goal is collective understanding and buy-in before implementation begins.* From 4572caec6a538ee76e43f4303d4ca9619de49409 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sun, 22 Mar 2026 17:23:08 -0700 Subject: [PATCH 02/17] docs: add STS token exchange design document Covers federated identity via OIDC, user-defined Roles and IdPs, claim constraint language, permission model, credential issuance, request-time authorization, and client tooling integration. Co-Authored-By: Claude Opus 4.6 --- .../2026-03-22-sts-token-exchange-design.md | 451 ++++++++++++++++++ 1 file changed, 451 insertions(+) create mode 100644 docs/plans/2026-03-22-sts-token-exchange-design.md diff --git a/docs/plans/2026-03-22-sts-token-exchange-design.md b/docs/plans/2026-03-22-sts-token-exchange-design.md new file mode 100644 index 0000000..f7ccd88 --- /dev/null +++ b/docs/plans/2026-03-22-sts-token-exchange-design.md @@ -0,0 +1,451 @@ +# STS Token Exchange Design + +## Problem + +Source Cooperative needs federated identity and fine-grained access control for its S3-compatible data proxy. Users and automated systems (CI/CD pipelines, data workflows) must obtain temporary credentials scoped to specific permissions without long-lived API keys. + +## Core Entities + +### Identity Providers (IdPs) + +IdPs exist at two tiers: + +**Platform IdPs** are pre-configured by Source Cooperative operators: +- `auth.source.coop` (Source Cooperative's Ory-based OIDC) +- `https://token.actions.githubusercontent.com` (GitHub Actions) +- `https://gitlab.com` (GitLab CI) +- Additional issuers added by operators over time + +Platform IdPs define: +```json +{ + "id": "github-actions", + "issuer_url": "https://token.actions.githubusercontent.com", + "display_name": "GitHub Actions", + "well_known_claims": ["repository", "repository_owner", "ref", "environment", "job_workflow_ref"], + "audience_hint": "https://data.source.coop" +} +``` + +The `well_known_claims` field provides documentation and UI hints for users creating Role bindings. The `audience_hint` is the recommended `aud` value callers should configure when requesting OIDC tokens. + +**Account IdPs** are registered by account owners (Individual or Organization): +- Must use HTTPS +- Issuer URL must not collide with any platform IdP (exact match after canonicalization) +- Must not duplicate another IdP on the same account +- Must serve a valid OIDC discovery document at `/.well-known/openid-configuration` +- Resolved IP must not be private, loopback, or link-local (SSRF protection) +- Fetch timeout: 3 seconds, response body limit: 256KB + +Account IdP stored record: +```json +{ + "id": "uuid", + "account_id": "my-org", + "issuer_url": "https://corp.okta.com/oauth2/default", + "display_name": "Our Corporate Okta", + "created_at": "2025-03-22T...", + "created_by": "user-id" +} +``` + +No JWKS is stored at registration time. JWKS is fetched and cached at STS exchange time from the OIDC discovery document's `jwks_uri`. + +**IdP deletion** is blocked if any Role references the IdP. The account must first remove the IdP binding from all Roles, then delete the IdP. + +### Roles + +Roles belong to an account (Individual or Organization), identified by URN: `source::{account_id}::role/{role_name}`. + +Each Role contains: +- **Identity constraints** — which IdPs can assume this Role, with what claim requirements +- **Permission statements** — what the Role's credentials can access +- **`max_session_duration`** — ceiling on credential TTL (default 1 hour, max 12 hours) + +Role schema: +```json +{ + "name": "github-publisher", + "display_name": "GitHub CI Publisher", + "max_session_duration": 3600, + "identity_constraints": [ + { + "idp": "github-actions", + "audience": "https://data.source.coop", + "claim_constraints": [ + {"claim": "repository", "operator": "equals", "value": "my-org/my-repo"}, + {"claim": "ref", "operator": "starts_with", "value": "refs/heads/"} + ] + }, + { + "idp": "uuid-of-account-idp", + "audience": "https://data.source.coop", + "claim_constraints": [ + {"claim": "sub", "operator": "equals", "value": "service-account-42"} + ] + } + ], + "permissions": [ + { + "actions": ["read", "write"], + "resources": ["source::my-org::product/climate-data/*"] + }, + { + "actions": ["read"], + "resources": ["source::my-org::product/reference-data/*"] + } + ] +} +``` + +**Built-in default Role:** `source::{account_id}::role/_default` +- Undeletable, always exists for every account +- Constrained to the `auth.source.coop` IdP only +- Permissions: `{"actions": ["read", "write"], "resources": ["*"]}` — unlimited ceiling, account's actual permissions are the sole constraint +- Account owners can add claim constraints but cannot change the IdP binding + +### Claim Constraint Language + +Three operators, deliberately minimal: + +| Operator | Behavior | Example | +|----------|----------|---------| +| `equals` | Exact string match | `repository` equals `my-org/my-repo` | +| `starts_with` | String prefix match | `ref` starts_with `refs/heads/` | +| `glob` | Wildcard: `*` (any chars), `?` (single char) | `repository` glob `my-org/*` | + +Rules: +- All claim values coerced to strings before comparison. Arrays and objects evaluate to false. +- All constraints within a single IdP binding are ANDed. +- Multiple IdP bindings on a Role are ORed. +- Missing claims evaluate to false (fail-closed). +- Top-level claims only — no nested path traversal. +- No regex. Glob is the most expressive operator. + +### Permission Statements + +Resource pattern format: +``` +* → all resources (unlimited ceiling) +source::{account_id}::product/* → all of an account's products +source::{account_id}::product/{product_name} → entire product +source::{account_id}::product/{product_name}/* → entire product (equivalent) +source::{account_id}::product/{product_name}/{prefix}/* → prefix-scoped +source::{account_id}::product/{product_name}/{key} → single object +``` + +Rules: +- Resource patterns can reference any account's products. A Role can delegate access to products the account has access to, even if owned by another account or org. +- `*` as the entire resource value means "all resources" — no ceiling; the account's actual permissions are the sole constraint. +- `*` at the end of a pattern matches any suffix (prefix matching). `*` is valid only as the final character or as the entire value. +- Actions are `read` and `write`. `read` maps to `GetObject`, `HeadObject`, `ListObjects`. `write` maps to `PutObject`, `DeleteObject`, and multipart operations. +- Permission statements are additive (allow-only). No explicit denies. +- Roles act as a ceiling — they can never exceed the account's own permissions. The request-time intersection `(Role permissions) ∩ (account's actual permissions)` is the sole enforcement mechanism. + +### Role Validation at Creation + +1. `name` must match `[a-z0-9][a-z0-9-]{0,62}` (lowercase, hyphens, max 63 chars) +2. Each IdP reference must exist (platform IdP by well-known ID, account IdP by UUID) +3. `max_session_duration` between 900 and 43200 seconds (15 min to 12 hours) +4. At least one identity constraint required +5. At least one permission statement required +6. Maximum 10 IdP bindings per Role, 20 claim constraints per binding, 50 permission statements per Role + +## STS Token Exchange + +### Endpoint + +``` +POST /.sts/assume-role-with-web-identity +``` + +Dot-prefixed account names are reserved as invalid, preventing routing conflicts. + +**Request format:** `application/x-www-form-urlencoded`, AWS STS-compatible: +``` +Action=AssumeRoleWithWebIdentity +&WebIdentityToken= +&RoleArn=source::my-org::role/github-publisher +&RoleSessionName=my-ci-job-42 +&DurationSeconds=3600 +``` + +**Response format:** XML, AWS STS-compatible: +```xml + + + + SCSTS... + derived-secret + eyJ... + 2025-03-22T13:00:00Z + + + source::my-org::role/github-publisher + SCSTS...:my-ci-job-42 + + + +``` + +This format enables `boto3.client('sts', endpoint_url='https://data.source.coop/.sts').assume_role_with_web_identity(...)` with a custom endpoint URL. The `RoleArn` parameter accepts Source Cooperative URN format — the AWS SDK passes the string through without client-side validation. + +### Exchange Flow + +1. Parse `RoleArn` → extract `account_id` and `role_name` +2. Load Role definition from policy store (cached, 30–60s TTL) +3. Extract `iss` from JWT (without verification) +4. Match `iss` against Role's allowed IdPs — reject immediately if no match +5. Fetch JWKS from the matched IdP (cached, 1hr TTL, 3s timeout, stale-while-revalidate on failure) +6. Verify JWT signature, `exp`, `nbf` (60s clock skew tolerance), and `aud` +7. Evaluate claim constraints for the matched IdP binding +8. Validate `DurationSeconds` ≤ Role's `max_session_duration` +9. Generate credentials and return response + +### Credential Issuance + +**AccessKeyId:** Random unique identifier prefixed `SCSTS` to distinguish from permanent keys. + +**SecretAccessKey:** Derived deterministically: `HMAC-SHA256(server_secret, AccessKeyId)`. Never stored, never transmitted in the SessionToken. The server reconstructs it on each request by re-deriving from the AccessKeyId. + +**SessionToken:** A signed JWT (ES256, asymmetric) containing: +```json +{ + "jti": "", + "sub": "source::my-org::role/github-publisher", + "account_id": "my-org", + "role_name": "github-publisher", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "assumed_by_issuer": "https://token.actions.githubusercontent.com", + "session_name": "my-ci-job-42", + "access_key_id": "SCSTS...", + "permissions": [ + {"actions": ["read", "write"], "resources": ["source::my-org::product/climate-data/*"]} + ], + "iat": 1711100000, + "exp": 1711103600, + "aud": "data.source.coop", + "kid": "" +} +``` + +Key properties: +- **Permissions embedded in the token** avoid per-request policy store lookups for Role ceiling evaluation. The account's underlying permissions are still checked dynamically. +- **`assumed_by`** preserves the original IdP subject for audit trails. +- **`access_key_id`** included so the server can derive the SecretAccessKey via HMAC. +- **SecretAccessKey is NOT in the token.** + +### Signing Key Management + +- Asymmetric signing: ES256 (ECDSA P-256) +- Private key stored in KMS, used only for token issuance +- Public key served at a JWKS endpoint for verification +- `kid` in JWT header supports key rotation: sign new tokens with new key, accept old key until all tokens signed with it expire +- Rotation procedure: generate new key → sign with new key → old key valid for `max_session_duration` after last use + +### Revocation + +- Lightweight deny-list of revoked `jti` values stored in Cloudflare KV (or equivalent) +- TTL on each KV entry matches the token's remaining lifetime (self-cleaning) +- Checked on every authenticated request (KV lookup ~1ms at edge) +- `POST /.sts/revoke-session-token` endpoint, callable by account admins + +### JWKS Caching + +- Cache key: canonicalized issuer URL +- TTL: 1 hour +- Stale-while-revalidate: if JWKS fetch fails and a cached copy exists, serve stale for up to 24 hours (with warning logged) +- If no cache and fetch fails → return `IDPCommunicationError` +- Max response body: 256KB + +## Request-Time Authorization + +### Step 1: Identify the Caller + +- **No credentials** → anonymous +- **Permanent API key** (non-`SCSTS` prefix) → legacy API key lookup via Source API +- **STS credentials** (`SCSTS` prefix) → derive SecretAccessKey via HMAC, verify SigV4, decode SessionToken JWT + +### Step 2: Role Action Check (in-memory) + +For anonymous callers, only read actions are permitted. + +For STS callers, the SessionToken's embedded permissions define the ceiling. If the requested action is not covered, deny immediately. + +### Step 3: Resource Resolution + +Map the S3 request to a Source Cooperative resource: +- Bucket name → `account_id/product_name` +- Object key → path within the product + +### Step 4: Public Resource Early Exit (cached, 60–300s TTL) + +For read requests on public products (`data_mode: open`), permit immediately. No further lookups. This is the fast path for the majority of traffic. + +### Step 5: Account Permission Lookup (cached, 30–60s TTL) + +For non-public resources or write operations: +1. Fetch the account's permissions from the policy store +2. Compute: `(Role ceiling permissions) ∩ (account's actual permissions)` +3. If the intersection includes the requested action on the requested resource → permit +4. Otherwise → deny + +### Step 6: Prefix Enforcement + +If the Role's permission statement includes a prefix constraint, verify the object key falls within that prefix. + +### Authorization Truth Table + +| Caller | Resource | Account has access? | Role permits? | Result | +|--------|----------|-------------------|--------------|--------| +| Anonymous | Public product | N/A | N/A | **Allow** (read only) | +| Anonymous | Private product | N/A | N/A | **Deny** | +| STS | Public product, read | N/A | Yes | **Allow** | +| STS | Public product, write | Yes | Yes | **Allow** | +| STS | Private product | Yes | Yes | **Allow** | +| STS | Private product | Yes | No (ceiling) | **Deny** | +| STS | Private product | No | Yes | **Deny** | + +## STS Error Responses + +Errors use AWS STS XML format for SDK compatibility: + +```xml + + + InvalidIdentityToken + JWT claim 'repository' value 'my-org/wrong-repo' does not match + constraint 'my-org/correct-repo' on role 'github-publisher' + + +``` + +| Condition | Error Code | HTTP Status | +|-----------|-----------|-------------| +| Role URN malformed | `MalformedPolicyDocument` | 400 | +| Role not found | `InvalidParameterValue` | 400 | +| JWT malformed or unparseable | `InvalidIdentityToken` | 400 | +| JWT issuer matches no IdP on Role | `InvalidIdentityToken` | 400 | +| JWT signature verification failed | `InvalidIdentityToken` | 400 | +| JWT expired | `ExpiredTokenException` | 400 | +| JWT `aud` mismatch | `InvalidIdentityToken` | 400 | +| Claim constraints not satisfied | `InvalidIdentityToken` | 400 | +| IdP JWKS endpoint unreachable | `IDPCommunicationError` | 400 | +| `DurationSeconds` exceeds max | `ValidationError` | 400 | + +Error messages include enough detail for callers to diagnose problems (which claim failed, expected vs. actual values). The Role definition is not secret — the account admin created it. + +## Observability + +### STS Exchange Logging + +Every exchange (success or failure) emits a structured log entry: +```json +{ + "event": "sts_exchange", + "timestamp": "2025-03-22T12:00:00Z", + "account_id": "my-org", + "role_name": "github-publisher", + "role_urn": "source::my-org::role/github-publisher", + "idp_issuer": "https://token.actions.githubusercontent.com", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "session_name": "my-ci-job-42", + "result": "success", + "access_key_id": "SCSTS...", + "duration_seconds": 3600, + "client_ip": "...", + "failure_reason": null +} +``` + +### Request-Time Access Logging + +Every S3 request with STS credentials logs: +```json +{ + "event": "s3_request", + "timestamp": "...", + "account_id": "my-org", + "role_name": "github-publisher", + "session_name": "my-ci-job-42", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "action": "PutObject", + "resource": "source::my-org::product/climate-data/2025/data.parquet", + "result": "allow", + "client_ip": "..." +} +``` + +## Migration & Client Tooling + +### Coexistence with Current Auth + +The STS system runs alongside existing permanent API key auth: +- AccessKeyId prefix `SCSTS` routes to STS credential path; all other prefixes route to legacy API key lookup +- No changes to existing API key auth +- No forced migration timeline — STS is additive + +### Anonymous Access + +Public data (`data_mode: open`) remains accessible with zero authentication. No STS exchange, no credentials. `--no-sign-request` keeps working. This is a hard requirement. + +### Client Tooling (v1) + +**GitHub Action: `source-cooperative/configure-credentials`** +```yaml +permissions: + id-token: write +steps: + - uses: source-cooperative/configure-credentials@v1 + with: + role-urn: source::my-org::role/github-publisher + - run: aws s3 cp data.parquet s3://data.source.coop/my-org/my-product/ +``` + +Requests a GitHub OIDC token with audience `https://data.source.coop`, calls the STS endpoint, and exports `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`. All downstream tools pick these up automatically. + +**source-coop CLI (`source-cooperative/source-coop-cli`)** + +The existing CLI already supports `source login` and `credential_process` integration. The STS work extends it so that login calls the new STS endpoint. + +Users configure `~/.aws/config` with role-specific profiles: +```ini +[profile source-read] +credential_process = source-coop creds --role-arn source::my-org::role/reader + +[profile source-write] +credential_process = source-coop creds --role-arn source::my-org::role/publisher +``` + +The `source-coop creds` command: checks for cached valid credentials → if expired, triggers OIDC login (or uses cached auth.source.coop token) → calls STS endpoint → returns credentials in `credential_process` JSON format. + +Users select the profile per tool: +```bash +aws s3 ls s3://data.source.coop/ --profile source-read +``` + +**Direct SDK usage** for programmatic integrations: +```python +sts = boto3.client('sts', endpoint_url='https://data.source.coop/.sts') +creds = sts.assume_role_with_web_identity( + RoleArn='source::my-org::role/github-publisher', + WebIdentityToken=token, + RoleSessionName='my-job' +) +``` + +## Role Management API + +``` +POST /api/accounts/{account_id}/idps +GET /api/accounts/{account_id}/idps +DELETE /api/accounts/{account_id}/idps/{idp_id} + +POST /api/accounts/{account_id}/roles +GET /api/accounts/{account_id}/roles +GET /api/accounts/{account_id}/roles/{role_name} +PUT /api/accounts/{account_id}/roles/{role_name} +DELETE /api/accounts/{account_id}/roles/{role_name} +``` + +Only account owners and org admins can manage IdPs and Roles. The `_default` Role cannot be deleted. From ea45687f940ef96edb34b82e872c5a72b7e31f56 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sun, 22 Mar 2026 18:08:07 -0700 Subject: [PATCH 03/17] docs: update ADRs for account-owned IdPs, user-defined Roles, and STS token exchange ADR-001: Replace embedded SecretAccessKey with HMAC derivation, add ES256 signing, revocation via jti deny-list, and updated SessionToken JWT structure. ADR-004: Rewrite for two-tier IdP model (platform + account-registered), user-defined Roles with claim constraints and permission statements, AWS STS-compatible request/response format, and removal of SC Credential Tokens. ADR-005: Replace fixed 3-role model with user-defined Roles as permission ceiling. Resolve grant schema with concrete permission statement format (read/write actions, URN resource patterns with prefix scoping). Update authorization flow to use Role ceiling from SessionToken intersected with dynamic account permissions. RFC-001: Update sections 4, 7, 8, 13, and 14 to reflect new design. Mark open question 7 (grant schema) as resolved. Add new open questions for org permission model, HMAC secret rotation, and multipart upload credential expiry. Co-Authored-By: Claude Opus 4.6 --- adrs/001-s3-credentials.md | 95 +++++--- adrs/004-sts.md | 440 +++++++++++++++++++++++++++++-------- adrs/005-authorization.md | 185 +++++++++++----- adrs/rfc-001.md | 252 +++++++++++---------- 4 files changed, 668 insertions(+), 304 deletions(-) diff --git a/adrs/001-s3-credentials.md b/adrs/001-s3-credentials.md index 5def272..9494f9a 100644 --- a/adrs/001-s3-credentials.md +++ b/adrs/001-s3-credentials.md @@ -1,7 +1,8 @@ # ADR-001: S3 API Compatibility and Temporary-Credentials-Only Credential Model -**Status:** Draft -**Date:** 2026-03-14 +**Status:** Draft +**Date:** 2026-03-14 +**Updated:** 2026-03-22 **RFC:** RFC-001 §4 --- @@ -31,46 +32,82 @@ This is unchanged from the current proxy. S3 API compatibility is a non-negotiab All SigV4 credentials issued by Source Cooperative are temporary session credentials — the same triplet shape that AWS STS issues: ``` -AccessKeyId (e.g. "ASIA...") -SecretAccessKey (short-lived derived key) -SessionToken (signed JWT encoding identity, role, and expiry) +AccessKeyId (e.g. "SCSTS...") +SecretAccessKey (HMAC-derived key) +SessionToken (signed JWT encoding identity, role, permissions, and expiry) ``` -Callers obtain these credentials by exchanging a trusted identity token at the STS endpoint (`POST /.sts/assume-role-with-web-identity`) before making S3 API calls. +Callers obtain these credentials by exchanging a trusted identity token at the STS endpoint (`POST /.sts/assume-role-with-web-identity`) before making S3 API calls. The `AccessKeyId` is prefixed with `SCSTS` to distinguish STS-issued credentials from any legacy permanent keys during the migration period. ### Session Token Design -The `SessionToken` is a stateless signed JWT. Its payload contains: +The `SessionToken` is a signed JWT using ES256 (ECDSA P-256) asymmetric signing. Its payload contains: ```json { - "user_id": "", - "role_id": "", - "access_key_id": "", - "secret_access_key": "", - "exp": "" + "jti": "", + "sub": "source::my-org::role/github-publisher", + "account_id": "my-org", + "role_name": "github-publisher", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "assumed_by_issuer": "https://token.actions.githubusercontent.com", + "session_name": "my-ci-job-42", + "access_key_id": "SCSTS...", + "permissions": [ + {"actions": ["read", "write"], "resources": ["source::my-org::product/climate-data/*"]} + ], + "iat": 1711100000, + "exp": 1711103600, + "aud": "data.source.coop", + "kid": "" } ``` +Key properties of this design: + +- **The `SecretAccessKey` is not in the token.** It is derived deterministically on each request: `SecretAccessKey = HMAC-SHA256(server_secret, AccessKeyId)`. The server reconstructs it by re-deriving from the `AccessKeyId`. This prevents a leaked SessionToken from directly yielding a complete credential set. +- **`assumed_by` and `assumed_by_issuer`** preserve the original IdP subject for audit trails, even though the credentials act on behalf of the account. +- **`permissions`** embed the Role's permission ceiling in the token, avoiding a per-request policy store lookup for Role evaluation. The account's underlying permissions are still resolved dynamically (see ADR-005). +- **`jti`** enables lightweight revocation via a deny-list (see below). +- **`kid`** in the JWT header supports signing key rotation. + +### SigV4 Verification Flow + The proxy verifies incoming SigV4 requests by: 1. Extracting the `AccessKeyId` from the `Authorization` header -2. Looking up the corresponding `SessionToken` — presented as the `X-Amz-Security-Token` header -3. Verifying the JWT signature against the proxy's public key -4. Checking `exp` has not passed -5. Reconstructing the expected SigV4 signature using the `SecretAccessKey` from the token payload and comparing it to the presented signature +2. Detecting the `SCSTS` prefix to identify this as an STS credential +3. Deriving the `SecretAccessKey` via `HMAC-SHA256(server_secret, AccessKeyId)` +4. Verifying the SigV4 signature using the derived secret +5. Extracting and verifying the `SessionToken` JWT from the `X-Amz-Security-Token` header — checking ES256 signature, `exp`, `aud`, and `jti` against the revocation deny-list +6. Proceeding to authorization (see ADR-005) using the token's embedded identity and permissions + +No external database lookup is required to verify a request or reconstruct the signing key. The token and HMAC derivation together are self-contained. + +### Signing Key Management -No external database lookup is required to verify a request. The token is self-contained. +- **Asymmetric signing:** ES256 (ECDSA P-256). The private key is used only for token issuance; the public key is served at a JWKS endpoint for verification. +- **Key storage:** Private key stored in KMS (AWS KMS or equivalent). +- **Key rotation:** The `kid` header in issued JWTs allows multiple active signing keys. During rotation, new tokens are signed with the new key while tokens signed with the old key remain valid until they expire. The old key is retired after one `max_session_duration` interval. +- **HMAC server secret:** A separate symmetric key used for SecretAccessKey derivation. Stored alongside the signing key in KMS. Rotation follows the same pattern — new AccessKeyIds use the new secret; existing sessions continue to work until expiry. -**Permissions are not encoded in the session token.** The token encodes identity and role only. Per-request permission resolution is handled by the authorisation layer (see ADR-005) by consulting the policy store at request time. This is the same model AWS uses: the STS token asserts role membership, and IAM evaluates the role's current policies live on each API call. +### Revocation + +Session tokens support lightweight revocation via a deny-list of `jti` values: + +- Revoked `jti` values are stored in Cloudflare KV (or equivalent) with a TTL matching the token's remaining lifetime. Entries self-clean as tokens expire. +- Every authenticated request checks the deny-list (~1ms KV lookup at edge). +- Account admins can revoke tokens via `POST /.sts/revoke-session-token`. + +This is a targeted mechanism for incident response, not a general session management system. The deny-list stays small because entries expire automatically. ### Accepted Trade-offs -**Tokens cannot be revoked once issued.** A compromised session token remains valid until its `exp`. Short TTLs (15–60 minutes recommended) limit the blast radius. Immediate revocation is out of scope for this iteration. +**HMAC derivation creates a shared secret dependency.** If the `server_secret` leaks, an attacker who also captures a SessionToken could derive the corresponding SecretAccessKey. This risk is bounded: the attacker needs both the server secret and a valid SessionToken (which requires the separate ES256 signing key to forge). The two secrets are independent. -**Callers must perform a token exchange before making S3 API calls.** This is a one-time step per session. All major AWS SDKs handle STS-derived session credentials natively via the credential provider chain. Tooling that accepts `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables works without modification. +**Callers must perform a token exchange before making S3 API calls.** This is a one-time step per session. The existing `source-coop` CLI supports `credential_process` integration, making the exchange transparent for tools that use the AWS credential provider chain. -**Documentation and CLI tooling must minimise the friction of the exchange step.** Users accustomed to copying a static key into a config file will encounter a new workflow. A `source login` CLI command and SDK credential provider helpers are planned to make the exchange step transparent. +**Documentation and CLI tooling must minimise the friction of the exchange step.** Users accustomed to copying a static key into a config file will encounter a new workflow. The `source-coop creds --role-arn ` command and GitHub Action handle this for the primary use cases. --- @@ -81,14 +118,18 @@ No external database lookup is required to verify a request. The token is self-c - No long-lived credentials anywhere in the system. Credentials expire automatically. - Full compatibility with the existing S3 tooling ecosystem — no client changes required. - The session token is stateless and self-verifying — no credential store on the hot path. -- Short-lived credentials limit blast radius of any credential compromise. +- The SecretAccessKey never appears in the SessionToken, limiting the blast radius of token leakage. +- Asymmetric signing (ES256) means verification requires only the public key; the private signing key has a minimal attack surface. +- Revocation is supported via a lightweight deny-list without adding a stateful dependency to the verification hot path. +- Short-lived credentials (15 min to 12 hours) further limit blast radius. - Composable with OIDC workload identity federation (see ADR-004) — the exchange step is the same regardless of the upstream identity source. **Costs / Risks** - Callers must perform a token exchange before first use. This is new friction compared to the current static key model. - The `/.sts` exchange endpoint is on the critical path for session establishment. Its availability affects whether callers can obtain credentials. -- Session tokens cannot be revoked. A credential leaked mid-session remains valid until TTL expiry. +- The HMAC server secret is a high-value target. Its compromise, combined with a captured SessionToken, yields the corresponding SecretAccessKey. +- The revocation deny-list introduces a KV dependency on the request path, though the lookup is fast (~1ms) and the system degrades gracefully if KV is unavailable (tokens remain valid until expiry). - S3 tooling that hardcodes static credential configuration (rather than using the SDK credential provider chain) may require workarounds. --- @@ -97,6 +138,10 @@ No external database lookup is required to verify a request. The token is self-c **Long-lived static credentials (current model)** — rejected. Persistent security liability; does not compose with workload identity federation; difficult to audit or rotate at scale. -**Short-lived credentials with a server-side revocation list** — considered. Would allow immediate invalidation of compromised credentials. Rejected for this iteration: adds a stateful dependency on the hot path of every request, increasing latency and operational complexity. Can be added in a future iteration if the threat model requires it. +**Embedding the SecretAccessKey in the SessionToken** — rejected. A leaked SessionToken would contain a complete, self-sufficient credential set. HMAC derivation separates the signing material from the bearer token, requiring compromise of both the server secret and a token to reconstruct credentials. + +**Server-side session store for SecretAccessKey** — considered. Generating a random SecretAccessKey per session and storing it in KV eliminates the HMAC shared secret risk entirely. Rejected for now: adds a mandatory KV read on every request for credential verification (not just revocation checks). The HMAC approach keeps verification fully stateless. Can be revisited if the threat model changes. + +**Symmetric signing (HS256)** — rejected. Would require the signing secret to be available on all verification endpoints, expanding the attack surface. ES256 limits the private key to the issuance path only. -**Custom non-S3 protocol** — rejected. Would require Source-specific client libraries and break compatibility with the entire existing ecosystem of data tooling. \ No newline at end of file +**Custom non-S3 protocol** — rejected. Would require Source-specific client libraries and break compatibility with the entire existing ecosystem of data tooling. diff --git a/adrs/004-sts.md b/adrs/004-sts.md index 51c43a1..a0ae088 100644 --- a/adrs/004-sts.md +++ b/adrs/004-sts.md @@ -1,8 +1,9 @@ -# ADR-004: Inbound Authentication — OIDC Federation, STS Exchange, and SC Credential Tokens +# ADR-004: Inbound Authentication — OIDC Federation, Account-Owned Identity Providers, and Role-Based STS Exchange -**Status:** Draft -**Date:** 2026-03-14 -**RFC:** RFC-001 §7 +**Status:** Draft +**Date:** 2026-03-14 +**Updated:** 2026-03-22 +**RFC:** RFC-001 §7 **Depends on:** ADR-001 --- @@ -14,163 +15,418 @@ Source Cooperative's data proxy must authenticate a wide range of callers: - **CI/CD pipelines** (GitHub Actions, GitLab CI, Azure DevOps, Terraform Cloud) running data ingestion or validation jobs - **Managed compute environments** (Databricks, AWS Lambda, GCP Cloud Run) running data processing workflows - **Interactive developers** working locally via notebooks, terminals, or the Source Cooperative CLI -- **Unattended scripts and third-party tools** that cannot interactively obtain a session token - **The Source Cooperative web application** (Next.js on Vercel) making requests on behalf of authenticated users and anonymous visitors -ADR-001 establishes that all callers must obtain short-lived SigV4 session credentials before making S3 API calls. This ADR defines how those credentials are obtained — the design of the STS exchange endpoint and the supported identity sources. +ADR-001 establishes that all callers must obtain short-lived SigV4 session credentials before making S3 API calls. This ADR defines how those credentials are obtained — the design of the STS exchange endpoint, the identity provider model, and the Role-based access scoping. The industry standard for secretless workload authentication is OIDC workload identity federation: a workload presents a signed JWT issued by its platform's OIDC provider to a Security Token Service, which validates it and returns short-lived scoped credentials. AWS, GCP, Azure, GitHub Actions, GitLab CI, Vercel, and many others all support this pattern. -The key insight is that any platform with a publicly reachable JWKS endpoint can be a trusted issuer. Trust is established by configuration (register the issuer URL and claim conditions), not by code. This means the STS is open to the entire OIDC ecosystem without modification. +The key design choice in this ADR is that **accounts own their Identity Providers and Roles**. Rather than a fixed set of platform-managed issuers and a small set of static roles, each account (Individual or Organization) can register IdPs and create Roles that scope access to their resources. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. --- ## Decision -### STS Exchange Endpoint +### Identity Providers (IdPs) -Both deployment targets (Cloudflare Workers and regional ECS) host an STS exchange endpoint at: +IdPs exist at two tiers: +#### Platform IdPs + +Pre-configured by Source Cooperative operators. Immutable by users. These represent well-known OIDC issuers relevant to data engineering workflows: + +| Platform | Issuer URL | Key Claims for Constraints | +| -------------- | --------------------------------------------- | ------------------------------------------------------- | +| Source Cooperative Auth | `auth.source.coop` | `sub`, `groups` | +| GitHub Actions | `https://token.actions.githubusercontent.com` | `repository`, `repository_owner`, `ref`, `environment`, `job_workflow_ref` | +| GitLab CI/CD | `https://gitlab.com` | `project_path`, `ref_type`, `environment` | +| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | +| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | +| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | + +Each platform IdP defines: + +```json +{ + "id": "github-actions", + "issuer_url": "https://token.actions.githubusercontent.com", + "display_name": "GitHub Actions", + "well_known_claims": ["repository", "repository_owner", "ref", "environment", "job_workflow_ref"], + "audience_hint": "https://data.source.coop" +} ``` -POST /.sts/assume-role-with-web-identity -``` -Request body (form-encoded or JSON): +The `well_known_claims` field provides documentation and UI hints — when a user creates a Role binding for this IdP, the UI can suggest these claims for constraints. The `audience_hint` is the recommended `aud` value callers should configure when requesting OIDC tokens. + +This list is illustrative, not exhaustive. Platform operators can add new issuers over time without code changes. + +#### Account IdPs + +Registered by account owners (Individual or Organization). These support corporate identity systems, self-hosted OIDC providers, or any issuer with a publicly reachable JWKS endpoint. + +**Registration API:** + ``` -identity_token= +POST /api/accounts/{account_id}/idps +{ + "display_name": "Our Corporate Okta", + "issuer_url": "https://corp.okta.com/oauth2/default" +} +``` + +**Validation at registration time:** + +1. `issuer_url` must be HTTPS +2. Must not match any platform IdP issuer URL (exact match after canonicalization — prevents impersonation of well-known issuers) +3. Must not duplicate another IdP already registered on the same account +4. Issuer URL canonicalized before storage: lowercase scheme and host, strip trailing slash +5. Fetch `{issuer_url}/.well-known/openid-configuration` — must return a valid OIDC discovery document +6. Resolved IP must not be private, loopback, or link-local (SSRF protection) +7. Fetch timeout: 3 seconds, response body limit: 256KB + +**Stored record:** + +```json +{ + "id": "uuid", + "account_id": "my-org", + "issuer_url": "https://corp.okta.com/oauth2/default", + "display_name": "Our Corporate Okta", + "created_at": "2025-03-22T...", + "created_by": "user-id" +} ``` -Response (on success): +No JWKS is stored at registration time. JWKS is fetched and cached at STS exchange time from the OIDC discovery document's `jwks_uri`. + +**Deletion:** Deleting an IdP is blocked if any Role references it. The account must first remove the IdP binding from all Roles, then delete the IdP. + +### Roles + +Roles belong to an account (Individual or Organization) and define two things: **who can assume the Role** (identity constraints) and **what the Role's credentials can access** (permission statements). + +Roles are identified by URN: `source::{account_id}::role/{role_name}` + +#### Role Schema + ```json { - "AccessKeyId": "...", - "SecretAccessKey": "...", - "SessionToken": "...", - "Expiration": "" + "name": "github-publisher", + "display_name": "GitHub CI Publisher", + "max_session_duration": 3600, + "identity_constraints": [ + { + "idp": "github-actions", + "audience": "https://data.source.coop", + "claim_constraints": [ + {"claim": "repository", "operator": "equals", "value": "my-org/my-repo"}, + {"claim": "ref", "operator": "starts_with", "value": "refs/heads/"} + ] + }, + { + "idp": "uuid-of-account-idp", + "audience": "https://data.source.coop", + "claim_constraints": [ + {"claim": "sub", "operator": "equals", "value": "service-account-42"} + ] + } + ], + "permissions": [ + { + "actions": ["read", "write"], + "resources": ["source::my-org::product/climate-data/*"] + }, + { + "actions": ["read"], + "resources": ["source::my-org::product/reference-data/*"] + } + ] } ``` -The STS exchange flow: -1. Extract the `iss` claim from the presented JWT (without verifying signature yet) -2. Look up the registered issuer configuration for that `iss` value -3. Fetch the issuer's JWKS (from cache if available) and verify the JWT signature -4. Evaluate any registered claim conditions for this issuer (e.g. `repository == "source-cooperative/data-pipeline"`) -5. Map the verified identity to an internal `role_id` and `user_id` -6. Issue and return a signed session token (see ADR-001 for token structure) +A Role acts as a **ceiling** on the account's existing permissions. The credentials issued for a Role can never exceed what the account itself has access to. At request time, the effective permission is the intersection of the Role's permission statements and the account's actual permissions from the policy store (see ADR-005). + +#### Identity Constraints + +Each Role specifies one or more IdP bindings. Each binding identifies an IdP (platform or account-registered) and a set of claim constraints that the presented JWT must satisfy. + +**Claim constraint operators** (deliberately minimal): + +| Operator | Behaviour | Example | +|----------|-----------|---------| +| `equals` | Exact string match | `repository` equals `my-org/my-repo` | +| `starts_with` | String prefix match | `ref` starts_with `refs/heads/` | +| `glob` | Wildcard: `*` (any chars), `?` (single char) | `repository` glob `my-org/*` | + +Rules: +- All claim values are coerced to strings before comparison. JWT claims that are arrays or objects evaluate to false. +- All constraints within a single IdP binding are ANDed — every constraint must match. +- Multiple IdP bindings on a Role are ORed — any one binding can match. +- A missing claim evaluates to false (fail-closed). +- Only top-level claims are supported — no nested path traversal. +- No regex. Glob is the most expressive operator. This avoids ReDoS and keeps the constraint language auditable. + +#### Permission Statements + +Permission statements define what the Role's credentials can access. Resource patterns use this format: + +``` +* → all resources (unlimited ceiling) +source::{account_id}::product/* → all of an account's products +source::{account_id}::product/{product_name} → entire product +source::{account_id}::product/{product_name}/* → entire product (equivalent) +source::{account_id}::product/{product_name}/{prefix}/* → prefix-scoped +source::{account_id}::product/{product_name}/{key} → single object +``` + +Rules: +- Resource patterns can reference any account's products. A Role can delegate access to products the account has access to, even if owned by another account or org. The request-time intersection enforces the real boundary. +- `*` as the entire resource value means "all resources" — no ceiling. The account's actual permissions are the sole constraint. +- Actions are `read` and `write`. `read` maps to `GetObject`, `HeadObject`, `ListObjects`. `write` maps to `PutObject`, `DeleteObject`, and multipart operations. +- Permission statements are additive (allow-only). No explicit denies. + +#### Built-in Default Role + +Every account has a built-in Role: `source::{account_id}::role/_default` + +- Cannot be deleted +- Constrained to only the `auth.source.coop` platform IdP +- Permissions: `{"actions": ["read", "write"], "resources": ["*"]}` — unlimited ceiling; the account's actual permissions are the sole constraint +- Account owners can add claim constraints to the `auth.source.coop` binding but cannot change the IdP binding itself + +This Role serves interactive users who authenticate via `auth.source.coop` and want full access to everything their account can reach. + +#### Role Validation at Creation + +1. `name` must match `[a-z0-9][a-z0-9-]{0,62}` (lowercase, hyphens, max 63 chars) +2. Each IdP reference must exist (platform IdP by well-known ID, account IdP by UUID) +3. `max_session_duration` between 900 and 43200 seconds (15 min to 12 hours) +4. At least one identity constraint required +5. At least one permission statement required +6. Maximum 10 IdP bindings per Role, 20 claim constraints per binding, 50 permission statements per Role + +#### Role Management API + +``` +POST /api/accounts/{account_id}/roles +GET /api/accounts/{account_id}/roles +GET /api/accounts/{account_id}/roles/{role_name} +PUT /api/accounts/{account_id}/roles/{role_name} +DELETE /api/accounts/{account_id}/roles/{role_name} +``` + +Only account owners and org admins can manage Roles. + +### STS Exchange Endpoint + +``` +POST /.sts/assume-role-with-web-identity +``` + +Dot-prefixed account names (`.sts`, etc.) are reserved as invalid, preventing routing conflicts with S3 API paths. + +**Request format:** `application/x-www-form-urlencoded`, AWS STS-compatible: + +``` +Action=AssumeRoleWithWebIdentity +&WebIdentityToken= +&RoleArn=source::my-org::role/github-publisher +&RoleSessionName=my-ci-job-42 +&DurationSeconds=3600 +``` + +**Response format:** XML, AWS STS-compatible: + +```xml + + + + SCSTS... + derived-secret + eyJ... + 2025-03-22T13:00:00Z + + + source::my-org::role/github-publisher + SCSTS...:my-ci-job-42 + + + +``` + +This format enables `boto3.client('sts', endpoint_url='https://data.source.coop/.sts').assume_role_with_web_identity(...)`. The AWS SDK passes the `RoleArn` string through without client-side ARN validation, so Source Cooperative URN format works. + +### STS Exchange Flow + +1. Parse `RoleArn` → extract `account_id` and `role_name` +2. Load Role definition from policy store (cached, 30–60s TTL) +3. Extract `iss` from JWT (without verification) +4. Match `iss` against the Role's allowed IdPs — reject immediately if no match +5. Fetch JWKS from the matched IdP (cached, 1hr TTL, 3s timeout, stale-while-revalidate on fetch failure) +6. Verify JWT signature, `exp`, `nbf` (60s clock skew tolerance), and `aud` +7. Evaluate claim constraints for the matched IdP binding +8. Validate `DurationSeconds` ≤ Role's `max_session_duration` +9. Generate credentials (see ADR-001 for token structure) and return response -Trust is established per-issuer by configuration: -- Issuer URL (must match `iss` claim) -- JWKS endpoint URL (defaults to `/.well-known/jwks.json` per OIDC discovery) -- Claim conditions (optional; constrain which tokens from this issuer are accepted) -- Role mapping (which `role_id` to assign to matching tokens) +The Role URN is required. The caller must know which Role to assume — the system does not scan Roles to find a match. This keeps the lookup path O(1) and deterministic, and mirrors AWS's `AssumeRoleWithWebIdentity` which requires a Role ARN. -**No code changes are required to add a new trusted issuer.** It is a configuration operation. +### JWKS Caching -### Trusted Issuer Registry — Well-Known Issuers +- Cache key: canonicalized issuer URL +- TTL: 1 hour +- Stale-while-revalidate: if the JWKS fetch fails and a cached copy exists, serve stale for up to 24 hours (with warning logged) +- If no cache and fetch fails → return `IDPCommunicationError` +- Max response body: 256KB -The following issuers are relevant to data engineering workflows and are the primary documentation and support targets. Any issuer with a publicly reachable JWKS endpoint can be registered. +### Error Responses -**CI/CD Platforms** +Errors use AWS STS XML format for SDK compatibility: -| Platform | Issuer URL | Key Claims for Conditions | -| -------------- | --------------------------------------------- | ------------------------------------------------------ | -| GitHub Actions | `https://token.actions.githubusercontent.com` | `repository`, `ref`, `environment`, `job_workflow_ref` | -| GitLab CI/CD | `https://gitlab.com/` | `project_path`, `ref_type`, `environment` | -| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | -| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | -| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | +```xml + + + InvalidIdentityToken + JWT claim 'repository' value 'my-org/wrong-repo' does not match + constraint 'my-org/correct-repo' on role 'github-publisher' + + +``` -**Managed Compute and Data Platforms** +| Condition | Error Code | HTTP Status | +|-----------|-----------|-------------| +| Role URN malformed | `MalformedPolicyDocument` | 400 | +| Role not found | `InvalidParameterValue` | 400 | +| JWT malformed or unparseable | `InvalidIdentityToken` | 400 | +| JWT issuer matches no IdP on Role | `InvalidIdentityToken` | 400 | +| JWT signature verification failed | `InvalidIdentityToken` | 400 | +| JWT expired | `ExpiredTokenException` | 400 | +| JWT `aud` mismatch | `InvalidIdentityToken` | 400 | +| Claim constraints not satisfied | `InvalidIdentityToken` | 400 | +| IdP JWKS endpoint unreachable | `IDPCommunicationError` | 400 | +| `DurationSeconds` exceeds max | `ValidationError` | 400 | -| Platform | Token Source | Notes | -| --------------------------- | ------------------------------------ | ----------------------------------------------- | -| AWS (EC2, ECS, Lambda) | Instance/task metadata service | IAM role identity token | -| GCP (Cloud Run, GKE) | Metadata server | Service account identity token, audience-scoped | -| Azure (AKS, Container Apps) | Managed Identity / Entra ID | Federated credential | -| Databricks Jobs | Platform SDK / environment injection | Workspace, job, cluster identity | +Error messages include enough detail for callers to diagnose problems. The Role definition is not secret — the account admin created it. -**Claim Conditions** +### Observability -Claim conditions prevent overly broad trust grants. For example, a GitHub Actions issuer registration should always include at minimum a `repository` claim condition to prevent any GitHub Actions workflow from obtaining Source Cooperative credentials. Recommended minimum conditions per issuer should be documented. +Every STS exchange (success or failure) emits a structured log entry: + +```json +{ + "event": "sts_exchange", + "timestamp": "2025-03-22T12:00:00Z", + "account_id": "my-org", + "role_name": "github-publisher", + "role_urn": "source::my-org::role/github-publisher", + "idp_issuer": "https://token.actions.githubusercontent.com", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "session_name": "my-ci-job-42", + "result": "success", + "access_key_id": "SCSTS...", + "duration_seconds": 3600, + "client_ip": "...", + "failure_reason": null +} +``` + +Every S3 request with STS credentials logs the `account_id`, `role_name`, `session_name`, `assumed_by`, action, resource, and result. ### Identity Source: Source Cooperative Auth (`auth.source.coop`) -The Source Cooperative auth system is Ory-based and issues standard OIDC access tokens. `auth.source.coop` is registered as a trusted issuer with the following mapping: -- Verified token → `role_id: "authenticated_user"`, `user_id` extracted from `sub` claim -- Admin users: identified by an Ory group membership claim → `role_id: "admin"` +The Source Cooperative auth system is Ory-based and issues standard OIDC tokens. `auth.source.coop` is configured as a platform IdP. Interactive users authenticate via this IdP and assume their account's `_default` Role, which passes through all of the account's permissions. This path is appropriate for: - Interactive local development (developer exchanges a browser session token) - CLI `source login` flow (browser device flow → Ory token → STS exchange) - Next.js client-side exchange for authenticated web users -### Identity Source: SC Credential Tokens +### Next.js and Front-End Authentication + +**Authenticated users:** The browser holds the Ory session token. The client exchanges it with `/.sts` using the user's `_default` Role to obtain short-lived session credentials. S3 API calls are made directly from the browser to the proxy. -For unattended workflows that lack an ambient OIDC token, users may generate a **Source Cooperative Credential Token** from the platform UI or API. +**Anonymous visitors:** The Next.js server uses its Vercel OIDC token to exchange for credentials via a platform-defined Role scoped to read-only access on public products. All anonymous traffic flows through the proxy's full middleware stack. -SC Credential Tokens are: -- Signed JWTs issued by Source Cooperative using the same private key infrastructure as the outbound OIDC tokens (see ADR-006) -- Validated via the proxy's published JWKS — Source Cooperative is registered as a trusted issuer of its own credential tokens -- Scoped at mint time: the JWT encodes `user_id`, permitted datasets/collections, and the `role_id` to assume -- Subject to a mandatory maximum TTL enforced at exchange time (recommended: 90 days) -- **Not revocable** — consistent with the stateless session token design in ADR-001 +**Admin users:** Same flow as authenticated users. Admin capabilities are determined by account permissions in the policy store, not by a special role type. -SC Credential Tokens flow through the same `/.sts` exchange endpoint as all other issuers. No special code path. +### CLI and SDK Support -The user stores the SC Credential Token in their workflow's secret manager (GitHub Actions secrets, a CI environment variable, etc.). At runtime, the workflow presents it to `/.sts` and receives short-lived SigV4 session credentials. The stored token is long-lived; the operational credentials are always short-lived. +**`source-coop` CLI:** -**Comparison of identity sources:** +The existing `source-coop-cli` handles authentication and credential management. Users configure `~/.aws/config` with role-specific profiles: -| Source | Suited for | Stored secret? | SigV4 credential TTL | -| --------------------------------------- | ----------------------------------------- | ------------------ | -------------------- | -| Ambient OIDC (GitHub, Databricks, etc.) | CI/CD, managed compute | No | 15–60 min | -| `auth.source.coop` / Ory | Interactive local dev, CLI | No (session-based) | 15–60 min | -| SC Credential Token | Unattended pipelines without ambient OIDC | Yes (token, ≤90d) | 15–60 min | +```ini +[profile source-read] +credential_process = source-coop creds --role-arn source::my-org::role/reader -### Next.js and Front-End Authentication +[profile source-write] +credential_process = source-coop creds --role-arn source::my-org::role/publisher +``` -**Authenticated users:** The browser holds the Ory session token. The client exchanges it directly with `/.sts` (client-side) to obtain SigV4 session credentials. S3 API calls are made directly from the browser to the proxy. Access is recorded in proxy metrics under the user's own identity. +The `source-coop creds` command checks for cached valid credentials, triggers OIDC login if needed, calls the STS endpoint, and returns credentials in `credential_process` JSON format. -**Anonymous visitors:** The Next.js server uses its Vercel OIDC token (`VERCEL_OIDC_TOKEN`, available in Vercel server-side contexts) to exchange for a `public-read`-scoped `anonymous` role credential. This credential is used for requests on behalf of anonymous visitors. All anonymous traffic flows through the proxy's full middleware stack — rate limiting and access metrics apply. +**GitHub Action: `source-cooperative/configure-credentials`** -**Admin users:** Same flow as authenticated users. Admin role is granted at STS exchange time based on an Ory group membership claim. +```yaml +permissions: + id-token: write +steps: + - uses: source-cooperative/configure-credentials@v1 + with: + role-urn: source::my-org::role/github-publisher + - run: aws s3 cp data.parquet s3://data.source.coop/my-org/my-product/ +``` -### CLI and SDK Support +Requests a GitHub OIDC token with audience `https://data.source.coop`, calls the STS endpoint, and exports `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`. All downstream tools pick these up automatically. -**`source` CLI:** -- `source login` — browser-based device flow via `auth.source.coop`, exchanges resulting Ory token at `/.sts`, writes SigV4 session credentials to `~/.aws/credentials` or exports as environment variables -- `source credentials export` — re-exchanges and outputs credentials in a specified format +**Direct SDK usage:** -**Python SDK:** -- A credential provider compatible with `boto3`'s credential provider chain and `fsspec`/`s3fs` -- Wraps the `/.sts` exchange transparently; handles token refresh before expiry +```python +sts = boto3.client('sts', endpoint_url='https://data.source.coop/.sts') +creds = sts.assume_role_with_web_identity( + RoleArn='source::my-org::role/github-publisher', + WebIdentityToken=token, + RoleSessionName='my-job' +) +``` --- ## Consequences **Benefits** -- No stored secrets for CI/CD and managed compute workflows that have ambient OIDC tokens -- Open to any OIDC-compliant issuer without code changes -- All credential paths converge on the same `/.sts` endpoint — one validation and issuance code path -- SC Credential Tokens reuse the outbound OIDC key infrastructure — no separate signing system -- Short-lived operational credentials regardless of how the caller authenticated + +- Accounts own their IdPs and Roles. No operator intervention required to add a new identity source or create new access scopes. +- Open to any OIDC-compliant issuer without code changes — both platform-provided and account-registered. +- All credential paths converge on the same `/.sts` endpoint — one validation and issuance code path. +- The Role-as-ceiling model prevents privilege escalation: a Role can never grant more access than the account itself has. +- AWS STS-compatible request/response format enables use of existing AWS SDKs with a custom endpoint URL. +- Claim constraints provide fine-grained control over which workloads can assume a Role (e.g., only GitHub Actions from a specific repo on the main branch). +- No stored secrets for CI/CD and managed compute workflows that have ambient OIDC tokens. +- Short-lived operational credentials regardless of how the caller authenticated. +- Detailed, actionable error messages aid debugging without leaking sensitive information. **Costs / Risks** + - The `/.sts` endpoint is on the critical path for session establishment. Its availability and latency directly affect all new sessions. - JWKS fetching and caching must be robust — a stale or unavailable JWKS causes all exchange attempts for that issuer to fail. -- Claim conditions must be carefully configured per issuer. Misconfigured conditions (too broad) are a privilege escalation vector. -- SC Credential Tokens reintroduce stored secrets for the unattended pipeline use case. Their 90-day TTL and narrow scope partially mitigate this; they cannot be revoked if compromised before expiry. -- Adding a new issuer requires operator access to register it. There is no self-service issuer registration (by design — unreviewed issuers are a security risk). +- Account-registered IdPs introduce an SSRF risk via the JWKS fetch. Mitigated by URL validation at registration time (HTTPS only, no private IPs, valid OIDC discovery document). +- Account IdP registration means account owners control their own trust anchors. A malicious or compromised account IdP can forge arbitrary JWT claims — this is safe because the Role ceiling model prevents escalation beyond the account's own permissions, but it means audit trails for account-registered IdPs reflect self-asserted identity. +- The claim constraint language is deliberately minimal (equals, starts_with, glob). Complex matching requirements (e.g., numeric comparisons, set membership) are not supported. This can be extended later if needed. +- Platform IdP issuer collision must be enforced at registration time — an account must not be able to register a custom IdP whose issuer URL matches a platform IdP. +- Adding a new platform IdP requires operator access. Self-service IdP registration covers the account-level case; platform-level additions are a governance decision. --- ## Alternatives Considered -**Fixed allowlist of supported OIDC issuers (code-driven)** — rejected. Would require code changes to add new platforms, limiting adoption by teams using less common tooling. Configuration-driven trust is strictly more flexible. +**Fixed platform-managed issuer registry (no account IdPs)** — rejected. Would require operator intervention for every custom identity source. Accounts using corporate Okta, self-hosted Keycloak, or other non-standard IdPs would have no path to OIDC federation without platform support. + +**Fixed role set (`anonymous`, `authenticated_user`, `admin`)** — rejected. Insufficient for the delegation use case. Accounts need to create Roles scoped to specific resources and constrained to specific IdPs — for example, "GitHub Actions from repo X can write to dataset Y." A fixed role set cannot express this. + +**IdP-first resolution (no Role URN required)** — rejected. Would require scanning all Roles across all accounts to find matches for a given JWT issuer, creating ambiguity when multiple Roles match. The caller-must-specify-role pattern keeps the lookup path O(1) and deterministic. -**Long-lived API keys instead of SC Credential Tokens** — considered. SC Credential Tokens are preferred because they are structured JWTs with embedded scope and expiry, flow through the same validation path as ambient OIDC tokens, and carry a hard expiry enforced at exchange time. A raw API key would require a separate validation code path and a database lookup on every exchange. +**Claim constraints on IdP registration (not on Role)** — rejected. Claim constraints belong on the Role because different Roles for the same account may need different constraints for the same IdP. For example, one Role for GitHub Actions might constrain to `refs/heads/main` while another allows any branch. -**RFC 8693 token exchange ("act-as") for Next.js server-side requests** — considered for the case where Next.js makes S3 requests on behalf of a specific authenticated user server-side. Rejected for initial implementation: passing the user's Ory token through client-side is simpler and achieves the same access metrics goal. RFC 8693 is worth revisiting if the audit log needs to distinguish "user accessed directly" from "Next.js accessed on user's behalf." +**SC Credential Tokens (long-lived, up to 90 days)** — not included. For workflows that need persistent access without ambient OIDC, the established pattern is direct access to the underlying storage bucket. The STS design focuses on short-lived, federated credentials. -**Session token revocation list** — considered. Would allow immediate invalidation of SC Credential Tokens and session tokens. Rejected for this iteration: adds a stateful dependency to the hot path of every request. Can be added later if the threat model requires it. +**RFC 8693 token exchange ("act-as") for Next.js server-side requests** — considered. Rejected for initial implementation: passing the user's Ory token through client-side is simpler and achieves the same access metrics goal. Can be revisited if audit logs need to distinguish "user accessed directly" from "Next.js accessed on user's behalf." diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md index e0fa7d1..2ad2cc3 100644 --- a/adrs/005-authorization.md +++ b/adrs/005-authorization.md @@ -1,7 +1,8 @@ -# ADR-005: Authorization Model — Dynamic Per-Request Policy Resolution +# ADR-005: Authorization Model — Role Ceiling with Dynamic Account Permission Resolution -**Status:** Pending +**Status:** Draft **Date:** 2026-03-14 +**Updated:** 2026-03-22 **RFC:** RFC-001 §8 **Depends on:** ADR-001, ADR-004 @@ -9,15 +10,15 @@ ## Context -ADR-001 establishes that session tokens are stateless JWTs encoding identity and role, but **not** permissions. This ADR defines how permissions are resolved at request time. +ADR-001 establishes that session tokens are stateless JWTs. ADR-004 introduces account-owned Roles with embedded permission statements that define a ceiling on what the Role's credentials can access. This ADR defines how permissions are resolved at request time. Two properties drive the design: -1. **Permissions are dynamic.** A user who creates a new organisation or dataset should be able to access it immediately. Encoding permissions in the session token would freeze them at exchange time, requiring re-exchange to reflect changes. +1. **The Role is a ceiling; account permissions are the grants.** The Role's permission statements (embedded in the SessionToken at exchange time) answer "what is the maximum scope of access for these credentials?" The per-account permission lookup answers "what can this account actually access?" The proxy enforces the intersection. A Role can narrow access but never widen it beyond what the account has. -2. **The role is a ceiling; user permissions are the grants.** The role answers "what classes of action are permitted for this identity type?" The per-user permission lookup answers "which specific resources can this identity access?" The proxy enforces the intersection. +2. **Account permissions are dynamic.** A user who joins an organisation or receives a grant on a new dataset should see that change reflected immediately. Because account permissions are resolved per-request from the policy store (not frozen in the token), changes propagate within the cache TTL. -This mirrors AWS IAM: a session token asserts role membership, and the role's current policies are evaluated live on each API call. +This mirrors AWS IAM: the session token asserts role membership with embedded permission boundaries, and the role's current policies are evaluated live on each API call. --- @@ -25,76 +26,137 @@ This mirrors AWS IAM: a session token asserts role membership, and the role's cu ### Identity Model -The session token carries three fields relevant to authorization: +The SessionToken (see ADR-001) carries these fields relevant to authorization: -- `user_id` — stable identifier for the authenticated principal -- `role_id` — one of: `anonymous`, `authenticated_user`, `admin` +- `account_id` — the account whose permissions form the base grants +- `role_name` — identifies the Role (for logging and ceiling lookup) +- `permissions` — the Role's permission statements, embedded at exchange time (the ceiling) +- `assumed_by` — the original IdP subject (for audit, not authorization) - `exp` — token expiry; checked before any policy evaluation -### Role Definitions +### How Roles Replace the Fixed Role Set -**`anonymous`** -- Permitted action classes: read-only (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`) -- Role-level filter: only buckets flagged `public = true` are visible and accessible -- No user permission lookup — role filter is the only guard +The previous design used three fixed roles: `anonymous`, `authenticated_user`, and `admin`. These are replaced by user-defined Roles (see ADR-004). The equivalent behaviour is achieved through Role configuration: -**`authenticated_user`** -- Permitted action classes: read and write (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`, `ListObjects`, `ListBuckets`, `CreateBucket`) -- Role-level filter: none — user permission lookup determines which resources are accessible -- User permission lookup is always performed for non-public resources +**Anonymous access** does not use a Role at all. Requests without credentials are treated as anonymous. Anonymous callers can only read public products — no Role lookup, no account permission lookup. -**`admin`** -- Permitted action classes: all -- Role-level filter: none -- User permission lookup: **skipped** — admin role has unconditional access to all resources -- Admin role assumption is gated on strong identity claims (e.g. Ory group membership) to prevent accidental privilege escalation +**Authenticated user access** uses the built-in `_default` Role, which has an unlimited ceiling (`"resources": ["*"]`). The account's actual permissions are the sole constraint. This is equivalent to the previous `authenticated_user` role. -### Per-Request Resolution Strategy +**Admin access** is determined by account permissions in the policy store, not by a special role type. An account with admin-level grants simply has broader permissions that the Role ceiling does not restrict (when using the `_default` Role with `*` resources). -Authorization proceeds in at most three steps, with early exits to minimise unnecessary lookups: +**Scoped access** is the new capability. A Role with specific permission statements (e.g., read-only on one product) creates a narrow ceiling. Even if the account has broad permissions, the credentials can only access what the Role allows. -**Step 1 — Role action check** -Does this role permit the requested action class? If the role is `anonymous` and the request is a `PutObject`, deny immediately. This is pure in-memory logic against the role definition — no lookup required. +### Per-Request Authorization -**Step 2 — Public resource early exit** -For read operations only: is the requested bucket flagged `public = true` in the policy store? If yes, and the role permits reads, permit immediately without a user permission lookup. This covers the majority of Source Cooperative traffic (public dataset access) and avoids a per-user lookup for every read of public data. +Authorization proceeds in steps, with early exits to minimise lookups: -**Step 3 — User permission lookup** -For non-public resources or write operations: fetch the user's permissions from the policy store and evaluate them against the requested resource. This reflects current organisation membership, dataset ownership, and explicit grants. +**Step 1 — Identify the caller** + +- **No credentials** → anonymous. Only read actions on public products are permitted. +- **Permanent API key** (non-`SCSTS` prefix) → legacy path. Look up account via Source API. +- **STS credentials** (`SCSTS` prefix) → derive SecretAccessKey via HMAC, verify SigV4, decode SessionToken JWT. + +**Step 2 — Role action check (in-memory, no lookup)** + +For anonymous callers, only read actions are permitted (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`). Deny writes immediately. + +For STS callers, check the SessionToken's embedded `permissions` array. If the requested action (read or write) on the requested resource does not match any permission statement, deny immediately. This is a local check against data already in the token — no network call. + +**Step 3 — Resource resolution** + +Map the S3 request to a Source Cooperative resource: +- Bucket name → `account_id/product_name` +- Object key → path within the product + +**Step 4 — Public resource early exit (cached, 60–300s TTL)** + +For read requests: if the product is public (`data_mode: open`), permit immediately. No further lookups. This is the fast path for the majority of traffic — public open data reads. + +**Step 5 — Account permission lookup (cached, 30–60s TTL)** + +For non-public resources or write operations: +1. Fetch the account's permissions from the policy store (the account referenced in the SessionToken's `account_id`) +2. Compute: `(Role ceiling permissions from token) ∩ (account's actual permissions from policy store)` +3. If the intersection includes the requested action on the requested resource → permit +4. Otherwise → deny + +**Step 6 — Prefix enforcement** + +If the Role's permission statement includes a prefix constraint (e.g., `source::my-org::product/my-dataset/uploads/*`), verify the object key falls within that prefix. This enforcement is part of Step 2 and Step 5 — the prefix is evaluated when matching the resource pattern. + +### Authorization Truth Table + +| Caller | Resource | Account has access? | Role permits? | Result | +|--------|----------|-------------------|--------------|--------| +| Anonymous | Public product | N/A | N/A | **Allow** (read only) | +| Anonymous | Private product | N/A | N/A | **Deny** | +| STS | Public product, read | N/A | Yes | **Allow** | +| STS | Public product, write | Yes | Yes | **Allow** | +| STS | Private product | Yes | Yes | **Allow** | +| STS | Private product | Yes | No (ceiling) | **Deny** | +| STS | Private product | No | Yes | **Deny** | ### Operation-Specific Behaviour **Single-resource operations (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`)** -After the role check and public early exit, a point lookup: does `(user_id, bucket_id)` resolve to an access grant? If the grant includes prefix restrictions, those are enforced against the requested object key. +After the Role ceiling check and public early exit, a point lookup: does the account have an access grant for this product? If the grant includes prefix restrictions, those are enforced against the requested object key. **`ListBuckets`** -The proxy constructs this response entirely from the policy store — the upstream is never called. Anonymous users see `public = true` buckets; authenticated users see all buckets they have grants for; admins see all buckets. +The proxy constructs this response entirely from the policy store — the upstream is never called: +1. Anonymous: return products with `public = true` +2. STS with `_default` Role (unlimited ceiling): return all products the account has grants for +3. STS with scoped Role: return only products that appear in both the Role's permission statements and the account's grants + +**`ListObjects` (within a product)** +After the Role ceiling check, public early exit, and account permission lookup: if the Role's permission statement includes a key prefix restriction, pass it as a filter to the upstream `ListObjects` call. + +### Permission Statement Matching -**`ListObjects` (within a bucket)** -After the role check, public early exit, and user permission lookup: if the grant includes a key prefix restriction, it is passed as a filter to the upstream `ListObjects` call so the upstream enforces the boundary. +When evaluating whether a request matches a Role's permission statements, the proxy checks: + +1. **Action match:** Does the statement's `actions` array include the requested action class (`read` or `write`)? +2. **Resource match:** Does the statement's `resources` array contain a pattern that matches the requested resource? + - `*` matches everything + - `source::{account}::product/{name}` or `source::{account}::product/{name}/*` matches the entire product + - `source::{account}::product/{name}/{prefix}/*` matches objects under the prefix + - `source::{account}::product/{name}/{key}` matches a single object + +If any statement matches both action and resource, the Role permits the request. The account permission lookup then determines whether the account actually has the underlying access. ### Cache Strategy -All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container). +All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container): | Lookup | Cache Key | TTL | |---|---|---| -| Role definition | `role_id` | In-memory constant | -| Bucket public flag | `bucket_id` | 60–300s | -| Single-resource user grant | `(user_id, bucket_id)` | 30–60s | -| User's full bucket list (`ListBuckets`) | `(user_id, role_id)` | 5–10s | +| Product public flag | `product_id` | 60–300s | +| Account permission for product | `(account_id, product_id)` | 30–60s | +| Account's full product list (`ListBuckets`) | `account_id` | 5–10s | + +The short TTL on the full product list ensures that account permission changes (new grants, org membership) are reflected within seconds. -The short TTL on the full bucket list ensures that a user who creates a new dataset sees the change within seconds. For Workers, cache is per-isolate and not shared across edge nodes; Workers KV is available as a shared tier if needed. +For Workers, cache is per-isolate and not shared across edge nodes. Workers KV is available as a shared tier if needed. -### Unresolved: Grant Schema +### Access Logging -The exact schema of user access grants is unresolved. Open questions include: +Every S3 request with STS credentials emits a structured log entry: -- Whether grants are bucket-level only or support sub-bucket prefix granularity -- Whether grants are additive (allow-only) or support explicit denies -- How organisation membership is modelled — derived grants from membership, or explicit per-bucket grants per member +```json +{ + "event": "s3_request", + "timestamp": "...", + "account_id": "my-org", + "role_name": "github-publisher", + "session_name": "my-ci-job-42", + "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", + "action": "PutObject", + "resource": "source::my-org::product/climate-data/2025/data.parquet", + "result": "allow", + "client_ip": "..." +} +``` -These questions are tracked in RFC-001 Open Question 7. +This provides full auditability: which account, which Role, which original identity, and what they accessed. --- @@ -102,25 +164,32 @@ These questions are tracked in RFC-001 Open Question 7. **Benefits** -- Permissions reflect current state — no re-exchange required after creating a new dataset or joining an organisation -- The majority of traffic (public dataset reads) resolves with no user-specific lookup -- Admin bypass eliminates unnecessary lookups for administrative operations -- Cache TTLs are tuned per-operation to balance freshness and performance -- The model is familiar to anyone who knows AWS IAM +- The Role ceiling is evaluated locally from the SessionToken — no network call required for the first authorization check. +- Account permissions reflect current state — no re-exchange required after creating a new dataset or joining an organisation. +- The majority of traffic (public dataset reads) resolves with no account-specific lookup. +- The permission statement format is concrete and resolved: actions are `read`/`write`, resources use a URN pattern with optional prefix scoping. +- The model supports delegation: a Role can reference products owned by other accounts that the Role's account has access to. +- Audit logs capture both the account identity and the original IdP subject, enabling attribution even though credentials act as the account. +- Anonymous access remains frictionless — no STS exchange, no credentials, just `--no-sign-request`. **Costs / Risks** -- Every non-public authenticated request requires a policy store lookup (mitigated by caching) -- The policy store is on the hot path — its availability affects request latency for cache misses -- Per-isolate caching in Workers means cache is not shared across edge nodes (cold isolate = cache miss) -- Grant schema is unresolved — the implementation cannot begin until the schema is defined +- Every non-public authenticated request requires an account permission lookup from the policy store (mitigated by caching). +- The policy store is on the hot path — its availability affects request latency for cache misses. +- Per-isolate caching in Workers means cache is not shared across edge nodes (cold isolate = cache miss). +- The permission model is additive (allow-only). Explicit denies are not supported in this iteration. If the access control model requires "grant access to everything except X," it must be expressed as individual grants for everything except X. +- The `ListBuckets` response for scoped Roles requires intersecting the Role's resource patterns with the account's grants, which is more complex than simply returning the account's full product list. --- ## Alternatives Considered -**Encode permissions in the session token** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens to see permission changes. Unacceptable for a platform where users create datasets and join organisations dynamically. +**Encode full permissions in the session token** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens to see permission changes. Unacceptable for a platform where users create datasets and join organisations dynamically. The hybrid approach (Role ceiling in token, account permissions dynamic) provides the best of both: the ceiling check is local, and permission changes propagate in near real-time. + +**Fixed role set (`anonymous`, `authenticated_user`, `admin`)** — superseded by user-defined Roles. The `_default` Role with unlimited ceiling achieves the same effect as `authenticated_user`. Admin access is determined by account grants, not a special role type. Scoped Roles provide new capability that the fixed set could not express. **Centralised permission cache (Redis / Workers KV as primary)** — considered. Would share cache across isolates and containers. Rejected as the primary tier: adds a network hop to every cache read. Per-isolate caching with optional Workers KV as a secondary tier is preferred. -**Explicit deny support in grants** — deferred. Additive (allow-only) grants are simpler to reason about and sufficient for the initial use cases. Explicit denies can be added later if the access control model requires it. +**Explicit deny support in grants** — deferred. Additive grants are simpler to reason about and sufficient for the initial use cases. Explicit denies can be added later if the access control model requires it. + +**Separate principal identity for delegated access** — considered. STS credentials would represent a distinct principal (e.g., "github-actions via account/role") rather than acting as the account. Rejected: adds complexity to the permission model (need grants for delegated principals) without clear benefit. The Role ceiling already constrains what the credentials can do. The `assumed_by` field in the SessionToken provides audit trail separation without requiring a separate authorization path. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index f756660..667a9cd 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -4,7 +4,7 @@ **Date:** 2026-03-14 **Authors:** @alukach **Replaces:** Current data proxy (ECS, Rust, long-lived credentials) -**Version:** 3 — see [Changelog](#changelog) for changes from v2 +**Version:** 4 — see [Changelog](#changelog) for changes from v3 --- @@ -133,13 +133,13 @@ This choice is a meaningful friction increase for users accustomed to copying a ### SigV4 Verification and Stateless Session Tokens -Incoming requests carry a SigV4 `Authorization` header. The proxy verifies the signature using the `SecretAccessKey` embedded within the `SessionToken` itself. +Incoming requests carry a SigV4 `Authorization` header. The proxy reconstructs the `SecretAccessKey` by deriving it from the `AccessKeyId` via `HMAC-SHA256(server_secret, AccessKeyId)` — the secret is never embedded in the token itself. -Session tokens are **stateless signed JWTs**. The token payload encodes the minimum information needed to identify the caller and verify the signature: `user_id`, `role_id`, `AccessKeyId`, `SecretAccessKey`, and `exp`. Crucially, the token does **not** encode the caller's full permission set — permissions are dynamic and are resolved per-request from the configuration layer (see §8). This mirrors how AWS STS works: the token asserts role membership, and the role's current policies are evaluated live on each API call. +Session tokens are **stateless signed JWTs** using ES256 (asymmetric). The token payload encodes: `account_id`, `role_name`, `access_key_id`, the Role's permission statements (ceiling), `assumed_by` (original IdP subject for audit), `session_name`, and `exp`. The token encodes the Role's permission ceiling but **not** the account's full permission set — account permissions are dynamic and resolved per-request from the configuration layer (see §8). At request time, the effective permission is the intersection of (Role ceiling from token) and (account's actual permissions from policy store). -The proxy validates the JWT signature against its own public key and enforces the `exp` claim. No external store lookup is required to verify the token itself. The subsequent per-request permission resolution is a separate step, covered in §8. +The proxy validates the JWT signature against its own public key, enforces `exp`, and checks the `jti` claim against a lightweight revocation deny-list (stored in Cloudflare KV with auto-expiring entries). No external database lookup is required to verify the token or reconstruct the signing key. -This design has a deliberate trade-off: **session tokens cannot be revoked once issued.** A compromised token remains valid until its `exp`. Short TTLs (e.g. 15–60 minutes) limit the blast radius. Immediate revocation is explicitly out of scope for this iteration. +Session tokens support **lightweight revocation** via a deny-list of `jti` values. Account admins can revoke tokens via `POST /.sts/revoke-session-token`. The deny-list entries self-clean as tokens expire. --- @@ -209,107 +209,99 @@ We are continuing with Rust as the implementation language for both the Workers ### Design Principle -Source Cooperative will not issue or accept long-lived static credentials. All authentication flows terminate in short-lived SigV4 session credentials obtained through a token exchange. The exchange endpoint at `/.sts` acts as a Security Token Service that accepts JWTs from trusted OIDC identity providers and returns temporary credentials. +Source Cooperative will not issue or accept long-lived static credentials. All authentication flows terminate in short-lived SigV4 session credentials obtained through a token exchange. The exchange endpoint at `/.sts` acts as a Security Token Service that accepts JWTs from trusted OIDC identity providers and returns temporary credentials scoped by a caller-specified Role. + +The key design choice is that **accounts own their Identity Providers and Roles**. Each account (Individual or Organization) can register IdPs and create Roles that scope access to their resources. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. + +### Identity Providers (IdPs) + +IdPs exist at two tiers: + +**Platform IdPs** are pre-configured by Source Cooperative operators — well-known OIDC issuers relevant to data engineering workflows (GitHub Actions, GitLab CI, `auth.source.coop`, Vercel, etc.). Platform operators can add new issuers over time without code changes. + +**Account IdPs** are registered by account owners. These support corporate identity systems (Okta, Entra ID), self-hosted providers (Keycloak), or any issuer with a publicly reachable JWKS endpoint. Account IdP registration validates the issuer URL (HTTPS only, no private IPs, valid OIDC discovery document) and prevents collision with platform IdP issuer URLs. + +### Roles + +Roles belong to an account and define two things: **who can assume the Role** (identity constraints with per-IdP claim matching) and **what the Role's credentials can access** (permission statements). Roles are identified by URN: `source::{account_id}::role/{role_name}`. + +A Role acts as a **ceiling** on the account's existing permissions. The credentials issued for a Role can never exceed what the account itself has access to. At request time, the effective permission is the intersection of the Role's permission statements and the account's actual permissions from the policy store. + +Every account has a built-in `_default` Role constrained to `auth.source.coop` with an unlimited ceiling — it passes through all of the account's permissions for interactive users. ### STS Token Exchange -The STS endpoint accepts a signed JWT from any registered trusted issuer and returns a session credential triplet: +The STS endpoint accepts a signed JWT and a Role URN, validates the JWT against the Role's identity constraints, and returns a session credential triplet: ``` POST /.sts/assume-role-with-web-identity -→ { AccessKeyId, SecretAccessKey, SessionToken, Expiration } +Content-Type: application/x-www-form-urlencoded + +Action=AssumeRoleWithWebIdentity&RoleArn=source::my-org::role/publisher&WebIdentityToken=&RoleSessionName=my-job&DurationSeconds=3600 +→ XML response (AWS STS-compatible): { AccessKeyId, SecretAccessKey, SessionToken, Expiration } ``` -The STS: -1. Resolves the issuer from the JWT `iss` claim -2. Fetches and caches the issuer's JWKS to verify the signature -3. Evaluates claim-based conditions against the registered role mapping -4. Issues a short-lived stateless session token (see §4) containing `user_id`, `role_id`, and `exp` +The request and response formats are AWS STS-compatible, enabling `boto3.client('sts', endpoint_url=...).assume_role_with_web_identity(...)`. + +The STS exchange flow: +1. Parse the Role URN to extract account and role name +2. Load the Role definition (cached) +3. Match the JWT's `iss` claim against the Role's allowed IdPs — reject if no match +4. Fetch and cache the IdP's JWKS, verify the JWT signature, `exp`, `nbf`, and `aud` +5. Evaluate the Role's claim constraints against the JWT claims +6. Issue a short-lived session token (see §4) containing account identity, Role permissions, and audit context -Trust is established per-issuer by configuration: register the issuer URL, JWKS endpoint, and claim conditions. No code changes are required to add a new issuer. +The Role URN is required — the caller must know which Role to assume. This keeps the lookup path O(1) and deterministic. ### Supported Identity Sources -#### Ambient OIDC — CI/CD and Managed Compute +#### Platform IdPs — CI/CD and Managed Compute Workflows running in environments that provide ambient OIDC tokens can exchange them directly. These environments require no stored secrets: -| Platform | OIDC Issuer | Distinguishing Claims | +| Platform | OIDC Issuer | Key Claims for Constraints | | --------------------------- | ---------------------------------------- | ----------------------------------------------- | | GitHub Actions | `token.actions.githubusercontent.com` | `repository`, `ref`, `environment` | -| GitLab CI/CD | `https://gitlab.com/` | `project_path`, `ref_type`, `environment` | +| GitLab CI/CD | `https://gitlab.com` | `project_path`, `ref_type`, `environment` | | Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | | HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | | Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | -| AWS (EC2, ECS, Lambda) | Instance/task metadata service | IAM role identity | -| GCP (Cloud Run, GKE) | Metadata server | Service account, audience-scoped | -| Azure (AKS, Container Apps) | Managed Identity / Entra ID | Federated credential | -| Databricks Jobs | Platform SDK / env injection | Workspace, job, cluster | -This list is illustrative, not exhaustive. Any issuer with a publicly reachable JWKS endpoint can be registered. +This list is illustrative, not exhaustive. Accounts can register additional IdPs for any issuer with a publicly reachable JWKS endpoint. #### Source Cooperative Auth (`auth.source.coop`) -Users authenticated interactively via Source Cooperative's Ory-based auth system may present their Ory access token to the STS exchange endpoint. The STS validates it against Ory's JWKS (`auth.source.coop`) and issues session credentials. This is appropriate for: +Users authenticated interactively via Source Cooperative's Ory-based auth system present their Ory token to `/.sts` with their account's `_default` Role. This is appropriate for interactive local development, ad-hoc data access, and CLI tooling. -- Interactive local development -- Ad-hoc data access from a notebook or terminal -- CLI tooling that performs a browser-based login flow - -It is not suited to unattended pipelines, as it requires an active user session. - -#### Source Cooperative Credential Tokens - -For unattended workflows that lack ambient OIDC tokens — local scripts, third-party orchestrators, environments not listed above — users may generate a **Source Cooperative Credential Token** from the platform UI or API. - -These tokens are: - -- Signed JWTs issued by Source Cooperative, using the same private key infrastructure as our outbound OIDC tokens -- Validated via our published JWKS through the same STS exchange path as all other inbound tokens -- Scoped at mint time: the JWT payload encodes user identity, permitted datasets/collections, and the internal role to assume -- Subject to a mandatory maximum TTL (e.g. 90 days) enforced at exchange time -- **Not revocable** — consistent with the stateless session token model. A compromised credential token remains exchangeable until its TTL expires. Short TTLs and narrow scope limit the blast radius. - -The user stores the token in their workflow's secret manager. At runtime, the workflow exchanges it for short-lived session credentials via the standard `/.sts` endpoint — the same flow used by all other issuers. - -| Mechanism | Suited for | Stored secret? | Credential TTL | -| --------------------------------------- | ---------------------- | ------------------ | -------------- | -| Ambient OIDC (GitHub, Databricks, etc.) | CI/CD, managed compute | No | Minutes | -| `auth.source.coop` / Ory token | Interactive local dev | No (session-based) | Minutes | -| SC Credential Token | Unattended pipelines | Yes (token, ≤90d) | Minutes | +| Mechanism | Suited for | Stored secret? | Credential TTL | +| --------------------------------------- | ---------------------- | ------------------ | ------------------ | +| Ambient OIDC (GitHub, GitLab, etc.) | CI/CD, managed compute | No | 15 min – 12 hours | +| `auth.source.coop` / Ory token | Interactive local dev | No (session-based) | 15 min – 12 hours | +| Account-registered IdP (Okta, etc.) | Corporate workflows | No | 15 min – 12 hours | ### Next.js and Front-End Authentication -The Source Cooperative web application (Next.js, deployed on Vercel) requires access to the proxy for two distinct caller types: authenticated users and anonymous visitors. - -**Authenticated users — client-side STS exchange** - -When a user is logged in, the browser holds an Ory session token from `auth.source.coop`. Rather than routing proxy requests through the Next.js server, the client exchanges the Ory token directly with `/.sts` to obtain short-lived session credentials. All subsequent S3 API calls are made directly from the browser to the proxy using those credentials. - -This approach is preferred because: -- Access is recorded in proxy metrics under the user's own identity, not aggregated under a server-side service account -- The Next.js server does not handle S3 credentials at all — it remains stateless with respect to data access -- The user's current permissions are always reflected (see §8 — permissions are resolved per-request, not embedded in the token) +**Authenticated users:** The browser holds the Ory session token. The client exchanges it with `/.sts` using the user's `_default` Role to obtain short-lived session credentials. S3 API calls are made directly from the browser to the proxy. Access is recorded under the user's own identity. -**Anonymous visitors — Vercel OIDC service identity** +**Anonymous visitors:** The Next.js server uses its Vercel OIDC token to exchange for credentials via a platform-defined Role scoped to read-only access on public products. All anonymous traffic flows through the proxy's full middleware stack. -Anonymous visitors have no Ory token. Since the Next.js application is deployed on Vercel, it has access to a Vercel-issued OIDC token (`VERCEL_OIDC_TOKEN`) in server-side contexts. The Next.js server exchanges this token at `/.sts` for a session credential bound to a `public-read` role. This credential can be embedded in the rendered page or used server-side to proxy requests on behalf of anonymous visitors. +**Admin users:** Same flow as authenticated users. Admin capabilities are determined by account permissions in the policy store, not by a special role type. -The `public-read` role is constrained to read-only operations on buckets flagged as publicly accessible. All anonymous traffic still flows through the proxy's full auth/authz/middleware stack, meaning rate limiting and access metrics apply even to unauthenticated requests. - -**Admin users** +### CLI and SDK Support -Admin users authenticate the same way as standard users (Ory token → STS exchange). Their `user_id` maps to the `admin` role at exchange time. The admin role bypasses resource-level permission checks — see §8. +**`source-coop` CLI:** The existing CLI handles authentication and credential management. Users configure `~/.aws/config` with role-specific profiles: -### CLI and SDK Support +```ini +[profile source-read] +credential_process = source-coop creds --role-arn source::my-org::role/reader -The credential exchange step should be invisible for common workflows. We intend to provide: +[profile source-write] +credential_process = source-coop creds --role-arn source::my-org::role/publisher +``` -- A **Source Cooperative CLI** that supports `source login` (browser-based device flow via `auth.source.coop`) and `source credentials export` (writes STS-derived session credentials to environment or AWS credentials file format, consumable by any S3 SDK or tool) -- An **SDK helper** (initially Python, given the data engineering audience) that implements a credential provider wrapping the STS exchange, compatible with `boto3`'s credential provider chain and `fsspec`/`s3fs` configuration +**GitHub Action:** `source-cooperative/configure-credentials` requests a GitHub OIDC token, calls the STS endpoint, and exports credentials as environment variables. -> [!NOTE] -> **TODO:** Define the CLI command surface and SDK packaging strategy. Determine whether the CLI performs the STS exchange itself or delegates to the AWS CLI credential provider chain. +**Direct SDK usage:** `boto3.client('sts', endpoint_url='https://data.source.coop/.sts').assume_role_with_web_identity(...)` works with Source Cooperative URN format. --- @@ -319,90 +311,82 @@ The credential exchange step should be invisible for common workflows. We intend Two properties drive the authorization design: -1. **Permissions are dynamic.** A user who creates a new organisation or dataset should be able to access it immediately. Permissions cannot be frozen into the session token at exchange time — they must be resolved from the policy store on each request, with short-lived caching to reduce latency. +1. **The Role is a ceiling; account permissions are the grants.** The Role's permission statements (embedded in the SessionToken at exchange time) define the maximum scope of access for these credentials. The per-account permission lookup determines what the account can actually access. The proxy enforces the intersection. A Role can narrow access but never widen it beyond what the account has. -2. **The role is a ceiling; user permissions are the grants.** The role answers "what classes of action are permitted for this identity type?" The per-user permission lookup answers "which specific resources can this identity access?" The proxy enforces the intersection. These are separate lookups with different cache characteristics. +2. **Account permissions are dynamic.** A user who joins an organisation or receives a grant on a new dataset should see that change reflected immediately. Because account permissions are resolved per-request from the policy store (not frozen in the token), changes propagate within the cache TTL. -This mirrors how AWS IAM works: a session token asserts role membership, and the role's current policies are evaluated live on each API call against the IAM policy store. +This mirrors how AWS IAM works: the session token asserts role membership with embedded permission boundaries, and the role's current policies are evaluated live on each API call. ### Identity Model -The session token (§4) carries three fields relevant to authorization: +The session token (§4) carries these fields relevant to authorization: -- `user_id` — stable identifier for the authenticated principal -- `role_id` — one of: `anonymous`, `authenticated_user`, `admin` +- `account_id` — the account whose permissions form the base grants +- `role_name` — identifies the Role (for logging and ceiling lookup) +- `permissions` — the Role's permission statements, embedded at exchange time (the ceiling) +- `assumed_by` — the original IdP subject (for audit, not authorization) - `exp` — token expiry; checked before any policy evaluation -### Role Definitions +### How Roles Replace the Fixed Role Set -**`anonymous`** -- Permitted action classes: read-only (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`) -- Role-level filter: only buckets flagged `public = true` are visible and accessible -- No user permission lookup — role filter is the only guard +**Anonymous access** does not use a Role. Requests without credentials can only read public products. -**`authenticated_user`** -- Permitted action classes: read and write (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`, `ListObjects`, `ListBuckets`, `CreateBucket`) -- Role-level filter: none — user permission lookup determines which resources are accessible -- User permission lookup is always performed for non-public resources +**Authenticated user access** uses the built-in `_default` Role with unlimited ceiling (`"resources": ["*"]`). The account's actual permissions are the sole constraint. -**`admin`** -- Permitted action classes: all -- Role-level filter: none -- User permission lookup: **skipped** — admin role has unconditional access to all resources -- Admin role assumption should be gated on strong identity claims (e.g. a specific Ory group membership claim) to prevent accidental privilege escalation +**Admin access** is determined by account permissions in the policy store, not by a special role type. + +**Scoped access** is the new capability. A Role with specific permission statements creates a narrow ceiling that constrains credentials regardless of the account's broader permissions. ### Per-Request Resolution Strategy -Authorization resolution proceeds in at most three steps, with early exits to minimise unnecessary lookups: +Authorization proceeds in steps, with early exits to minimise lookups: -**Step 1 — Role action check** -Does this role permit the requested action class? If the role is `anonymous` and the request is a `PutObject`, deny immediately. This check is pure in-memory logic against the role definition — no lookup required. +**Step 1 — Identify the caller.** No credentials → anonymous (read-only, public products only). `SCSTS` prefix → STS credential path. Other prefix → legacy API key lookup. -**Step 2 — Public resource early exit** -For read operations only: is the requested bucket flagged `public = true` in the policy store? If yes, and the role permits reads, permit immediately without a user permission lookup. This early exit covers the majority of Source Cooperative traffic (public dataset access) and avoids a per-user lookup for every anonymous or authenticated read of public data. The `public` flag is cached aggressively (60–300 seconds) since it changes rarely. +**Step 2 — Role action check (in-memory).** Check the SessionToken's embedded `permissions` array. If the requested action on the requested resource does not match any permission statement, deny immediately. This is a local check — no network call. -**Step 3 — User permission lookup** -For non-public resources, or write operations: fetch the user's permissions from the policy store and evaluate them against the requested resource. This is the dynamic lookup that reflects current membership in organisations, ownership of datasets, and any explicit grants. +**Step 3 — Public resource early exit (cached, 60–300s TTL).** For read requests on public products, permit immediately. No further lookups. This is the fast path for the majority of traffic. -### Operation-Specific Behaviour +**Step 4 — Account permission lookup (cached, 30–60s TTL).** For non-public resources or write operations: fetch the account's permissions from the policy store. Compute `(Role ceiling) ∩ (account permissions)`. Permit if the intersection covers the requested action and resource; deny otherwise. -**Single-resource operations (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`)** -After the role check and public early exit, a single point lookup is performed: does `(user_id, bucket_id)` resolve to an access grant? If the grant exists and includes any prefix restrictions, those are enforced against the requested object key. Deny if no grant exists. +### Permission Statement Format -**`ListBuckets`** -This operation cannot be delegated to the upstream — the upstream would return all buckets in the cloud account, not the user's logical buckets. The proxy must construct the `ListBuckets` response entirely from the policy store: +Role permission statements use this structure: -1. Role check: does the role permit `ListBuckets`? -2. If `anonymous`: fetch all buckets with `public = true` from the policy store -3. If `authenticated_user`: fetch all bucket IDs the user has an access grant for; apply role filter -4. If `admin`: return all buckets -5. Construct and return the response — no upstream call is made +```json +{ + "actions": ["read", "write"], + "resources": ["source::my-org::product/climate-data/*"] +} +``` -Because `ListBuckets` requires a full enumeration of the user's accessible buckets, it is the most expensive and freshness-sensitive lookup. A short cache TTL (5–10 seconds) is warranted here — this is the operation most likely to reflect a user's recent creation of a new organisation or dataset. +- Actions are `read` (GetObject, HeadObject, ListObjects) and `write` (PutObject, DeleteObject, multipart operations). +- Resources use URN patterns with optional prefix scoping. `*` as the entire resource means "no ceiling." +- Resource patterns can reference any account's products — a Role can delegate access to products the account has access to, even if owned by another account or org. The request-time intersection enforces the real boundary. +- Permission statements are additive (allow-only). No explicit denies. -**`ListObjects` (within a bucket)** -1. Role check: does the role permit `ListObjects`? -2. Public early exit: is the bucket public? -3. User permission lookup: does `(user_id, bucket_id)` have an access grant? -4. If the grant includes a key prefix restriction, pass it as a filter to the upstream `ListObjects` call so the upstream enforces the boundary, rather than filtering the result set after the fact +### Operation-Specific Behaviour -### Cache Strategy +**Single-resource operations (`GetObject`, `PutObject`, `HeadObject`, `DeleteObject`)** +After the Role ceiling check and public early exit, a point lookup: does the account have an access grant for this product? Prefix restrictions from both the Role and the account grant are enforced. -All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container instance). Cache keys and TTLs: +**`ListBuckets`** +The proxy constructs this response from the policy store — the upstream is never called. Anonymous users see public products. STS callers with unlimited ceiling see all products the account has grants for. STS callers with scoped Roles see only the intersection of Role resource patterns and account grants. -| Lookup | Cache Key | TTL | -| --------------------------------------- | ---------------------- | ------------------------------------------------ | -| Role definition | `role_id` | In-memory constant — role definitions are static | -| Bucket public flag | `bucket_id` | 60–300s | -| Single-resource user grant | `(user_id, bucket_id)` | 30–60s | -| User's full bucket list (`ListBuckets`) | `(user_id, role_id)` | 5–10s | +**`ListObjects` (within a product)** +After the Role ceiling check, public early exit, and account permission lookup: if the Role includes a key prefix restriction, it is passed as a filter to the upstream `ListObjects` call. -The short TTL on the full bucket list (5–10 seconds) ensures that a user who creates a new dataset or joins an organisation sees that change reflected in `ListBuckets` within a few seconds — acceptable for a UI interaction, where the user would typically navigate to the new resource after creating it. +### Cache Strategy -For the Workers deployment, cache is per-isolate and is not shared across edge nodes. This is acceptable: the worst case is a single additional policy store lookup per edge node per cache interval, not a cache miss storm. Workers KV is available for a globally consistent cache tier if per-isolate TTLs prove insufficient. +All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container): -> [!NOTE] -> **TODO:** Define the exact schema of user access grants in the policy store — whether grants are bucket-level only or support sub-bucket prefix granularity; whether grants are additive (explicit allows) only or also support explicit denies; and how organisation membership is modelled (membership → derived grants, or explicit per-bucket grants per member). +| Lookup | Cache Key | TTL | +| --------------------------------------- | -------------------------- | ------- | +| Product public flag | `product_id` | 60–300s | +| Account permission for product | `(account_id, product_id)` | 30–60s | +| Account's full product list | `account_id` | 5–10s | + +The short TTL on the full product list ensures that account permission changes are reflected within seconds. For Workers, cache is per-isolate and not shared across edge nodes. Workers KV is available as a shared tier if needed. --- @@ -599,6 +583,12 @@ For access control decisions, eventual consistency is generally acceptable — a The following questions are unresolved and are the primary focus of this RFC review. Answers will be captured in the ADRs listed in [Section 14](#14-decision-index). +### Resolved + +7. ~~**Policy language and grant schema.**~~ **Resolved in ADR-004 and ADR-005 (v4).** Permission statements use `read`/`write` actions with URN resource patterns supporting product-level and prefix-level granularity. Grants are additive (allow-only). The Role acts as a ceiling on the account's existing permissions. Organisation membership is modelled through account-level grants in the policy store. + +### Open + 1. **Regional proxy access restriction.** How do we ensure regional ECS proxy deployments are only accessible to in-region consumers? VPC-only endpoints, IP range allowlisting, region-scoped session credentials, and audience claims are candidate mechanisms. What are the operational tradeoffs? 2. **Configuration store implementation.** Given that the policy store is definitively on the hot path, should we use the REST API with aggressive caching (lower risk, availability dependency) or direct DynamoDB access (faster, schema governance risk)? A hybrid approach is also possible. What schema enforcement mechanisms can mitigate the DynamoDB drift risk in Option B? @@ -611,10 +601,14 @@ The following questions are unresolved and are the primary focus of this RFC rev 6. **Middleware event backend.** What is the initial target for usage recording and billing event emission? A push-based stream (Kinesis, Pub/Sub), a pull-based log (S3, R2), a webhook, or something else? -7. **Policy language and grant schema.** Are grants bucket-level only, or do they support sub-bucket prefix granularity? Are explicit denies supported, or is the model additive (allow-only)? How is organisation membership modelled — derived grants from membership, or explicit per-bucket grants per member? - 8. **Crate governance.** What is the publication, maintenance, and contribution model for the core crates on `crates.io`? +9. **Organisation permission model.** How does organisation membership translate to account-level grants? When a user joins an org, do they automatically inherit grants for all org products, or must grants be assigned explicitly per product? How do org admin vs. member tiers affect default grants? + +10. **HMAC server secret rotation.** The HMAC derivation (`SecretAccessKey = HMAC-SHA256(server_secret, AccessKeyId)`) creates a dependency on a shared symmetric key. What is the rotation procedure? Can multiple active HMAC secrets coexist during rotation (try new key first, fall back to old)? + +11. **Multipart upload credential expiry.** Large uploads spanning many parts over slow connections may outlast the STS credential TTL. The upload cannot be resumed with new credentials (SigV4 ties the signature to the specific AccessKeyId). Should the system support credential refresh mid-upload, or is documentation (use longer TTLs for large uploads) sufficient? + --- ## 14. Decision Index @@ -626,8 +620,8 @@ The following ADRs will be produced as decisions are ratified through this RFC p | ADR-001 | S3 API compatibility and temporary-credentials-only model | Draft | | ADR-002 | Runtime: Cloudflare Workers + regional ECS strategy | Pending | | ADR-003 | Rust as implementation language | Pending | -| ADR-004 | Inbound authentication — OIDC federation, STS, SC Credential Tokens | Draft | -| ADR-005 | Authorization model — dynamic per-request policy resolution | Pending | +| ADR-004 | Inbound authentication — OIDC federation, account-owned IdPs and Roles | Draft | +| ADR-005 | Authorization model — Role ceiling with dynamic account permission resolution | Draft | | ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | Pending | | ADR-007 | Middleware architecture — rate limiting, metering, billing hooks | Pending | | ADR-008 | Modular crate architecture and community reuse model | Pending | From f0a92840346642b12fcc3b715519d5216d690576 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 23 Mar 2026 15:52:46 -0700 Subject: [PATCH 04/17] docs: finalize first draft of ADRs and RFC --- adrs/001-s3-credentials.md | 58 +++--- adrs/002-runtimes.md | 49 ++--- adrs/003-rust.md | 16 +- adrs/004-sts.md | 128 +++++-------- adrs/005-authorization.md | 43 +++-- adrs/006-outbound-storage.md | 12 +- adrs/007-middleware.md | 52 ++---- adrs/008-crate-architecture.md | 62 ++----- adrs/009-configuration.md | 105 +++++++---- adrs/rfc-001.md | 323 +++++++++++++-------------------- 10 files changed, 363 insertions(+), 485 deletions(-) diff --git a/adrs/001-s3-credentials.md b/adrs/001-s3-credentials.md index 9494f9a..25a4831 100644 --- a/adrs/001-s3-credentials.md +++ b/adrs/001-s3-credentials.md @@ -1,8 +1,7 @@ # ADR-001: S3 API Compatibility and Temporary-Credentials-Only Credential Model -**Status:** Draft +**Status:** Proposed **Date:** 2026-03-14 -**Updated:** 2026-03-22 **RFC:** RFC-001 §4 --- @@ -32,12 +31,12 @@ This is unchanged from the current proxy. S3 API compatibility is a non-negotiab All SigV4 credentials issued by Source Cooperative are temporary session credentials — the same triplet shape that AWS STS issues: ``` -AccessKeyId (e.g. "SCSTS...") +AccessKeyId (e.g. "SCSTS1...") SecretAccessKey (HMAC-derived key) SessionToken (signed JWT encoding identity, role, permissions, and expiry) ``` -Callers obtain these credentials by exchanging a trusted identity token at the STS endpoint (`POST /.sts/assume-role-with-web-identity`) before making S3 API calls. The `AccessKeyId` is prefixed with `SCSTS` to distinguish STS-issued credentials from any legacy permanent keys during the migration period. +Callers obtain these credentials by exchanging a trusted identity token at the STS endpoint (`POST /.sts/assume-role-with-web-identity`) before making S3 API calls. The `AccessKeyId` is prefixed with `SCSTS` to identify STS-issued credentials and reserve namespace for future credential types (see [Permanent API Keys](#permanent-api-keys)). ### Session Token Design @@ -45,18 +44,18 @@ The `SessionToken` is a signed JWT using ES256 (ECDSA P-256) asymmetric signing. ```json { - "jti": "", - "sub": "source::my-org::role/github-publisher", + "sub": "sc::my-org::role/github-publisher", "account_id": "my-org", "role_name": "github-publisher", "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", "assumed_by_issuer": "https://token.actions.githubusercontent.com", "session_name": "my-ci-job-42", - "access_key_id": "SCSTS...", + "access_key_id": "SCSTS1...", "permissions": [ - {"actions": ["read", "write"], "resources": ["source::my-org::product/climate-data/*"]} + {"actions": ["read", "write"], "resources": ["sc::my-org::product/climate-data/*"]} ], "iat": 1711100000, + "nbf": 1711100000, "exp": 1711103600, "aud": "data.source.coop", "kid": "" @@ -68,7 +67,8 @@ Key properties of this design: - **The `SecretAccessKey` is not in the token.** It is derived deterministically on each request: `SecretAccessKey = HMAC-SHA256(server_secret, AccessKeyId)`. The server reconstructs it by re-deriving from the `AccessKeyId`. This prevents a leaked SessionToken from directly yielding a complete credential set. - **`assumed_by` and `assumed_by_issuer`** preserve the original IdP subject for audit trails, even though the credentials act on behalf of the account. - **`permissions`** embed the Role's permission ceiling in the token, avoiding a per-request policy store lookup for Role evaluation. The account's underlying permissions are still resolved dynamically (see ADR-005). -- **`jti`** enables lightweight revocation via a deny-list (see below). +- **`nbf`** (not-before) prevents token use before the issued time. Set equal to `iat` at issuance; the verifier applies a 60-second clock skew tolerance. +- **`permissions`** are readable by anyone who intercepts the SessionToken. This is acceptable: the permission ceiling reveals the Role's scope but does not grant access without the corresponding SecretAccessKey (which requires the server secret to derive). - **`kid`** in the JWT header supports signing key rotation. ### SigV4 Verification Flow @@ -76,10 +76,10 @@ Key properties of this design: The proxy verifies incoming SigV4 requests by: 1. Extracting the `AccessKeyId` from the `Authorization` header -2. Detecting the `SCSTS` prefix to identify this as an STS credential -3. Deriving the `SecretAccessKey` via `HMAC-SHA256(server_secret, AccessKeyId)` +2. Detecting the `SCSTS` prefix to identify this as an STS credential. The digit following `SCSTS` is the HMAC key version (e.g., `SCSTS1...` uses key version 1), enabling key rotation without invalidating active sessions +3. Deriving the `SecretAccessKey` via `HMAC-SHA256(server_secret[version], AccessKeyId)` 4. Verifying the SigV4 signature using the derived secret -5. Extracting and verifying the `SessionToken` JWT from the `X-Amz-Security-Token` header — checking ES256 signature, `exp`, `aud`, and `jti` against the revocation deny-list +5. Extracting and verifying the `SessionToken` JWT from the `X-Amz-Security-Token` header — checking ES256 signature, `exp`, `nbf` (with 60s clock skew tolerance), and `aud` 6. Proceeding to authorization (see ADR-005) using the token's embedded identity and permissions No external database lookup is required to verify a request or reconstruct the signing key. The token and HMAC derivation together are self-contained. @@ -89,17 +89,17 @@ No external database lookup is required to verify a request or reconstruct the s - **Asymmetric signing:** ES256 (ECDSA P-256). The private key is used only for token issuance; the public key is served at a JWKS endpoint for verification. - **Key storage:** Private key stored in KMS (AWS KMS or equivalent). - **Key rotation:** The `kid` header in issued JWTs allows multiple active signing keys. During rotation, new tokens are signed with the new key while tokens signed with the old key remain valid until they expire. The old key is retired after one `max_session_duration` interval. -- **HMAC server secret:** A separate symmetric key used for SecretAccessKey derivation. Stored alongside the signing key in KMS. Rotation follows the same pattern — new AccessKeyIds use the new secret; existing sessions continue to work until expiry. +- **HMAC server secret:** A separate symmetric key used for SecretAccessKey derivation. Stored alongside the signing key in KMS. The initial implementation uses a single HMAC key version (`SCSTS1`). The version indicator in the AccessKeyId prefix is reserved for future key rotation support. -### Revocation - -Session tokens support lightweight revocation via a deny-list of `jti` values: +> [!NOTE] +> **Future extension: HMAC key rotation.** The `SCSTS1` prefix embeds a key version indicator. When rotation is needed, the proxy can be updated to support multiple active key versions (e.g., `SCSTS1` → `SCSTS2`): new sessions are issued with the new version, the proxy derives the SecretAccessKey using the version indicated by the prefix, and the old key is retired after one `max_session_duration` interval beyond the last issuance. For incident response before rotation is implemented, replacing the single HMAC server secret invalidates all active sessions. -- Revoked `jti` values are stored in Cloudflare KV (or equivalent) with a TTL matching the token's remaining lifetime. Entries self-clean as tokens expire. -- Every authenticated request checks the deny-list (~1ms KV lookup at edge). -- Account admins can revoke tokens via `POST /.sts/revoke-session-token`. +### Revocation -This is a targeted mechanism for incident response, not a general session management system. The deny-list stays small because entries expire automatically. +> [!NOTE] +> **Deferred.** Per-token revocation (via a `jti` deny-list checked on every request) is not included in the initial implementation. Short-lived credentials (15 min to 12 hours) bound the exposure window of a compromised token. For incident response, rotating the HMAC server secret or the JWT signing key invalidates all active sessions. +> +> Per-token revocation can be added later by: (1) adding a `jti` claim to the SessionToken, (2) storing revoked `jti` values in Cloudflare KV with TTLs matching remaining token lifetime, and (3) checking the deny-list on each authenticated request. This is a backwards-compatible addition — existing tokens without `jti` are simply not revocable. ### Accepted Trade-offs @@ -120,8 +120,7 @@ This is a targeted mechanism for incident response, not a general session manage - The session token is stateless and self-verifying — no credential store on the hot path. - The SecretAccessKey never appears in the SessionToken, limiting the blast radius of token leakage. - Asymmetric signing (ES256) means verification requires only the public key; the private signing key has a minimal attack surface. -- Revocation is supported via a lightweight deny-list without adding a stateful dependency to the verification hot path. -- Short-lived credentials (15 min to 12 hours) further limit blast radius. +- Short-lived credentials (15 min to 12 hours) limit blast radius, eliminating the need for per-token revocation in the initial implementation. - Composable with OIDC workload identity federation (see ADR-004) — the exchange step is the same regardless of the upstream identity source. **Costs / Risks** @@ -129,18 +128,25 @@ This is a targeted mechanism for incident response, not a general session manage - Callers must perform a token exchange before first use. This is new friction compared to the current static key model. - The `/.sts` exchange endpoint is on the critical path for session establishment. Its availability affects whether callers can obtain credentials. - The HMAC server secret is a high-value target. Its compromise, combined with a captured SessionToken, yields the corresponding SecretAccessKey. -- The revocation deny-list introduces a KV dependency on the request path, though the lookup is fast (~1ms) and the system degrades gracefully if KV is unavailable (tokens remain valid until expiry). +- No per-token revocation in the initial implementation. The only incident response option is rotating the server-wide HMAC secret or JWT signing key, which invalidates all active sessions. Per-token revocation can be added later (see [Revocation](#revocation)). - S3 tooling that hardcodes static credential configuration (rather than using the SDK credential provider chain) may require workarounds. --- +## Permanent API Keys + +> [!NOTE] +> **Not included in the initial implementation.** The proxy supports only STS-issued session credentials and anonymous access. Long-lived API keys may be added in the future for workflows where neither workload identity federation nor interactive authentication via `auth.source.coop` is feasible — for example, on-premises instruments, legacy ETL systems, or environments without OIDC support. +> +> API keys would be exchanged for temporary STS credentials at the `/.sts` endpoint — the same way OIDC tokens are exchanged today. This keeps the proxy's request-time verification uniform: only short-lived STS credentials are accepted on S3 API calls. No second authorization path is needed. + +--- + ## Alternatives Considered **Long-lived static credentials (current model)** — rejected. Persistent security liability; does not compose with workload identity federation; difficult to audit or rotate at scale. -**Embedding the SecretAccessKey in the SessionToken** — rejected. A leaked SessionToken would contain a complete, self-sufficient credential set. HMAC derivation separates the signing material from the bearer token, requiring compromise of both the server secret and a token to reconstruct credentials. - -**Server-side session store for SecretAccessKey** — considered. Generating a random SecretAccessKey per session and storing it in KV eliminates the HMAC shared secret risk entirely. Rejected for now: adds a mandatory KV read on every request for credential verification (not just revocation checks). The HMAC approach keeps verification fully stateless. Can be revisited if the threat model changes. +**Server-side session store for SecretAccessKey** — considered. Generating a random SecretAccessKey per session and storing it in a server-side store (KV or database) eliminates the HMAC shared secret risk entirely — there is no single key whose compromise affects all sessions. Rejected for now: adds a mandatory store read on every request for credential verification. The HMAC approach keeps verification fully stateless — the server derives the SecretAccessKey from the AccessKeyId without any external lookup. Can be revisited if the threat model changes or if a per-request store dependency is introduced for other reasons. **Symmetric signing (HS256)** — rejected. Would require the signing secret to be available on all verification endpoints, expanding the attack surface. ES256 limits the private key to the issuance path only. diff --git a/adrs/002-runtimes.md b/adrs/002-runtimes.md index 4bde41d..220893c 100644 --- a/adrs/002-runtimes.md +++ b/adrs/002-runtimes.md @@ -1,6 +1,6 @@ -# ADR-002: Runtime — Cloudflare Workers (Primary) + Regional ECS (Secondary) +# ADR-002: Runtime — Cloudflare Workers -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §5 @@ -10,20 +10,15 @@ Source Cooperative's data proxy serves users globally, but most upstream data resides in AWS `us-west-2`. Users far from that region experience significant latency. Replicating data to additional regions is cost-prohibitive. -The proxy needs two deployment modes to serve distinct access patterns: - -1. **Global, latency-sensitive reads** — the majority of traffic. Users worldwide reading public datasets. These benefit from edge deployment close to the caller. -2. **High-throughput, in-region workflows** — data pipelines (Spark, Databricks, Polars) running in `us-west-2` reading large volumes from S3 in the same region. Routing this traffic through an edge node adds unnecessary hops and latency. - -The current proxy is a single ECS deployment. It handles both patterns, but serves neither optimally. +The current proxy is a single ECS deployment. It works, but provides no edge presence for global users. --- ## Decision -### Cloudflare Workers (Primary) +### Cloudflare Workers -The primary deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. Workers deploy to Cloudflare's edge network (330+ locations worldwide) automatically. +The deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. Workers deploy to Cloudflare's edge network (330+ locations worldwide) automatically. Key properties: @@ -34,25 +29,8 @@ Key properties: - **Predictable, low cost.** $5/mo base, $0.30/M requests, $0.02/M CPU-ms; 10M requests + 30M CPU-ms included. - **WASM compatibility.** Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`). -### Regional ECS Deployments (Secondary) - -Traditional containerised Rust services deployed into specific cloud regions on demand. Intended for high-throughput, in-region workflows where: - -- Egress fees are zero or near-zero when traffic stays within the region -- Network throughput is higher and latency is lower than routing through an edge node -- The Workers path adds unnecessary hops - -Regional deployments share the same Rust core as the Workers deployment. The proxy logic, auth, and authz layers are identical; only the runtime adapter differs. - -### Shared STS and Credential Interoperability - -Each deployment target hosts its own STS endpoint at `/.sts`. Workers and all regional ECS deployments share the same signing key material, so session credentials issued by any target are valid across all targets. - -### Accepted Trade-offs - -**Regional access restriction is unresolved.** Regional proxies should only be accessible to in-region consumers. Candidate mechanisms include VPC-only endpoints, IP range allowlisting, region-scoped audience claims, and regional-specific session credentials. Each has tradeoffs around operational complexity and developer experience. See RFC-001 Open Question 1. - -**Two deployment targets increase operational surface.** The shared Rust core mitigates code divergence, but deployment, monitoring, and key management are duplicated. +> [!NOTE] +> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — data pipelines (Spark, Databricks, Polars) running in the same cloud region as the source data — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. The trait-based architecture (ADR-008) is designed to support additional runtime targets without code divergence. This can be pursued when there is demonstrated demand. --- @@ -61,17 +39,14 @@ Each deployment target hosts its own STS endpoint at `/.sts`. Workers and all re **Benefits** - Global users experience lower latency without data replication -- In-region workflows avoid unnecessary edge hops and egress charges - No Cloudflare egress fees for the majority of traffic -- Effectively zero cold start for the primary deployment target -- Shared core ensures behavioural consistency across targets +- Effectively zero cold start +- Single deployment target keeps operational surface small **Costs / Risks** -- Two deployment targets to build, test, deploy, and monitor -- WASM compilation constrains library choices for the shared core (no `std` features that don't work in WASM) -- Regional access restriction mechanism is unresolved -- Credential interoperability across targets requires shared key material and coordinated rotation +- WASM compilation constrains library choices (no `std` features that don't work in WASM) +- In-region, high-throughput workflows (e.g. bulk ETL in `us-west-2`) route through the edge rather than staying within the region — this adds latency and may incur upstream egress fees that an in-region proxy would avoid --- @@ -81,6 +56,6 @@ Each deployment target hosts its own STS endpoint at `/.sts`. Workers and all re **CDN in front of ECS** — considered. A traditional CDN (CloudFront, Cloudflare) can cache static responses, but the proxy's responses are not cacheable in a general-purpose CDN sense (authenticated, per-user). The proxy logic must run at the edge, not just caching. -**Workers only (no regional ECS)** — considered. Simpler operationally, but penalises high-throughput in-region workflows with unnecessary hops and potentially higher latency for large data transfers within the same cloud region. +**Workers + Regional ECS** — considered as the initial deployment. Simpler to start with Workers only and add regional ECS deployments when demand materialises. The trait-based architecture supports this without requiring upfront investment in a second deployment target. **Lambda@Edge / CloudFront Functions** — considered. More limited runtime environment, tighter CPU and memory constraints, and AWS-specific. Workers offer a more capable and provider-neutral edge compute model. diff --git a/adrs/003-rust.md b/adrs/003-rust.md index e4c8865..fbf029a 100644 --- a/adrs/003-rust.md +++ b/adrs/003-rust.md @@ -1,6 +1,6 @@ # ADR-003: Rust as Implementation Language -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §6 @@ -8,7 +8,7 @@ ## Context -The re-architected proxy must compile to two targets: WebAssembly for Cloudflare Workers (ADR-002) and a native binary for regional ECS deployments. The language must support both targets from a single codebase. Performance matters for the ECS target (streaming large objects with tight latency requirements). The proxy handles security-sensitive operations: cryptographic signature verification, credential issuance, and access policy evaluation. +The re-architected proxy must compile to WebAssembly for Cloudflare Workers (ADR-002). The language must also support native compilation from the same codebase to enable future deployment targets. The proxy handles security-sensitive operations: cryptographic signature verification, credential issuance, and access policy evaluation. The current proxy is written in Rust. The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known but is unsuitable for the WASM target. @@ -16,13 +16,13 @@ The current proxy is written in Rust. The Source Cooperative contributor communi ## Decision -We continue with **Rust** as the implementation language for both deployment targets. +We continue with **Rust** as the implementation language. ### Rationale **WASM maturity.** Rust has the most mature and production-ready toolchain for compiling to WebAssembly. The `worker-rs` crate provides idiomatic bindings to the Cloudflare Workers runtime. This is a well-trodden path, not a bet on emerging capability. -**Performance.** For the regional ECS deployment, raw throughput matters. Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that streams large objects with tight latency requirements. This was already proven by the current proxy. +**Performance.** Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that streams large objects with tight latency requirements. This was already proven by the current proxy. **Type system and correctness.** The proxy handles authentication tokens, credential issuance, cryptographic signature verification, and access policy evaluation. Rust's type system — and in particular its trait system — encodes invariants that would be runtime errors in other languages. This is increasingly valuable in a codebase where AI-assisted development is part of the workflow: a strong type system provides a correctness harness that catches generated code that compiles but violates domain constraints. @@ -36,7 +36,7 @@ We continue with **Rust** as the implementation language for both deployment tar **Benefits** -- Single codebase compiles to both WASM and native targets +- Single codebase supports WASM and native compilation targets - Zero-cost abstractions and no GC pauses for high-throughput streaming - Trait system enables the modular, community-extensible architecture - Strong type system as a correctness harness for security-sensitive code @@ -47,7 +47,7 @@ We continue with **Rust** as the implementation language for both deployment tar - Steeper learning curve for new contributors compared to Go or Python - Longer compilation times than Go - WASM target constrains which crates and `std` features can be used in the shared core -- Async ecosystem split (`tokio` for ECS, `worker-rs` primitives for Workers) requires careful abstraction +- Async runtime differs between Workers (`worker-rs` primitives) and native targets (`tokio`), requiring careful abstraction if additional deployment targets are added --- @@ -55,8 +55,8 @@ We continue with **Rust** as the implementation language for both deployment tar **Go** — considered. Strong WASM support is emerging but less mature than Rust's. Lacks the trait system needed for the modularity goals. GC pauses are a concern for high-throughput streaming. Fewer Rust contributors would need to learn a new language than Go contributors. -**TypeScript (native Workers language)** — considered. First-class Workers support, but unsuitable for the ECS target's performance requirements. No type-level enforcement of security invariants comparable to Rust's ownership and trait system. +**TypeScript (native Workers language)** — considered. First-class Workers support, but limited performance for streaming workloads. No type-level enforcement of security invariants comparable to Rust's ownership and trait system. -**Python** — rejected. Does not compile to WASM. Runtime overhead incompatible with the regional proxy's performance goals. +**Python** — rejected. Does not compile to WASM. Runtime overhead unsuitable for a streaming proxy. **C++** — rejected. Less community familiarity than Rust. Memory safety concerns for security-sensitive code. No comparable trait system for extensibility. diff --git a/adrs/004-sts.md b/adrs/004-sts.md index a0ae088..2419cbd 100644 --- a/adrs/004-sts.md +++ b/adrs/004-sts.md @@ -1,8 +1,7 @@ -# ADR-004: Inbound Authentication — OIDC Federation, Account-Owned Identity Providers, and Role-Based STS Exchange +# ADR-004: Inbound Authentication — OIDC Federation, Platform IdPs, and Role-Based STS Exchange -**Status:** Draft +**Status:** Proposed **Date:** 2026-03-14 -**Updated:** 2026-03-22 **RFC:** RFC-001 §7 **Depends on:** ADR-001 @@ -21,7 +20,7 @@ ADR-001 establishes that all callers must obtain short-lived SigV4 session crede The industry standard for secretless workload authentication is OIDC workload identity federation: a workload presents a signed JWT issued by its platform's OIDC provider to a Security Token Service, which validates it and returns short-lived scoped credentials. AWS, GCP, Azure, GitHub Actions, GitLab CI, Vercel, and many others all support this pattern. -The key design choice in this ADR is that **accounts own their Identity Providers and Roles**. Rather than a fixed set of platform-managed issuers and a small set of static roles, each account (Individual or Organization) can register IdPs and create Roles that scope access to their resources. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. +The key design choice in this ADR is that **accounts own their Roles**. Rather than a small set of static roles, each account (Individual or Organization) can create Roles that scope access to their resources, constrained to platform-registered IdPs. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. --- @@ -60,52 +59,14 @@ The `well_known_claims` field provides documentation and UI hints — when a use This list is illustrative, not exhaustive. Platform operators can add new issuers over time without code changes. -#### Account IdPs - -Registered by account owners (Individual or Organization). These support corporate identity systems, self-hosted OIDC providers, or any issuer with a publicly reachable JWKS endpoint. - -**Registration API:** - -``` -POST /api/accounts/{account_id}/idps -{ - "display_name": "Our Corporate Okta", - "issuer_url": "https://corp.okta.com/oauth2/default" -} -``` - -**Validation at registration time:** - -1. `issuer_url` must be HTTPS -2. Must not match any platform IdP issuer URL (exact match after canonicalization — prevents impersonation of well-known issuers) -3. Must not duplicate another IdP already registered on the same account -4. Issuer URL canonicalized before storage: lowercase scheme and host, strip trailing slash -5. Fetch `{issuer_url}/.well-known/openid-configuration` — must return a valid OIDC discovery document -6. Resolved IP must not be private, loopback, or link-local (SSRF protection) -7. Fetch timeout: 3 seconds, response body limit: 256KB - -**Stored record:** - -```json -{ - "id": "uuid", - "account_id": "my-org", - "issuer_url": "https://corp.okta.com/oauth2/default", - "display_name": "Our Corporate Okta", - "created_at": "2025-03-22T...", - "created_by": "user-id" -} -``` - -No JWKS is stored at registration time. JWKS is fetched and cached at STS exchange time from the OIDC discovery document's `jwks_uri`. - -**Deletion:** Deleting an IdP is blocked if any Role references it. The account must first remove the IdP binding from all Roles, then delete the IdP. +> [!NOTE] +> **Future extension: Account-registered IdPs.** The initial implementation supports platform IdPs only. Account-level IdP registration — allowing account owners to register corporate identity systems (Okta, Entra ID), self-hosted providers (Keycloak), or any OIDC-compliant issuer — can be added later without changing the Role schema or STS exchange flow. The Role's `idp` field already accepts an IdP identifier; extending it to reference account-registered IdPs requires: (1) a registration API with SSRF-safe URL validation, (2) an account IdP storage table, and (3) IdP deletion guards when Roles reference it. Platform operators can add new well-known issuers to the platform IdP list at any time without code changes, covering the most common needs in the interim. ### Roles Roles belong to an account (Individual or Organization) and define two things: **who can assume the Role** (identity constraints) and **what the Role's credentials can access** (permission statements). -Roles are identified by URN: `source::{account_id}::role/{role_name}` +Roles are identified by URN: `sc::{account_id}::role/{role_name}` #### Role Schema @@ -122,23 +83,16 @@ Roles are identified by URN: `source::{account_id}::role/{role_name}` {"claim": "repository", "operator": "equals", "value": "my-org/my-repo"}, {"claim": "ref", "operator": "starts_with", "value": "refs/heads/"} ] - }, - { - "idp": "uuid-of-account-idp", - "audience": "https://data.source.coop", - "claim_constraints": [ - {"claim": "sub", "operator": "equals", "value": "service-account-42"} - ] } ], "permissions": [ { "actions": ["read", "write"], - "resources": ["source::my-org::product/climate-data/*"] + "resources": ["sc::my-org::product/climate-data/*"] }, { "actions": ["read"], - "resources": ["source::my-org::product/reference-data/*"] + "resources": ["sc::my-org::product/reference-data/*"] } ] } @@ -148,15 +102,17 @@ A Role acts as a **ceiling** on the account's existing permissions. The credenti #### Identity Constraints -Each Role specifies one or more IdP bindings. Each binding identifies an IdP (platform or account-registered) and a set of claim constraints that the presented JWT must satisfy. +Each Role specifies one or more IdP bindings. Each binding identifies a platform IdP and a set of claim constraints that the presented JWT must satisfy. -**Claim constraint operators** (deliberately minimal): +**Claim constraint operators:** | Operator | Behaviour | Example | |----------|-----------|---------| | `equals` | Exact string match | `repository` equals `my-org/my-repo` | | `starts_with` | String prefix match | `ref` starts_with `refs/heads/` | -| `glob` | Wildcard: `*` (any chars), `?` (single char) | `repository` glob `my-org/*` | + +> [!NOTE] +> **Future extension: `glob` operator.** A glob operator (wildcard `*`, single-char `?`) was considered but deferred. `equals` and `starts_with` cover the initial use cases (constraining to a specific repo, a branch prefix, etc.). Adding `glob` later is backwards-compatible — it introduces a new operator value without changing existing constraint evaluation. It will be added if users need mid-string wildcards (e.g., `refs/heads/release-*`). Rules: - All claim values are coerced to strings before comparison. JWT claims that are arrays or objects evaluate to false. @@ -164,7 +120,6 @@ Rules: - Multiple IdP bindings on a Role are ORed — any one binding can match. - A missing claim evaluates to false (fail-closed). - Only top-level claims are supported — no nested path traversal. -- No regex. Glob is the most expressive operator. This avoids ReDoS and keeps the constraint language auditable. #### Permission Statements @@ -172,22 +127,22 @@ Permission statements define what the Role's credentials can access. Resource pa ``` * → all resources (unlimited ceiling) -source::{account_id}::product/* → all of an account's products -source::{account_id}::product/{product_name} → entire product -source::{account_id}::product/{product_name}/* → entire product (equivalent) -source::{account_id}::product/{product_name}/{prefix}/* → prefix-scoped -source::{account_id}::product/{product_name}/{key} → single object +sc::{account_id}::product/* → all of an account's products +sc::{account_id}::product/{product_name} → entire product +sc::{account_id}::product/{product_name}/* → entire product (equivalent) +sc::{account_id}::product/{product_name}/{prefix}/* → prefix-scoped +sc::{account_id}::product/{product_name}/{key} → single object ``` Rules: - Resource patterns can reference any account's products. A Role can delegate access to products the account has access to, even if owned by another account or org. The request-time intersection enforces the real boundary. - `*` as the entire resource value means "all resources" — no ceiling. The account's actual permissions are the sole constraint. -- Actions are `read` and `write`. `read` maps to `GetObject`, `HeadObject`, `ListObjects`. `write` maps to `PutObject`, `DeleteObject`, and multipart operations. +- Actions are `read` and `write`. `read` maps to `GetObject`, `HeadObject`, `ListObjects`. `write` maps to `PutObject`, `DeleteObject`, and multipart operations. Finer-grained actions (e.g., separating `upload` from `delete`) can be introduced later as new action values without breaking existing Role definitions. - Permission statements are additive (allow-only). No explicit denies. #### Built-in Default Role -Every account has a built-in Role: `source::{account_id}::role/_default` +Every account has a built-in Role: `sc::{account_id}::role/_default` - Cannot be deleted - Constrained to only the `auth.source.coop` platform IdP @@ -199,7 +154,7 @@ This Role serves interactive users who authenticate via `auth.source.coop` and w #### Role Validation at Creation 1. `name` must match `[a-z0-9][a-z0-9-]{0,62}` (lowercase, hyphens, max 63 chars) -2. Each IdP reference must exist (platform IdP by well-known ID, account IdP by UUID) +2. Each IdP reference must be a valid platform IdP well-known ID 3. `max_session_duration` between 900 and 43200 seconds (15 min to 12 hours) 4. At least one identity constraint required 5. At least one permission statement required @@ -230,7 +185,7 @@ Dot-prefixed account names (`.sts`, etc.) are reserved as invalid, preventing ro ``` Action=AssumeRoleWithWebIdentity &WebIdentityToken= -&RoleArn=source::my-org::role/github-publisher +&RoleArn=sc::my-org::role/github-publisher &RoleSessionName=my-ci-job-42 &DurationSeconds=3600 ``` @@ -241,14 +196,14 @@ Action=AssumeRoleWithWebIdentity - SCSTS... + SCSTS1... derived-secret eyJ... 2025-03-22T13:00:00Z - source::my-org::role/github-publisher - SCSTS...:my-ci-job-42 + sc::my-org::role/github-publisher + SCSTS1...:my-ci-job-42 @@ -278,6 +233,8 @@ The Role URN is required. The caller must know which Role to assume — the syst - If no cache and fetch fails → return `IDPCommunicationError` - Max response body: 256KB +In a distributed deployment (multiple Workers isolates), each instance maintains its own JWKS cache. Different isolates may briefly hold different versions of an issuer's JWKS during key rotation. In practice this is unlikely to cause issues — OIDC providers publish new keys well before retiring old ones, and the 1-hour TTL keeps caches reasonably fresh. If transient validation failures are observed during IdP key rotations, a shared cache tier (e.g. Workers KV) can be introduced. + ### Error Responses Errors use AWS STS XML format for SDK compatibility: @@ -286,8 +243,7 @@ Errors use AWS STS XML format for SDK compatibility: InvalidIdentityToken - JWT claim 'repository' value 'my-org/wrong-repo' does not match - constraint 'my-org/correct-repo' on role 'github-publisher' + JWT claims do not satisfy the identity constraints for role 'github-publisher' ``` @@ -305,7 +261,7 @@ Errors use AWS STS XML format for SDK compatibility: | IdP JWKS endpoint unreachable | `IDPCommunicationError` | 400 | | `DurationSeconds` exceeds max | `ValidationError` | 400 | -Error messages include enough detail for callers to diagnose problems. The Role definition is not secret — the account admin created it. +Error messages identify the failing condition without revealing the expected constraint values. The Role name is included because the caller already knows it (they specified it in the request). Constraint values (expected repository, expected ref pattern, etc.) are omitted to avoid leaking the Role's trust policy to unauthorized callers. Detailed constraint mismatch information is available in the server-side audit log for account admins to diagnose. ### Observability @@ -317,12 +273,12 @@ Every STS exchange (success or failure) emits a structured log entry: "timestamp": "2025-03-22T12:00:00Z", "account_id": "my-org", "role_name": "github-publisher", - "role_urn": "source::my-org::role/github-publisher", + "role_urn": "sc::my-org::role/github-publisher", "idp_issuer": "https://token.actions.githubusercontent.com", "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", "session_name": "my-ci-job-42", "result": "success", - "access_key_id": "SCSTS...", + "access_key_id": "SCSTS1...", "duration_seconds": 3600, "client_ip": "...", "failure_reason": null @@ -356,10 +312,10 @@ The existing `source-coop-cli` handles authentication and credential management. ```ini [profile source-read] -credential_process = source-coop creds --role-arn source::my-org::role/reader +credential_process = source-coop creds --role-arn sc::my-org::role/reader [profile source-write] -credential_process = source-coop creds --role-arn source::my-org::role/publisher +credential_process = source-coop creds --role-arn sc::my-org::role/publisher ``` The `source-coop creds` command checks for cached valid credentials, triggers OIDC login if needed, calls the STS endpoint, and returns credentials in `credential_process` JSON format. @@ -372,7 +328,7 @@ permissions: steps: - uses: source-cooperative/configure-credentials@v1 with: - role-urn: source::my-org::role/github-publisher + role-urn: sc::my-org::role/github-publisher - run: aws s3 cp data.parquet s3://data.source.coop/my-org/my-product/ ``` @@ -383,7 +339,7 @@ Requests a GitHub OIDC token with audience `https://data.source.coop`, calls the ```python sts = boto3.client('sts', endpoint_url='https://data.source.coop/.sts') creds = sts.assume_role_with_web_identity( - RoleArn='source::my-org::role/github-publisher', + RoleArn='sc::my-org::role/github-publisher', WebIdentityToken=token, RoleSessionName='my-job' ) @@ -395,8 +351,8 @@ creds = sts.assume_role_with_web_identity( **Benefits** -- Accounts own their IdPs and Roles. No operator intervention required to add a new identity source or create new access scopes. -- Open to any OIDC-compliant issuer without code changes — both platform-provided and account-registered. +- Accounts own their Roles. No operator intervention required to create new access scopes. +- Open to any platform-registered OIDC issuer without code changes. Account-level IdP registration can be added later. - All credential paths converge on the same `/.sts` endpoint — one validation and issuance code path. - The Role-as-ceiling model prevents privilege escalation: a Role can never grant more access than the account itself has. - AWS STS-compatible request/response format enables use of existing AWS SDKs with a custom endpoint URL. @@ -409,17 +365,15 @@ creds = sts.assume_role_with_web_identity( - The `/.sts` endpoint is on the critical path for session establishment. Its availability and latency directly affect all new sessions. - JWKS fetching and caching must be robust — a stale or unavailable JWKS causes all exchange attempts for that issuer to fail. -- Account-registered IdPs introduce an SSRF risk via the JWKS fetch. Mitigated by URL validation at registration time (HTTPS only, no private IPs, valid OIDC discovery document). -- Account IdP registration means account owners control their own trust anchors. A malicious or compromised account IdP can forge arbitrary JWT claims — this is safe because the Role ceiling model prevents escalation beyond the account's own permissions, but it means audit trails for account-registered IdPs reflect self-asserted identity. -- The claim constraint language is deliberately minimal (equals, starts_with, glob). Complex matching requirements (e.g., numeric comparisons, set membership) are not supported. This can be extended later if needed. -- Platform IdP issuer collision must be enforced at registration time — an account must not be able to register a custom IdP whose issuer URL matches a platform IdP. -- Adding a new platform IdP requires operator access. Self-service IdP registration covers the account-level case; platform-level additions are a governance decision. +- Only platform-registered IdPs are supported initially. Accounts that need corporate identity systems (Okta, Keycloak) must request that the platform operator add the issuer, or wait for account-level IdP registration (see Future extension note above). +- The claim constraint language is deliberately minimal (`equals`, `starts_with`). Complex matching requirements (e.g., glob patterns, numeric comparisons, set membership) are not supported. This can be extended later if needed (see the `glob` future extension note). +- Adding a new platform IdP requires operator access. This is a governance decision, not a user self-service action. --- ## Alternatives Considered -**Fixed platform-managed issuer registry (no account IdPs)** — rejected. Would require operator intervention for every custom identity source. Accounts using corporate Okta, self-hosted Keycloak, or other non-standard IdPs would have no path to OIDC federation without platform support. +**Account-registered IdPs (self-service)** — deferred. Would allow accounts to register corporate Okta, self-hosted Keycloak, or any OIDC issuer without operator intervention. Introduces SSRF risk (JWKS fetch to user-controlled URLs), DNS rebinding concerns, and self-asserted identity trust issues. The platform IdP list covers the primary use cases; account-level IdP registration can be added when demand materialises. **Fixed role set (`anonymous`, `authenticated_user`, `admin`)** — rejected. Insufficient for the delegation use case. Accounts need to create Roles scoped to specific resources and constrained to specific IdPs — for example, "GitHub Actions from repo X can write to dataset Y." A fixed role set cannot express this. diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md index 2ad2cc3..974cfa2 100644 --- a/adrs/005-authorization.md +++ b/adrs/005-authorization.md @@ -1,8 +1,7 @@ # ADR-005: Authorization Model — Role Ceiling with Dynamic Account Permission Resolution -**Status:** Draft +**Status:** Proposed **Date:** 2026-03-14 -**Updated:** 2026-03-22 **RFC:** RFC-001 §8 **Depends on:** ADR-001, ADR-004 @@ -53,9 +52,11 @@ Authorization proceeds in steps, with early exits to minimise lookups: **Step 1 — Identify the caller** - **No credentials** → anonymous. Only read actions on public products are permitted. -- **Permanent API key** (non-`SCSTS` prefix) → legacy path. Look up account via Source API. - **STS credentials** (`SCSTS` prefix) → derive SecretAccessKey via HMAC, verify SigV4, decode SessionToken JWT. +> [!NOTE] +> **Future extension: Permanent API keys.** The initial implementation supports only STS credentials and anonymous access. Long-lived API keys may be needed in the future for workflows where neither workload identity federation nor interactive authentication via `auth.source.coop` is feasible — for example, on-premises instruments, legacy ETL systems, or environments without OIDC support. Rather than adding a second authorization path to the proxy, API keys would be exchanged for temporary STS credentials at the `/.sts` endpoint — the same way OIDC tokens are. The proxy's request-time authorization remains uniform: only short-lived STS credentials are accepted on S3 API calls. + **Step 2 — Role action check (in-memory, no lookup)** For anonymous callers, only read actions are permitted (`GetObject`, `HeadObject`, `ListObjects`, `ListBuckets`). Deny writes immediately. @@ -75,14 +76,16 @@ For read requests: if the product is public (`data_mode: open`), permit immediat **Step 5 — Account permission lookup (cached, 30–60s TTL)** For non-public resources or write operations: -1. Fetch the account's permissions from the policy store (the account referenced in the SessionToken's `account_id`) -2. Compute: `(Role ceiling permissions from token) ∩ (account's actual permissions from policy store)` +1. Fetch the account's permissions from the Source Cooperative API (the account referenced in the SessionToken's `account_id`) +2. Compute: `(Role ceiling permissions from token) ∩ (account's actual permissions from API)` 3. If the intersection includes the requested action on the requested resource → permit 4. Otherwise → deny +The proxy does not evaluate org membership or permission inheritance logic — the API resolves these internally. When a user belongs to an organisation, the API includes permissions inherited through that membership in the account's resolved grants. The proxy treats the API response as the authoritative set of permissions for the account. + **Step 6 — Prefix enforcement** -If the Role's permission statement includes a prefix constraint (e.g., `source::my-org::product/my-dataset/uploads/*`), verify the object key falls within that prefix. This enforcement is part of Step 2 and Step 5 — the prefix is evaluated when matching the resource pattern. +If the Role's permission statement includes a prefix constraint (e.g., `sc::my-org::product/my-dataset/uploads/*`), verify the object key falls within that prefix. This enforcement is part of Step 2 and Step 5 — the prefix is evaluated when matching the resource pattern. ### Authorization Truth Table @@ -114,18 +117,18 @@ After the Role ceiling check, public early exit, and account permission lookup: When evaluating whether a request matches a Role's permission statements, the proxy checks: -1. **Action match:** Does the statement's `actions` array include the requested action class (`read` or `write`)? +1. **Action match:** Does the statement's `actions` array include the requested action class (`read` or `write`)? See ADR-004 for the definition of action classes. 2. **Resource match:** Does the statement's `resources` array contain a pattern that matches the requested resource? - `*` matches everything - - `source::{account}::product/{name}` or `source::{account}::product/{name}/*` matches the entire product - - `source::{account}::product/{name}/{prefix}/*` matches objects under the prefix - - `source::{account}::product/{name}/{key}` matches a single object + - `sc::{account}::product/{name}` or `sc::{account}::product/{name}/*` matches the entire product + - `sc::{account}::product/{name}/{prefix}/*` matches objects under the prefix + - `sc::{account}::product/{name}/{key}` matches a single object If any statement matches both action and resource, the Role permits the request. The account permission lookup then determines whether the account actually has the underlying access. ### Cache Strategy -All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container): +All policy store lookups are cached in-process (per-isolate): | Lookup | Cache Key | TTL | |---|---|---| @@ -150,7 +153,7 @@ Every S3 request with STS credentials emits a structured log entry: "session_name": "my-ci-job-42", "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", "action": "PutObject", - "resource": "source::my-org::product/climate-data/2025/data.parquet", + "resource": "sc::my-org::product/climate-data/2025/data.parquet", "result": "allow", "client_ip": "..." } @@ -158,6 +161,22 @@ Every S3 request with STS credentials emits a structured log entry: This provides full auditability: which account, which Role, which original identity, and what they accessed. +### S3 Error Responses + +When authorization denies a request, the proxy returns a standard S3 error response: + +```xml + + AccessDenied + Access Denied + ... + +``` + +HTTP status is `403 Forbidden`. The error body does not reveal whether the denial was due to the Role ceiling, missing account permissions, or a non-existent product — this prevents information leakage about resource existence. + +For `ListBuckets` and `ListObjects`, the proxy filters results silently rather than returning errors. The caller sees only the resources they have access to. + --- ## Consequences diff --git a/adrs/006-outbound-storage.md b/adrs/006-outbound-storage.md index 0779b30..e6c2ef9 100644 --- a/adrs/006-outbound-storage.md +++ b/adrs/006-outbound-storage.md @@ -1,6 +1,6 @@ # ADR-006: Outbound Connectivity — OIDC Issuer Model and `object_store` Adoption -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §9 **Depends on:** ADR-002 @@ -39,11 +39,11 @@ This model means: - The trust relationship is declarative and auditable - Key rotation at the proxy level propagates automatically without reconfiguring upstream providers -### Outbound Authentication — Stored Secrets (Fallback) +### Outbound Authentication — Stored Credentials (Fallback) -For upstream providers or storage systems that do not support OIDC workload identity federation, credentials may be stored as encrypted secrets and injected into the proxy's configuration at startup. +The current proxy fetches static cloud credentials (access key ID and secret access key) from the Source Cooperative API for each data connection. The API stores these credentials and serves them to the proxy on demand, cached with a short TTL. -This is a fallback, not the preferred path. +For upstream providers or storage systems that do not support OIDC workload identity federation, this model continues: the proxy fetches stored credentials from the API and uses them to authenticate to the upstream backend. This is not a preferred path — stored credentials must be rotated manually, create a larger blast radius if compromised, and require the platform to hold long-lived secrets on behalf of providers. Data providers should be encouraged to configure OIDC trust relationships where their cloud supports it. ### Data Provider Hosting @@ -55,10 +55,6 @@ Data providers get: - **Exposure** — data is discoverable via the Source Cooperative platform and UI - **Outbound auth flexibility** — the provider's own cloud credentials (or OIDC trust relationship) are used for the proxy's outbound connection -### Unresolved: Provider Credential Operations - -For provider-hosted datasets where the provider's cloud does not support OIDC federation, the operational model for storing and rotating credentials securely is unresolved. Open questions include per-provider isolation and the trust boundary (what can Source Cooperative access in a provider's backend). See RFC-001 Open Question 5. - --- ## Consequences diff --git a/adrs/007-middleware.md b/adrs/007-middleware.md index 8f38182..9d01028 100644 --- a/adrs/007-middleware.md +++ b/adrs/007-middleware.md @@ -1,6 +1,6 @@ -# ADR-007: Middleware Architecture — Rate Limiting, Metering, and Billing Hooks +# ADR-007: Middleware Architecture -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §10 **Depends on:** ADR-005, ADR-008 @@ -9,14 +9,7 @@ ## Context -A general-purpose data proxy needs behaviours beyond authentication and object retrieval. Source Cooperative specifically requires: - -- **Rate limiting** — fine-grained and dynamic: different limits per bucket, per user, per organisation, or per role -- **Data metering** — tracking cumulative data transfer per identity or dataset, and enforcing access thresholds (e.g. denying access once a monthly quota is reached) -- **Usage tracking and billing hooks** — recording access events with enough fidelity to support downstream billing -- **Audit logging** — a complete, tamper-resistant record of who accessed what and when - -These concerns are cross-cutting: they apply to every request regardless of the specific storage backend or dataset, but their configuration and behaviour differ across deployments and use cases. Provider-hosted datasets may carry additional metering and quota requirements beyond what Source Cooperative's own datasets need. +A general-purpose data proxy needs behaviours beyond authentication and object retrieval — access logging, usage analytics, rate limiting, and cost attribution. These cross-cutting concerns are best implemented as composable middleware wrapping the core request handler. --- @@ -27,37 +20,29 @@ These concerns are cross-cutting: they apply to every request regardless of the Cross-cutting concerns are implemented as a **composable middleware stack** wrapping the core request handler. Each middleware layer: - Receives the request context (resolved identity, role, resource, action) and may modify or enrich it -- May short-circuit the request with a denial response (e.g. quota exceeded, rate limit hit) -- May record an event (e.g. to a metering store or audit log) +- May short-circuit the request with a denial response +- May record an event (e.g. to a log or metrics store) - Passes the request to the next layer if permitted ### Middleware as Rust Traits Middleware components are defined as Rust traits, making them first-class extension points. Source Cooperative ships standard implementations; operators can add their own without forking the core (see ADR-008). -### Configuration Scope - -The middleware stack is configured per-deployment and potentially per-dataset. A dataset with no billing requirements carries a lightweight stack; a provider-hosted dataset with metered access carries additional quota and event-recording middleware. - -### Standard Middleware (Planned) - -| Middleware | Behaviour | -|---|---| -| Rate limiter | Per-identity or per-bucket request rate enforcement, configurable limits | -| Quota enforcer | Cumulative data transfer tracking; deny on threshold exceeded | -| Usage recorder | Structured event emission per request (bytes transferred, identity, resource, latency) | -| Audit logger | Tamper-evident request log for compliance and forensics | -| Billing emitter | Usage event publication to a configurable billing backend | +> [!NOTE] +> **Future extension: Access logging and analytics.** The middleware architecture is designed to support structured request logging for usage analytics (which products and files are most popular, which accounts drive the most traffic) and cost attribution (distinguishing open data program buckets, Source Cooperative-owned buckets, and third-party provider-hosted buckets). The log backend, schema, storage, and analytics pipeline are significant decisions that will require a dedicated ADR. +> +> **Future extension: Rate limiting, quotas, and billing.** The following capabilities are deferred until there is concrete demand and a defined operational model: +> +> - **Rate limiting** — per-identity or per-product request rate enforcement +> - **Quota enforcement** — cumulative data transfer tracking with access thresholds +> - **Billing event emission** — publishing usage events to a billing backend +> - **Audit logging** — tamper-evident request logs for compliance +> +> Each of these fits the middleware trait interface and can be added without modifying the core proxy. ### Unresolved -The following details require further design: - - **Middleware trait interface** — the exact trait signature, including how request context is threaded and how middleware ordering is enforced -- **Per-dataset configuration** — how middleware stacks are expressed per-deployment and per-bucket -- **Event schema** — the structured format for usage recording and billing events -- **Event backend** — the initial target for event emission (Kinesis stream, S3/R2 log, webhook, or other). See RFC-001 Open Question 6 -- **Middleware ordering** — whether order-dependent behaviours are made explicit or left to the operator --- @@ -67,14 +52,13 @@ The following details require further design: - Cross-cutting concerns are composable and configurable, not hardcoded - New middleware can be contributed by the community without forking the core -- Per-dataset middleware stacks support the data provider hosting model +- Request logging provides the foundation for usage analysis and debugging from day one - The trait-based design enforces a consistent interface across all middleware **Costs / Risks** - Middleware on the hot path adds per-request overhead (mitigated by keeping middleware lightweight) -- Per-dataset middleware configuration adds operational complexity -- The middleware trait interface, event schema, and event backend are all unresolved — implementation cannot begin until these are defined +- The middleware trait interface is unresolved — implementation cannot begin until it is defined - Middleware ordering can introduce subtle bugs if order-dependent behaviours are not made explicit --- diff --git a/adrs/008-crate-architecture.md b/adrs/008-crate-architecture.md index 547dd3a..3701126 100644 --- a/adrs/008-crate-architecture.md +++ b/adrs/008-crate-architecture.md @@ -1,67 +1,47 @@ # ADR-008: Modular Crate Architecture and Community Reuse Model -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §11 +**Depends on:** ADR-003, ADR-004, ADR-005 --- ## Context -The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. It is difficult for external operators to deploy a version of the proxy for their own datasets, and equally difficult for contributors to improve the proxy in ways that are reusable outside Source Cooperative's deployment. +The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration is a thin layer on top. -The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration is a thin layer on top. - -This requires a clean separation between the general-purpose proxy framework and Source Cooperative-specific concerns. All Source Cooperative-specific behaviour must be expressed through the same trait interfaces that any other operator would use. +This work builds on [`multistore`](https://github.com/developmentseed/multistore), an existing effort to create a composable S3-compatible proxy in Rust. --- ## Decision -### Crate Structure +### Separation of Concerns via Crates -The proxy is structured as a set of Rust crates with well-defined trait boundaries between layers: +The proxy is structured as separate Rust crates to promote composability. Concerns like auth, authorization, storage backend resolution, and middleware are separated behind trait boundaries so that they can be developed, tested, and reused independently. -| Crate | Responsibility | SC-specific? | -|---|---|---| -| `proxy-core` | Request routing, SigV4 verification, session credential management, middleware stack execution | No | -| `proxy-auth` | STS exchange logic, OIDC issuer registry, JWT validation, SC Credential Token minting | No — issuer list is configuration | -| `proxy-authz` | Role resolution, per-request policy evaluation, policy store interface trait | No — store backend is pluggable | -| `proxy-storage` | `object_store`-based backend abstraction | No | -| `proxy-middleware` | Middleware trait definition and standard implementations (rate limiter, quota enforcer, usage recorder, etc.) | No | -| `proxy-workers` | Cloudflare Workers runtime adapter, WASM build target | No | -| `proxy-ecs` | Traditional server runtime adapter, Hyper/Tokio based | No | +The exact crate boundaries will emerge during implementation. The principle is separation of concerns, not a fixed crate map. Key areas of separation: -**Nothing Source Cooperative-specific lives in the core crates.** An operator building their own proxy instantiates the core with their own implementations of the configuration traits — providing their own backend resolver, role mapping, middleware stack — without forking any crate. +- **Request routing and SigV4 verification** — the core proxy mechanics +- **STS exchange and JWT validation** — inbound authentication (ADR-004) +- **Authorization and policy evaluation** — Role ceiling, account permissions (ADR-005) +- **Storage backend resolution** — mapping products to `object_store` configurations +- **Middleware** — request logging and future cross-cutting concerns (ADR-007) +- **Runtime adapters** — Cloudflare Workers (WASM) and traditional server (Hyper/Tokio) -Source Cooperative's own deployment is the reference implementation of this pattern. +**Nothing Source Cooperative-specific lives in the core crates.** All Source Cooperative-specific behaviour is expressed through the same trait interfaces that any other operator would use. ### Trait-Based Extension Points -Each layer defines traits that downstream operators implement: - -- **Auth:** issuer registry, claim condition evaluator, role mapper -- **Authz:** policy store, grant resolver -- **Storage:** backend resolver (maps bucket ID to `object_store` configuration) -- **Middleware:** middleware trait for custom cross-cutting concerns -- **Configuration:** configuration source trait for deployment-specific settings +Each area of concern defines traits that downstream operators implement. This allows operators to provide their own IdP configurations, policy store backends, storage resolvers, and middleware without forking the core. ### Publication and Licensing Core crates are intended for publication to `crates.io` under a permissive licence. -### Unresolved: Governance - -The following governance questions are unresolved: - -- Crate naming conventions -- Licence choice (MIT, Apache-2.0, or dual) -- API stability guarantees (semver policy, MSRV policy) -- Whether community-contributed crates live in the same repository, a separate organisation, or are fully external -- What "supported" means for community-contributed middleware or backends -- Contribution model and review process - -These are tracked in RFC-001 Open Question 8. +> [!NOTE] +> **TODO:** Finalise crate boundaries, naming, and licensing as the implementation progresses. --- @@ -70,17 +50,15 @@ These are tracked in RFC-001 Open Question 8. **Benefits** - Community members can build their own data proxies on the same foundation -- Contributions to the core (new middleware, new storage backends, auth improvements) benefit all deployments -- Source Cooperative's infrastructure demonstrates the framework's capabilities, aiding adoption +- Contributions to the core benefit all deployments - Clean trait boundaries prevent Source Cooperative-specific concerns from leaking into the framework - No forking required for custom deployments **Costs / Risks** - Maintaining trait stability across crate versions requires discipline and a clear semver policy -- Multiple crates increase the build and release coordination overhead -- Trait boundaries must be designed carefully upfront — changing a public trait is a breaking change -- Community governance and contribution model are unresolved +- Multiple crates increase build and release coordination overhead +- Trait boundaries must be designed carefully — changing a public trait is a breaking change --- diff --git a/adrs/009-configuration.md b/adrs/009-configuration.md index f0de2a8..724a64d 100644 --- a/adrs/009-configuration.md +++ b/adrs/009-configuration.md @@ -1,80 +1,107 @@ # ADR-009: Configuration Layer — Policy Store Implementation and Caching Strategy -**Status:** Pending +**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §12 -**Depends on:** ADR-005 +**Depends on:** ADR-004, ADR-005 --- ## Context -The authorization model (ADR-005) requires per-request lookups against a policy store for every non-public authenticated request. This is not optional — it is what enables dynamic permissions to reflect changes (new organisations, new dataset grants) in near real-time. Unlike a design that encodes permissions in the session token, this design explicitly trades token self-sufficiency for permission freshness. - -This constraint means the policy store is on the **hot path** of every authenticated request to a non-public resource. The question is not *whether* the proxy needs a policy store at request time, but *how* that access is implemented with acceptable latency and availability. +The authorization model (ADR-005) requires per-request lookups against a policy store for every non-public authenticated request. The STS exchange (ADR-004) requires lookups for Role definitions and IdP records during token issuance. Together, these create two distinct hot paths: per-S3-request authorization and per-session STS exchange. The policy store must serve both with acceptable latency and availability. --- ## Decision +### Managed Entities + +The policy store manages the following entities: + +| Entity | Owner | Written by | Read by | +|--------|-------|-----------|---------| +| **Product metadata** (public flag, backend config) | Platform | Next.js app or proxy (TBD) | Proxy (per-request) | +| **Account permission grants** | Platform | Next.js app or proxy (TBD) | Proxy (per-request) | +| **Role definitions** (identity constraints, permission statements) | Account owner | Management API (TBD) | Proxy (STS exchange) | +| **Platform IdP records** (issuer URL, well-known claims) | Platform operator | Configuration / deployment | Proxy (STS exchange) | + +The management API for Roles (`/api/accounts/{account_id}/roles`) is defined in ADR-004. Which component serves this API — the proxy, the Next.js application, or a dedicated service — is unresolved and tied to the implementation choice below. + ### Access Patterns -The proxy's configuration access has two distinct profiles: +The proxy's configuration access has three distinct profiles: -**High-frequency, latency-sensitive (per-request)** -- Bucket public flag lookup — `bucket_id -> {public, backend_config}` -- User grant lookup — `(user_id, bucket_id) -> {granted, prefix_restrictions}` -- User bucket list — `user_id -> [bucket_ids]` +**High-frequency, latency-sensitive (per S3 request)** +- Product public flag lookup — `product_id → {public, backend_config}` +- Account permission lookup — `(account_id, product_id) → {granted, prefix_restrictions}` +- Account's full product list — `account_id → [product_ids]` These must complete in single-digit milliseconds. In-process caching absorbs most of the load; the underlying lookup must be fast for cache misses. +**Medium-frequency, latency-sensitive (per STS exchange)** +- Role definition lookup — `(account_id, role_name) → Role` +- Platform IdP record lookup — `idp_id → IdP` + +STS exchanges happen once per session (not per request), but they are on the critical path for session establishment. Role and IdP lookups should complete in single-digit milliseconds with caching (30–60s TTL). + **Low-frequency, management (background)** -- Issuer JWKS refresh -- Role definition updates +- Issuer JWKS fetch and cache refresh (1hr TTL, stale-while-revalidate) - Provider credential rotation +- Role CRUD operations -These are not on the request hot path and can tolerate higher latency. - -### Implementation Options (Unresolved) +These tolerate higher latency and are not on any request hot path. -The implementation choice between the following options is unresolved and is the primary focus of RFC review: +### `backend_config` -**Option A — REST API intermediary with aggressive caching** +The product metadata record includes a `backend_config` that bridges authorization (ADR-005) and outbound storage (ADR-006): -The proxy calls the existing Source Cooperative API for configuration lookups, wrapped in multi-layer caching: in-process (per-isolate or per-container) with short TTL, backed by Workers KV or ElastiCache as a shared distributed cache tier. +```json +{ + "public": true, + "backend_config": { + "storage_url": "s3://provider-bucket/prefix/", + "credential_ref": "oidc-trust-provider-x", + "region": "us-west-2" + } +} +``` -*Advantages:* The Next.js application remains the schema owner; the proxy does not need direct database credentials; the API can enforce schema constraints. -*Risks:* The REST API is an availability dependency on the hot path. A cache miss on a cold Workers isolate hitting a degraded API directly impacts request latency. +The `credential_ref` identifies either an OIDC trust relationship or a stored credential secret (see ADR-006). The exact schema is defined by the `proxy-storage` crate's backend resolver trait (ADR-008). -**Option B — Direct DynamoDB access** +### Implementation Approach -The proxy connects directly to DynamoDB tables for configuration lookups. In-process caching still applies. +The proxy calls the existing Source Cooperative API for all lookups, wrapped in multi-layer caching: in-process (per-isolate) with short TTL, backed by Workers KV as a shared distributed cache tier. -*Advantages:* DynamoDB read latency (single-digit milliseconds) is appropriate for the hot path; eliminates availability coupling to the Next.js application. -*Risks:* Two systems (proxy and Next.js) accessing the same DynamoDB tables creates a schema governance problem. DynamoDB's schemaless nature means there is no DDL to enforce consistency — schema drift between consumers is possible and difficult to detect until runtime failure. +The Next.js application remains the sole schema owner. The proxy does not need direct database credentials. The API enforces schema constraints before data reaches the proxy. Management APIs for Roles and IdPs are served by the Next.js app. -**Option C — Proxy as data model authority** +The REST API is an availability dependency on the hot path for cache misses. In-process caching absorbs the majority of lookups. If profiling reveals the API as a latency bottleneck, direct DynamoDB access can be introduced for the highest-frequency lookups (product flags, account grants) while keeping management operations on the API. -The proxy owns and is the sole writer of the policy store schema. The Next.js application reads policy data through the proxy's API. +### Cache Strategy -*Advantages:* Single schema owner eliminates drift risk. -*Risks:* Expands the proxy's scope; requires refactoring the Next.js application; tightly couples front-end and proxy deployment cycles. +All lookups are cached in-process (per-isolate): -**Hybrid option** — Direct DynamoDB for high-frequency per-request lookups (bucket flags, user grants); REST API for management operations (issuer registration, role updates). +| Lookup | Cache Key | TTL | Notes | +|--------|-----------|-----|-------| +| Product public flag | `product_id` | 60–300s | Rarely changes | +| Account permission for product | `(account_id, product_id)` | 30–60s | Reflects grants, org membership | +| Account's full product list | `account_id` | 5–10s | Freshness-sensitive for UI | +| Role definition | `(account_id, role_name)` | 30–60s | Changes infrequently | +| JWKS | `issuer_url` | 1 hour | Stale-while-revalidate on failure | ### Workers Caching Stack For the Workers deployment: -- **In-process cache** — per-isolate, not shared across edge nodes, with TTLs from ADR-005 -- **Workers KV** — eventually consistent, globally distributed key-value store; serves as a shared cache tier that survives isolate recycling +- **In-process cache** — per-isolate, not shared across edge nodes, with TTLs above +- **Workers KV** — eventually consistent, globally distributed; available as a shared cache tier for policy data that survives isolate recycling For access control decisions, eventual consistency is generally acceptable — a grant created seconds ago but not yet visible in KV is a minor inconvenience, not a security failure. ### Unresolved -- The implementation choice between Options A, B, C, and the hybrid is the primary open question. See RFC-001 Open Question 2. -- The full caching stack for Workers (which lookups use Workers KV vs. in-process only, cache warming strategy for cold isolates) requires further design. +- The full caching stack for Workers (which lookups use Workers KV vs. in-process only, cache warming strategy for cold isolates). +- How the `_default` Role is provisioned — synthesized at runtime (recommended) or materialized in storage when accounts are created. --- @@ -86,20 +113,24 @@ For access control decisions, eventual consistency is generally acceptable — a - In-process caching absorbs the majority of lookup load - Workers KV provides a shared cache tier for the edge deployment - The configuration layer is behind a trait interface, allowing different implementations per deployment +- All managed entities are explicitly cataloged with ownership and access patterns **Costs / Risks** -- The policy store is a single point of failure for authenticated requests to non-public resources +- The REST API is an availability dependency for cache misses on the hot path - Cache misses on cold Workers isolates add latency to the first request -- Schema governance between the proxy and Next.js application is a risk regardless of implementation choice -- The implementation decision is blocked pending team discussion +- If the API proves to be a bottleneck, migrating high-frequency lookups to direct DynamoDB access will require schema governance discipline --- ## Alternatives Considered -**Encode permissions in the session token (no policy store on hot path)** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens after any permission change. See ADR-005. +**Encode permissions in the session token (no policy store on hot path)** — rejected. Freezes permissions at exchange time. Users would need to re-exchange tokens after any permission change. The current design embeds the Role ceiling in the token (avoiding one lookup) while keeping account permissions dynamic. See ADR-005. **Global strongly-consistent cache (e.g. Durable Objects)** — considered. Would eliminate eventual-consistency concerns. Rejected: Durable Objects are single-region, adding latency for global edge requests. Eventual consistency is acceptable for the access control use case. +**Direct DynamoDB access** — considered. Eliminates the REST API availability dependency and provides single-digit millisecond reads. Rejected as the initial approach: two systems (proxy and Next.js) accessing the same DynamoDB tables creates a schema governance problem that is difficult to detect until runtime failure. Can be introduced later for specific high-frequency lookups if profiling indicates the API is a bottleneck. + +**Proxy as data model authority** — considered. The proxy owns the policy store schema and the Next.js application reads through the proxy's API. Rejected: significantly expands the proxy's scope and tightly couples front-end and proxy deployment cycles. + **Push-based cache invalidation** — considered. The policy store pushes updates to Workers KV when grants change, rather than relying on TTL-based expiry. Worth exploring as an optimisation but adds operational complexity. Deferred. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 667a9cd..129a8cc 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -1,10 +1,9 @@ # RFC-001: Source Cooperative Data Proxy Re-Architecture -**Status:** Draft — Request for Comment +**Status:** Proposed — Request for Comment **Date:** 2026-03-14 **Authors:** @alukach **Replaces:** Current data proxy (ECS, Rust, long-lived credentials) -**Version:** 4 — see [Changelog](#changelog) for changes from v3 --- @@ -27,10 +26,10 @@ This is not a final specification. It is the basis for a conversation. Decisions 7. [Inbound Authentication](#7-inbound-authentication) 8. [Authorization](#8-authorization) 9. [Outbound Connectivity](#9-outbound-connectivity) -10. [Extensibility — Middleware and Metering](#10-extensibility--middleware-and-metering) +10. [Extensibility — Middleware](#10-extensibility--middleware) 11. [Modular Architecture and Community Reuse](#11-modular-architecture-and-community-reuse) 12. [Configuration and Data Layer](#12-configuration-and-data-layer) -13. [Open Questions](#13-open-questions) +13. [Future Work](#13-future-work) 14. [Decision Index](#14-decision-index) --- @@ -68,7 +67,7 @@ Several pressures have converged to make a re-architecture worthwhile rather tha - Short-lived, scoped credentials only — no long-lived static keys - A globally distributed deployment model that improves latency for international users without data replication costs - Support for any standards-compliant OIDC identity provider as an authentication source -- An authorization model expressive enough to support per-dataset, per-user, and per-organisation access control, rate limiting, metering, and billing hooks +- An authorization model expressive enough to support per-dataset, per-user, and per-organisation access control, with extensibility for future rate limiting, metering, and billing - A modular, trait-based Rust implementation that the community can depend on, extend, and contribute to - First-class support for data providers hosting their own datasets through Source Cooperative's access control and distribution layer - Support for all object storage backends provided by the `object_store` crate, including AWS S3, GCS, Azure Blob Storage, Cloudflare R2, and HTTP @@ -83,35 +82,28 @@ Several pressures have converged to make a re-architecture worthwhile rather tha ## 3. System Overview -The re-architected proxy consists of two complementary deployment targets that share a common Rust core: +The initial deployment is a single global proxy on Cloudflare Workers: -**Cloudflare Workers (primary, global)** -A WASM-compiled Rust service deployed across Cloudflare's global edge network. Handles the majority of traffic. Requests are routed through the Cloudflare network to upstream object storage, reducing latency for users far from origin storage regions. Suited for read-heavy, latency-sensitive workloads. +**Cloudflare Workers (global)** +A WASM-compiled Rust service deployed across Cloudflare's global edge network. Handles all traffic. Requests are routed through the Cloudflare network to upstream object storage, reducing latency for users far from origin storage regions. Suited for read-heavy, latency-sensitive workloads. -**Regional ECS deployments (secondary, on-demand)** -Traditional containerised Rust services deployed into specific cloud regions on demand. Intended for high-throughput, in-region workflows — for example, a Databricks cluster in `us-west-2` reading large volumes of data from an S3 bucket in the same region, where egress fees and network hops are a concern. Access to regional deployments should be restricted to in-region consumers (see [Open Questions](#13-open-questions)). +The Workers deployment hosts an STS endpoint at `/.sts` for credential exchange. -Each deployment target hosts its own STS endpoint at `/.sts`. The Workers deployment and all regional ECS deployments share the same signing key material, so session credentials issued by any target are valid across all targets. Whether credential interoperability across targets is a required property — versus each target issuing and verifying its own credentials independently — is an open question worth discussing. +```mermaid +flowchart TD + Caller["Caller / Workflow"] + Caller -- "1. Present OIDC token
to /.sts" --> Workers + + Workers["Cloudflare Workers
/.sts + S3 proxy"] + + Workers -- "2. Short-lived SigV4 creds
3. Authenticated S3 request" --> Storage + + Storage[("Upstream Object Storage
S3, GCS, R2, Azure, …")] ``` - ┌──────────────────────────────────────────┐ - │ Caller / Workflow │ - └──────────────┬───────────────────────────┘ - │ 1. Present OIDC token - │ to /.sts on either target - ▼ - ┌─────────────────────────┐ ┌──────────────────────────┐ - │ Cloudflare Workers │ │ Regional ECS Proxy │ - │ /.sts + proxy │ │ /.sts + proxy │ - └────────────┬────────────┘ └─────────────┬────────────┘ - │ 2. Short-lived SigV4 creds │ - │ (shared keys; interoperable)│ - │ 3. Authenticated S3 request │ - ▼ ▼ - ┌──────────────────────────────────────────────────────────────┐ - │ Upstream Object Storage (S3, GCS, R2, Azure, …) │ - └──────────────────────────────────────────────────────────────┘ -``` + +> [!NOTE] +> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — for example, a Databricks cluster in `us-west-2` reading large volumes from S3 in the same region — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. This introduces open questions around access restriction (ensuring regional proxies only serve in-region consumers), credential interoperability (whether credentials issued by Workers should be valid against regional proxies), and operational overhead (two deployment targets to build, test, and monitor). The shared Rust core and trait-based architecture (§11) are designed to support this without code divergence when the need arises. --- @@ -137,17 +129,17 @@ Incoming requests carry a SigV4 `Authorization` header. The proxy reconstructs t Session tokens are **stateless signed JWTs** using ES256 (asymmetric). The token payload encodes: `account_id`, `role_name`, `access_key_id`, the Role's permission statements (ceiling), `assumed_by` (original IdP subject for audit), `session_name`, and `exp`. The token encodes the Role's permission ceiling but **not** the account's full permission set — account permissions are dynamic and resolved per-request from the configuration layer (see §8). At request time, the effective permission is the intersection of (Role ceiling from token) and (account's actual permissions from policy store). -The proxy validates the JWT signature against its own public key, enforces `exp`, and checks the `jti` claim against a lightweight revocation deny-list (stored in Cloudflare KV with auto-expiring entries). No external database lookup is required to verify the token or reconstruct the signing key. +The proxy validates the JWT signature against its own public key and enforces `exp` and `nbf`. No external database lookup is required to verify the token or reconstruct the signing key — verification is fully stateless. -Session tokens support **lightweight revocation** via a deny-list of `jti` values. Account admins can revoke tokens via `POST /.sts/revoke-session-token`. The deny-list entries self-clean as tokens expire. +Short-lived credentials (15 min to 12 hours) bound the exposure window of a compromised token, eliminating the need for per-token revocation in the initial implementation (see ADR-001). --- ## 5. Runtime and Deployment -### Cloudflare Workers (Primary) +### Cloudflare Workers -The primary deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. This choice is motivated by several properties of the Workers platform: +The deployment target is Cloudflare Workers, with the proxy compiled to WebAssembly. This choice is motivated by several properties of the Workers platform: **Global distribution without operational overhead.**[^1] Workers deploy to Cloudflare's edge network (330+ locations worldwide) automatically. Requests are served from the location closest to the caller, and onward routing to upstream object storage traverses the Cloudflare backbone[^2] rather than the public internet. For users in Europe, Asia-Pacific, or South America accessing data stored in `us-west-2`, this meaningfully reduces latency without requiring us to replicate the underlying data. @@ -167,39 +159,61 @@ The primary deployment target is Cloudflare Workers, with the proxy compiled to [^4]: Cloudflare Workers docs: [How Workers Works](https://developers.cloudflare.com/workers/reference/how-workers-works/) — isolates start ~100× faster than a Node.js process in a container. [^5]: Cloudflare blog: [Eliminating Cold Starts 2: Shard and Conquer](https://blog.cloudflare.com/eliminating-cold-starts-2-shard-and-conquer/) — describes the consistent hashing technique that achieves a 99.99% warm request rate. -### Regional ECS Deployments (Secondary) +### Deployment Topology -Regional ECS deployments will be made available on demand for in-region, high-throughput workflows. The motivating case is a data pipeline running inside the same cloud region as its source data — for example, a Spark or Databricks job in `us-west-2` reading terabytes of data from S3 in the same region. In this scenario: +```mermaid +flowchart TD + subgraph Callers + CLI["CLI, CI/CD, SDKs, Browsers"] + end -- Egress fees are zero or near-zero when traffic stays within the region -- Network throughput is higher and latency is lower than routing through an edge node -- The Cloudflare Workers path adds unnecessary hops + subgraph Workers["Cloudflare Workers (Global)"] + W_STS["/.sts exchange"] + W_Proxy["S3 API Proxy
SigV4 verify, authz, route"] + W_Cache["In-process cache
(per-isolate, 60s TTL)"] + W_STS --> W_Cache + W_Proxy --> W_Cache + end -Regional deployments share the same Rust core as the Workers deployment. The deployment wrapper and runtime adapter differ; the proxy logic, auth, and authz layers are identical. + subgraph Shared["Shared Infrastructure"] + API["Source Cooperative API
Products, Grants, Roles, Platform IdPs"] + KMS["KMS
ES256 signing key
HMAC server secret"] + JWKS["IdP JWKS Endpoints
GitHub, GitLab, Ory, Vercel
(cached 1hr TTL)"] + Storage[("Upstream Object Storage
S3, GCS, Azure Blob, R2
via object_store")] + Platform["Source Cooperative Platform
Next.js (Vercel) + auth.source.coop (Ory)
Manages accounts, products, grants"] + end -**Access restriction.** Regional proxies should only be accessible to in-region consumers — allowing general internet access would route traffic that should go to the edge through a less optimal path and would undercut the cost model. + CLI --> Workers -> [!NOTE] -> **Open Question:** How do we restrict regional proxy access to in-region consumers? Options include: VPC-only exposure (no public endpoint), IP allowlisting by cloud provider IP ranges, requiring a region-specific audience claim in the STS exchange, or issuing regional-specific session credentials that are only accepted by the corresponding regional proxy. Each has tradeoffs around operational complexity and developer experience. See [Open Questions](#13-open-questions). + W_Cache -- cache miss --> API + W_Cache -- cache miss --> JWKS -### Deployment Topology + W_STS --> KMS -> [!NOTE] -> **TODO:** Diagram the full deployment topology, including how the STS exchange endpoint is deployed across both Workers and ECS targets, how JWKS endpoints are cached at the edge, and how regional ECS instances are provisioned and registered. + W_Proxy --> Storage +``` + +**Key properties:** + +- The Workers deployment hosts the `/.sts` endpoint and S3 proxy on Cloudflare's global edge network. +- Signing key material (ES256 private key, HMAC server secret) is managed via KMS. +- The proxy maintains per-isolate in-process caching. JWKS for platform IdPs is fetched from upstream issuers and cached with a 1-hour TTL (stale-while-revalidate on failure). +- The policy store is the Source Cooperative API, with in-process and distributed caching (Workers KV) to absorb hot-path load. +- The proxy authenticates to upstream storage via OIDC federation (preferred) or stored credentials (fallback). See §9. --- ## 6. Implementation Language — Rust -We are continuing with Rust as the implementation language for both the Workers and ECS deployments. The reasoning is as follows: +We are continuing with Rust as the implementation language. The reasoning is as follows: **WASM maturity.** Rust has the most mature and production-ready toolchain for compiling to WebAssembly of any systems language. The `worker-rs` crate provides idiomatic bindings to the Cloudflare Workers runtime. This is not a bet on an emerging capability — it is a well-trodden path. -**Performance.** For the regional ECS deployment, raw throughput matters. Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that may stream large objects with tight latency requirements. This was already proven by the current proxy. +**Performance.** Rust's zero-cost abstractions and lack of garbage collection pauses make it well-suited to a proxy that may stream large objects with tight latency requirements. This was already proven by the current proxy. **Type system and correctness.** The proxy handles authentication tokens, credential issuance, cryptographic signature verification, and access policy evaluation. Rust's type system — and in particular its trait system — makes it practical to encode invariants that would be runtime errors in other languages. This is increasingly valuable in a codebase where AI-assisted development is part of the workflow: a strong type system provides a correctness harness that catches generated code that compiles but violates domain constraints. -**Community familiarity.** The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known, but is not suitable for the WASM target and carries runtime overhead incompatible with the regional proxy's performance goals. Rust is the best fit given the actual pool of contributors. +**Community familiarity.** The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known, but is not suitable for the WASM target. Rust is the best fit given the actual pool of contributors. **Trait-based extensibility.** The Rust trait system is central to the modularity goals described in [Section 11](#11-modular-architecture-and-community-reuse). Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. This is difficult to achieve cleanly in languages without a comparable abstraction. @@ -211,7 +225,7 @@ We are continuing with Rust as the implementation language for both the Workers Source Cooperative will not issue or accept long-lived static credentials. All authentication flows terminate in short-lived SigV4 session credentials obtained through a token exchange. The exchange endpoint at `/.sts` acts as a Security Token Service that accepts JWTs from trusted OIDC identity providers and returns temporary credentials scoped by a caller-specified Role. -The key design choice is that **accounts own their Identity Providers and Roles**. Each account (Individual or Organization) can register IdPs and create Roles that scope access to their resources. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. +The key design choice is that **accounts own their Roles**. Each account (Individual or Organization) can create Roles that scope access to their resources, constrained to platform-registered IdPs. This mirrors how AWS IAM allows accounts to configure their own trust policies and roles. ### Identity Providers (IdPs) @@ -219,11 +233,12 @@ IdPs exist at two tiers: **Platform IdPs** are pre-configured by Source Cooperative operators — well-known OIDC issuers relevant to data engineering workflows (GitHub Actions, GitLab CI, `auth.source.coop`, Vercel, etc.). Platform operators can add new issuers over time without code changes. -**Account IdPs** are registered by account owners. These support corporate identity systems (Okta, Entra ID), self-hosted providers (Keycloak), or any issuer with a publicly reachable JWKS endpoint. Account IdP registration validates the issuer URL (HTTPS only, no private IPs, valid OIDC discovery document) and prevents collision with platform IdP issuer URLs. +> [!NOTE] +> **Future extension: Account-registered IdPs.** The initial implementation supports platform IdPs only. Account-level IdP registration (for corporate Okta, self-hosted Keycloak, etc.) can be added later without changing the Role schema or STS exchange flow. Platform operators can add new well-known issuers at any time without code changes. ### Roles -Roles belong to an account and define two things: **who can assume the Role** (identity constraints with per-IdP claim matching) and **what the Role's credentials can access** (permission statements). Roles are identified by URN: `source::{account_id}::role/{role_name}`. +Roles belong to an account and define two things: **who can assume the Role** (identity constraints binding platform IdPs with claim matching) and **what the Role's credentials can access** (permission statements). Roles are identified by URN: `sc::{account_id}::role/{role_name}`. A Role acts as a **ceiling** on the account's existing permissions. The credentials issued for a Role can never exceed what the account itself has access to. At request time, the effective permission is the intersection of the Role's permission statements and the account's actual permissions from the policy store. @@ -237,7 +252,7 @@ The STS endpoint accepts a signed JWT and a Role URN, validates the JWT against POST /.sts/assume-role-with-web-identity Content-Type: application/x-www-form-urlencoded -Action=AssumeRoleWithWebIdentity&RoleArn=source::my-org::role/publisher&WebIdentityToken=&RoleSessionName=my-job&DurationSeconds=3600 +Action=AssumeRoleWithWebIdentity&RoleArn=sc::my-org::role/publisher&WebIdentityToken=&RoleSessionName=my-job&DurationSeconds=3600 → XML response (AWS STS-compatible): { AccessKeyId, SecretAccessKey, SessionToken, Expiration } ``` @@ -259,25 +274,25 @@ The Role URN is required — the caller must know which Role to assume. This kee Workflows running in environments that provide ambient OIDC tokens can exchange them directly. These environments require no stored secrets: -| Platform | OIDC Issuer | Key Claims for Constraints | -| --------------------------- | ---------------------------------------- | ----------------------------------------------- | -| GitHub Actions | `token.actions.githubusercontent.com` | `repository`, `ref`, `environment` | -| GitLab CI/CD | `https://gitlab.com` | `project_path`, `ref_type`, `environment` | -| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | -| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | -| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | +| Platform | OIDC Issuer | Key Claims for Constraints | +| ----------------------- | --------------------------------------------- | ----------------------------------------------- | +| Source Cooperative Auth | `auth.source.coop` | `sub`, `groups` | +| GitHub Actions | `https://token.actions.githubusercontent.com` | `repository`, `ref`, `environment` | +| GitLab CI/CD | `https://gitlab.com` | `project_path`, `ref_type`, `environment` | +| Azure DevOps | `https://vstoken.dev.azure.com/` | project, pipeline, environment | +| HCP Terraform | `https://app.terraform.io` | `terraform_workspace_id`, `terraform_run_phase` | +| Vercel | `https://oidc.vercel.com/` | `owner`, `project`, `environment` | -This list is illustrative, not exhaustive. Accounts can register additional IdPs for any issuer with a publicly reachable JWKS endpoint. +This list is illustrative, not exhaustive. Platform operators can add new issuers over time without code changes. #### Source Cooperative Auth (`auth.source.coop`) Users authenticated interactively via Source Cooperative's Ory-based auth system present their Ory token to `/.sts` with their account's `_default` Role. This is appropriate for interactive local development, ad-hoc data access, and CLI tooling. -| Mechanism | Suited for | Stored secret? | Credential TTL | -| --------------------------------------- | ---------------------- | ------------------ | ------------------ | -| Ambient OIDC (GitHub, GitLab, etc.) | CI/CD, managed compute | No | 15 min – 12 hours | -| `auth.source.coop` / Ory token | Interactive local dev | No (session-based) | 15 min – 12 hours | -| Account-registered IdP (Okta, etc.) | Corporate workflows | No | 15 min – 12 hours | +| Mechanism | Suited for | Stored secret? | Credential TTL | +| ----------------------------------- | ---------------------- | ------------------ | ----------------- | +| Ambient OIDC (GitHub, GitLab, etc.) | CI/CD, managed compute | No | 15 min – 12 hours | +| `auth.source.coop` / Ory token | Interactive local dev | No (session-based) | 15 min – 12 hours | ### Next.js and Front-End Authentication @@ -293,10 +308,10 @@ Users authenticated interactively via Source Cooperative's Ory-based auth system ```ini [profile source-read] -credential_process = source-coop creds --role-arn source::my-org::role/reader +credential_process = source-coop creds --role-arn sc::my-org::role/reader [profile source-write] -credential_process = source-coop creds --role-arn source::my-org::role/publisher +credential_process = source-coop creds --role-arn sc::my-org::role/publisher ``` **GitHub Action:** `source-cooperative/configure-credentials` requests a GitHub OIDC token, calls the STS endpoint, and exports credentials as environment variables. @@ -341,7 +356,7 @@ The session token (§4) carries these fields relevant to authorization: Authorization proceeds in steps, with early exits to minimise lookups: -**Step 1 — Identify the caller.** No credentials → anonymous (read-only, public products only). `SCSTS` prefix → STS credential path. Other prefix → legacy API key lookup. +**Step 1 — Identify the caller.** No credentials → anonymous (read-only, public products only). `SCSTS` prefix → STS credential path. **Step 2 — Role action check (in-memory).** Check the SessionToken's embedded `permissions` array. If the requested action on the requested resource does not match any permission statement, deny immediately. This is a local check — no network call. @@ -356,7 +371,7 @@ Role permission statements use this structure: ```json { "actions": ["read", "write"], - "resources": ["source::my-org::product/climate-data/*"] + "resources": ["sc::my-org::product/climate-data/*"] } ``` @@ -378,13 +393,13 @@ After the Role ceiling check, public early exit, and account permission lookup: ### Cache Strategy -All policy store lookups are cached in-process (Workers: per-isolate; ECS: per-container): +All policy store lookups are cached in-process (per-isolate): -| Lookup | Cache Key | TTL | -| --------------------------------------- | -------------------------- | ------- | -| Product public flag | `product_id` | 60–300s | -| Account permission for product | `(account_id, product_id)` | 30–60s | -| Account's full product list | `account_id` | 5–10s | +| Lookup | Cache Key | TTL | +| ------------------------------ | -------------------------- | ------- | +| Product public flag | `product_id` | 60–300s | +| Account permission for product | `(account_id, product_id)` | 30–60s | +| Account's full product list | `account_id` | 5–10s | The short TTL on the full product list ensures that account permission changes are reflected within seconds. For Workers, cache is per-isolate and not shared across edge nodes. Workers KV is available as a shared tier if needed. @@ -438,44 +453,20 @@ This model offers data providers: --- -## 10. Extensibility — Middleware and Metering +## 10. Extensibility — Middleware ### Motivation -A general-purpose data proxy needs to support behaviours beyond simple authentication and object retrieval. Source Cooperative specifically requires: - -- **Rate limiting** that is fine-grained and dynamic: different limits per bucket, per user, per organisation, or per role — not a single global rate -- **Data metering:** tracking cumulative data transfer per identity or dataset, and enforcing access thresholds (e.g. denying access once a monthly quota is reached) -- **Usage tracking and billing hooks:** recording access events with enough fidelity to support downstream billing — charging data providers, charging consumers, or both -- **Audit logging:** a complete, tamper-resistant record of who accessed what and when - -These concerns are cross-cutting: they apply to every request regardless of the specific storage backend or dataset, but their configuration and behaviour differ across deployments and use cases. - -### Middleware Pattern - -These concerns are implemented as a **middleware stack** composing around the core request handler. Each middleware layer: +A general-purpose data proxy needs to support behaviours beyond simple authentication and object retrieval — access logging, usage analytics, rate limiting, and cost attribution. These cross-cutting concerns are implemented as a composable middleware stack wrapping the core request handler. -- Receives the request context (resolved identity, role, resource, action) and may modify or enrich it -- May short-circuit the request with a denial response (e.g. quota exceeded, rate limit hit) -- May record an event (e.g. to a metering store or audit log) -- Passes the request to the next layer if permitted +### Middleware Stack -The middleware stack is configured per-deployment and potentially per-dataset. A dataset with no billing requirements carries a lightweight stack; a provider-hosted dataset with metered access carries additional quota-check and event-recording middleware. - -Middleware components are defined as Rust traits, making them first-class extension points for community contributions. Source Cooperative ships a set of standard middleware implementations; operators can add their own without forking the core. - -### Standard Middleware (Planned) - -| Middleware | Behaviour | -| --------------- | -------------------------------------------------------------------------------------- | -| Rate limiter | Per-identity or per-bucket request rate enforcement, configurable limits | -| Quota enforcer | Cumulative data transfer tracking; deny on threshold exceeded | -| Usage recorder | Structured event emission per request (bytes transferred, identity, resource, latency) | -| Audit logger | Tamper-evident request log for compliance and forensics | -| Billing emitter | Usage event publication to a configurable billing backend | +Each middleware layer receives the request context (resolved identity, role, resource, action), may short-circuit with a denial, may record an event, and passes the request onward. Middleware components are defined as Rust traits (see §11), making them first-class extension points for community contributions. > [!NOTE] -> **TODO:** Define the middleware trait interface in detail. Specify how middleware configuration is expressed (per-deployment, per-bucket, per-role). Determine the event schema for usage recording and billing emission, and which backend systems are targeted initially (e.g. a Kinesis stream, a webhook, a DynamoDB table). Consider how middleware ordering is enforced and whether order-dependent behaviours are made explicit. +> **Future extension: Access logging and analytics.** The middleware architecture is designed to support structured request logging for usage analytics (which products and files are most popular, which accounts drive the most traffic) and cost attribution (distinguishing open data program buckets, Source Cooperative-owned buckets, and third-party provider-hosted buckets). The log backend, schema, storage, and analytics pipeline are significant decisions that will require a dedicated ADR. +> +> **Future extension: Rate limiting, quotas, and billing.** Rate limiting, quota enforcement, billing event emission, and audit logging are deferred until there is concrete demand. Each fits the middleware trait interface and can be added without modifying the core proxy. --- @@ -483,36 +474,18 @@ Middleware components are defined as Rust traits, making them first-class extens ### Motivation -The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. It is difficult for external operators to deploy a version of the proxy for their own datasets, and equally difficult for contributors to improve the proxy in ways that are reusable outside Source Cooperative's own deployment. +The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration of that framework is a thin layer on top. -The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration of that framework is a thin layer on top. +This work builds on [`multistore`](https://github.com/developmentseed/multistore), an early-stage effort to create a composable S3-compatible proxy in Rust. ### Design Approach -The core proxy is structured as a set of Rust crates with well-defined trait boundaries between layers: - -- **`proxy-core`** — request routing, SigV4 verification, session credential management, middleware stack execution. No Source Cooperative specifics. -- **`proxy-auth`** — STS exchange logic, OIDC issuer registry, JWT validation, SC Credential Token minting. Configurable via traits; the Source Cooperative issuer list is one configuration. -- **`proxy-authz`** — role resolution, per-request policy evaluation, policy store interface trait. The IAM-inspired evaluation logic is the default; the store backend is pluggable. -- **`proxy-storage`** — `object_store`-based backend abstraction. New backends are `object_store` implementations. -- **`proxy-middleware`** — middleware trait definition and standard implementations (rate limiter, quota enforcer, usage recorder, etc.) -- **`proxy-workers`** — Cloudflare Workers runtime adapter, WASM build target -- **`proxy-ecs`** — traditional server runtime adapter, Hyper/Tokio based - -An operator building their own proxy instantiates the core with their own implementations of the configuration traits — providing their own backend resolver, their own role mapping, their own middleware stack — without forking any of the above crates. +The proxy is structured as separate Rust crates to promote composability. Concerns like auth, authorization, storage backend resolution, and middleware are separated behind trait boundaries so that they can be developed, tested, and reused independently. The exact crate boundaries will emerge during implementation — the principle is separation of concerns, not a fixed crate map. -Source Cooperative's own deployment is the reference implementation of this pattern. - -### Community Contribution Model - -By publishing the core crates to `crates.io` under a permissive licence: - -- Community members can build their own data proxies on the same foundation -- Contributions to the core (new middleware, new storage backends, auth improvements) benefit all deployments -- Source Cooperative's own infrastructure becomes a demonstration of the framework's capabilities, which aids adoption +Source Cooperative-specific behaviour (IdP configuration, policy store implementation, deployment configuration) is expressed through the same trait interfaces that any other operator would use. Nothing Source Cooperative-specific lives in the core crates. > [!NOTE] -> **TODO:** Finalise crate naming, licensing, and governance model. Determine what "supported" means for community-contributed crates — whether they live in the same repository, a separate organisation, or are fully external. Define the public API stability guarantees for each crate. +> **TODO:** Finalise crate boundaries, naming, licensing, and governance model as the implementation progresses. --- @@ -520,7 +493,7 @@ By publishing the core crates to `crates.io` under a permissive licence: ### Constraint: The Policy Store Is on the Hot Path -The authorization model described in §8 requires the proxy to perform per-request lookups against a policy store for every non-public authenticated request. This is not optional — it is what enables dynamic permissions to reflect changes (new organisations, new dataset grants) in near real-time. Unlike the previous design (which attempted to encode permissions in the session token), this design explicitly trades token self-sufficiency for permission freshness. +The authorization model described in §8 requires the proxy to perform per-request lookups against a policy store for every non-public authenticated request. This is not optional — it is what enables dynamic permissions to reflect changes (new organisations, new dataset grants) in near real-time. Unlike a token-only model (which would freeze permissions at exchange time), this design explicitly trades token self-sufficiency for permission freshness. This constraint changes the framing of the configuration layer question. The question is no longer *whether* the proxy needs access to a policy store at request time — it does — but rather *how* that access is implemented with acceptable latency and availability. @@ -529,9 +502,9 @@ This constraint changes the framing of the configuration layer question. The que The proxy's access to the configuration layer has two distinct profiles: **High-frequency, latency-sensitive (per-request)** -- Bucket public flag lookup — `bucket_id → {public, backend_config}` -- User grant lookup — `(user_id, bucket_id) → {granted, prefix_restrictions}` -- User bucket list — `user_id → [bucket_ids]` +- Product public flag lookup — `product_id → {public, backend_config}` +- Account permission lookup — `(account_id, product_id) → {granted, prefix_restrictions}` +- Account product list — `account_id → [product_ids]` These must complete in single-digit milliseconds. In-process caching (§8) absorbs most of the load; the underlying lookup needs to be fast for cache misses. @@ -542,72 +515,34 @@ These must complete in single-digit milliseconds. In-process caching (§8) absor These are not on the request hot path and can tolerate higher latency. -### Implementation Options - -**Option A — REST API intermediary with aggressive caching** - -The proxy calls the existing Source Cooperative API for configuration lookups, but wraps every call in a multi-layer cache: in-process (per-isolate or per-container) with short TTL, backed by Workers KV or ElastiCache for a shared distributed cache tier. - -*Advantages:* The Next.js application remains the schema owner; the proxy does not need direct database credentials; the API can enforce schema constraints before data reaches the proxy. -*Risks:* The REST API is an availability dependency on the hot path. Even with caching, a cache miss on a cold Workers isolate hitting a degraded API will directly impact request latency. The API must be engineered for the proxy's read performance requirements, not just the application's UX needs. +### Implementation Approach -**Option B — Direct DynamoDB access** +The proxy calls the existing Source Cooperative API for all configuration lookups, wrapped in multi-layer caching: in-process (per-isolate) with short TTL, backed by Workers KV as a shared distributed cache tier. -The proxy connects directly to DynamoDB tables for configuration lookups, eliminating the REST API hop. In-process caching still applies. +The Next.js application remains the sole schema owner. The proxy does not need direct database credentials. The API enforces schema constraints before data reaches the proxy. This keeps the operational surface small and avoids the schema governance problems that arise when multiple systems access the same database directly. -*Advantages:* DynamoDB read latency (single-digit milliseconds) is appropriate for the hot path; eliminates the availability coupling to the Next.js application. -*Risks:* Two systems (proxy and Next.js application) accessing the same DynamoDB tables creates a schema governance problem. DynamoDB's schemaless nature means there is no DDL to enforce consistency — schema drift between consumers is possible and difficult to detect until it causes a runtime failure. - -**Option C — Proxy as data model authority** - -The proxy owns and is the sole writer of the policy store schema. The Next.js application reads policy data through the proxy's API rather than directly from DynamoDB. - -*Advantages:* Single schema owner eliminates drift risk; the proxy API becomes the contract. -*Risks:* Significantly expands the proxy's scope of responsibility; requires refactoring the Next.js application's direct DynamoDB access; tightly couples front-end and proxy deployment cycles. - -> [!NOTE] -> **Open Question:** Options A and B are the most practical near-term choices. Option A is lower risk but introduces an availability dependency on the hot path; Option B is faster and more resilient but creates a schema governance problem with DynamoDB. A hybrid is possible — direct DynamoDB for the high-frequency per-request lookups (bucket flags, user grants), REST API for management operations (issuer registration, role updates) — but adds operational complexity. What is the team's appetite for managing DynamoDB schema consistency without a schema enforcement layer? +The REST API is an availability dependency on the hot path for cache misses. In-process caching absorbs the majority of lookups, so the API is only hit on cold starts and TTL expiry. If profiling reveals the API as a latency bottleneck, direct DynamoDB access can be introduced for the highest-frequency lookups (product flags, account grants) while keeping management operations on the API. ### Configuration in Workers Cloudflare Workers have access to Workers KV (eventually consistent, globally distributed key-value store) and Durable Objects (strongly consistent, single-region). For the Workers deployment, the in-process cache described in §8 is per-isolate and not shared. Workers KV provides a shared distributed cache tier for policy data that survives isolate recycling and is consistent across edge nodes within its eventual-consistency window (typically seconds). -For access control decisions, eventual consistency is generally acceptable — a grant that was created 2 seconds ago but not yet visible in KV is a minor inconvenience, not a security failure. Revocation is a different matter (not in scope for this iteration per §4). +For access control decisions, eventual consistency is generally acceptable — a grant that was created 2 seconds ago but not yet visible in KV is a minor inconvenience, not a security failure. > [!NOTE] > **TODO:** Design the full caching stack for the Workers deployment: in-process TTLs, Workers KV usage for shared policy cache, and the propagation path from the authoritative policy store to the edge. Specify which lookups require Workers KV vs. in-process only, and define the cache warming strategy for cold isolate starts. --- -## 13. Open Questions - -The following questions are unresolved and are the primary focus of this RFC review. Answers will be captured in the ADRs listed in [Section 14](#14-decision-index). - -### Resolved - -7. ~~**Policy language and grant schema.**~~ **Resolved in ADR-004 and ADR-005 (v4).** Permission statements use `read`/`write` actions with URN resource patterns supporting product-level and prefix-level granularity. Grants are additive (allow-only). The Role acts as a ceiling on the account's existing permissions. Organisation membership is modelled through account-level grants in the policy store. - -### Open - -1. **Regional proxy access restriction.** How do we ensure regional ECS proxy deployments are only accessible to in-region consumers? VPC-only endpoints, IP range allowlisting, region-scoped session credentials, and audience claims are candidate mechanisms. What are the operational tradeoffs? - -2. **Configuration store implementation.** Given that the policy store is definitively on the hot path, should we use the REST API with aggressive caching (lower risk, availability dependency) or direct DynamoDB access (faster, schema governance risk)? A hybrid approach is also possible. What schema enforcement mechanisms can mitigate the DynamoDB drift risk in Option B? - -3. **STS endpoint deployment.** Both Workers and ECS targets host `/.sts`. How is the JWKS cache managed consistently across the edge? Is there a risk of JWKS cache inconsistency between edge nodes causing transient validation failures? - -4. **Credential interoperability across targets.** Is it a requirement that session credentials issued by the Workers STS be valid against a regional ECS proxy and vice versa? If so, what operational procedures govern shared key material and rotation across deployment targets? - -5. **Outbound OIDC vs. stored secrets — default and fallback.** For provider-hosted datasets where the provider's cloud does not support OIDC federation, what is the operational model for storing and rotating credentials securely? - -6. **Middleware event backend.** What is the initial target for usage recording and billing event emission? A push-based stream (Kinesis, Pub/Sub), a pull-based log (S3, R2), a webhook, or something else? - -8. **Crate governance.** What is the publication, maintenance, and contribution model for the core crates on `crates.io`? - -9. **Organisation permission model.** How does organisation membership translate to account-level grants? When a user joins an org, do they automatically inherit grants for all org products, or must grants be assigned explicitly per product? How do org admin vs. member tiers affect default grants? +## 13. Future Work -10. **HMAC server secret rotation.** The HMAC derivation (`SecretAccessKey = HMAC-SHA256(server_secret, AccessKeyId)`) creates a dependency on a shared symmetric key. What is the rotation procedure? Can multiple active HMAC secrets coexist during rotation (try new key first, fall back to old)? +The following areas are acknowledged as future work. Each is designed to be additive — the initial architecture supports them without breaking changes. Details are captured as future extension notes in the relevant ADRs. -11. **Multipart upload credential expiry.** Large uploads spanning many parts over slow connections may outlast the STS credential TTL. The upload cannot be resumed with new credentials (SigV4 ties the signature to the specific AccessKeyId). Should the system support credential refresh mid-upload, or is documentation (use longer TTLs for large uploads) sufficient? +- **Access logging, analytics, and cost attribution** — the middleware architecture (§10) supports structured request logging, but the log backend, schema, storage, and analytics pipeline warrant a dedicated ADR (ADR-007) +- **Rate limiting, quota enforcement, and billing** — deferred until there is concrete demand (ADR-007) +- **Regional ECS deployments** — for high-throughput, in-region workflows where edge routing adds unnecessary hops and egress fees (ADR-002) +- **Account-registered IdPs** — allowing accounts to register corporate OIDC issuers (Okta, Keycloak) without operator intervention (ADR-004) +- **Permanent API keys** — long-lived credentials exchangeable for temporary STS credentials, for environments without OIDC support (ADR-001, ADR-005) --- @@ -615,17 +550,17 @@ The following questions are unresolved and are the primary focus of this RFC rev The following ADRs will be produced as decisions are ratified through this RFC process. Links will be added as documents are published. -| ADR | Decision | Status | -| ------- | ---------------------------------------------------------------------- | ------- | -| ADR-001 | S3 API compatibility and temporary-credentials-only model | Draft | -| ADR-002 | Runtime: Cloudflare Workers + regional ECS strategy | Pending | -| ADR-003 | Rust as implementation language | Pending | -| ADR-004 | Inbound authentication — OIDC federation, account-owned IdPs and Roles | Draft | -| ADR-005 | Authorization model — Role ceiling with dynamic account permission resolution | Draft | -| ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | Pending | -| ADR-007 | Middleware architecture — rate limiting, metering, billing hooks | Pending | -| ADR-008 | Modular crate architecture and community reuse model | Pending | -| ADR-009 | Configuration layer — policy store implementation and caching strategy | Pending | +| ADR | Decision | +| ------- | ------------------------------------------------------------------------------- | +| ADR-001 | S3 API compatibility and temporary-credentials-only model | +| ADR-002 | Runtime: Cloudflare Workers | +| ADR-003 | Rust as implementation language | +| ADR-004 | Inbound authentication — OIDC federation, platform IdPs and account-owned Roles | +| ADR-005 | Authorization model — Role ceiling with dynamic account permission resolution | +| ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | +| ADR-007 | Middleware architecture | +| ADR-008 | Modular crate architecture and community reuse model | +| ADR-009 | Configuration layer — policy store implementation and caching strategy | --- From 2fd17f2387bd65bcc041f69893d58d63c0c9742f Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 23 Mar 2026 16:08:08 -0700 Subject: [PATCH 05/17] Rm status field --- adrs/001-s3-credentials.md | 1 - adrs/002-runtimes.md | 1 - adrs/003-rust.md | 1 - adrs/004-sts.md | 1 - adrs/005-authorization.md | 1 - adrs/006-outbound-storage.md | 1 - adrs/007-middleware.md | 1 - adrs/008-crate-architecture.md | 1 - adrs/009-configuration.md | 1 - adrs/rfc-001.md | 1 - 10 files changed, 10 deletions(-) diff --git a/adrs/001-s3-credentials.md b/adrs/001-s3-credentials.md index 25a4831..8e3ec56 100644 --- a/adrs/001-s3-credentials.md +++ b/adrs/001-s3-credentials.md @@ -1,6 +1,5 @@ # ADR-001: S3 API Compatibility and Temporary-Credentials-Only Credential Model -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §4 diff --git a/adrs/002-runtimes.md b/adrs/002-runtimes.md index 220893c..f940063 100644 --- a/adrs/002-runtimes.md +++ b/adrs/002-runtimes.md @@ -1,6 +1,5 @@ # ADR-002: Runtime — Cloudflare Workers -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §5 diff --git a/adrs/003-rust.md b/adrs/003-rust.md index fbf029a..a3367d7 100644 --- a/adrs/003-rust.md +++ b/adrs/003-rust.md @@ -1,6 +1,5 @@ # ADR-003: Rust as Implementation Language -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §6 diff --git a/adrs/004-sts.md b/adrs/004-sts.md index 2419cbd..f9a768c 100644 --- a/adrs/004-sts.md +++ b/adrs/004-sts.md @@ -1,6 +1,5 @@ # ADR-004: Inbound Authentication — OIDC Federation, Platform IdPs, and Role-Based STS Exchange -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §7 **Depends on:** ADR-001 diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md index 974cfa2..f19cf99 100644 --- a/adrs/005-authorization.md +++ b/adrs/005-authorization.md @@ -1,6 +1,5 @@ # ADR-005: Authorization Model — Role Ceiling with Dynamic Account Permission Resolution -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §8 **Depends on:** ADR-001, ADR-004 diff --git a/adrs/006-outbound-storage.md b/adrs/006-outbound-storage.md index e6c2ef9..e02ce00 100644 --- a/adrs/006-outbound-storage.md +++ b/adrs/006-outbound-storage.md @@ -1,6 +1,5 @@ # ADR-006: Outbound Connectivity — OIDC Issuer Model and `object_store` Adoption -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §9 **Depends on:** ADR-002 diff --git a/adrs/007-middleware.md b/adrs/007-middleware.md index 9d01028..7652b5d 100644 --- a/adrs/007-middleware.md +++ b/adrs/007-middleware.md @@ -1,6 +1,5 @@ # ADR-007: Middleware Architecture -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §10 **Depends on:** ADR-005, ADR-008 diff --git a/adrs/008-crate-architecture.md b/adrs/008-crate-architecture.md index 3701126..5cf12b0 100644 --- a/adrs/008-crate-architecture.md +++ b/adrs/008-crate-architecture.md @@ -1,6 +1,5 @@ # ADR-008: Modular Crate Architecture and Community Reuse Model -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §11 **Depends on:** ADR-003, ADR-004, ADR-005 diff --git a/adrs/009-configuration.md b/adrs/009-configuration.md index 724a64d..1ae4825 100644 --- a/adrs/009-configuration.md +++ b/adrs/009-configuration.md @@ -1,6 +1,5 @@ # ADR-009: Configuration Layer — Policy Store Implementation and Caching Strategy -**Status:** Proposed **Date:** 2026-03-14 **RFC:** RFC-001 §12 **Depends on:** ADR-004, ADR-005 diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 129a8cc..4861463 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -1,6 +1,5 @@ # RFC-001: Source Cooperative Data Proxy Re-Architecture -**Status:** Proposed — Request for Comment **Date:** 2026-03-14 **Authors:** @alukach **Replaces:** Current data proxy (ECS, Rust, long-lived credentials) From dc7c40c3f81c9025901f55b993c451633986b1f8 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 23 Mar 2026 19:54:47 -0700 Subject: [PATCH 06/17] chore: rm plan --- .../2026-03-22-sts-token-exchange-design.md | 451 ------------------ 1 file changed, 451 deletions(-) delete mode 100644 docs/plans/2026-03-22-sts-token-exchange-design.md diff --git a/docs/plans/2026-03-22-sts-token-exchange-design.md b/docs/plans/2026-03-22-sts-token-exchange-design.md deleted file mode 100644 index f7ccd88..0000000 --- a/docs/plans/2026-03-22-sts-token-exchange-design.md +++ /dev/null @@ -1,451 +0,0 @@ -# STS Token Exchange Design - -## Problem - -Source Cooperative needs federated identity and fine-grained access control for its S3-compatible data proxy. Users and automated systems (CI/CD pipelines, data workflows) must obtain temporary credentials scoped to specific permissions without long-lived API keys. - -## Core Entities - -### Identity Providers (IdPs) - -IdPs exist at two tiers: - -**Platform IdPs** are pre-configured by Source Cooperative operators: -- `auth.source.coop` (Source Cooperative's Ory-based OIDC) -- `https://token.actions.githubusercontent.com` (GitHub Actions) -- `https://gitlab.com` (GitLab CI) -- Additional issuers added by operators over time - -Platform IdPs define: -```json -{ - "id": "github-actions", - "issuer_url": "https://token.actions.githubusercontent.com", - "display_name": "GitHub Actions", - "well_known_claims": ["repository", "repository_owner", "ref", "environment", "job_workflow_ref"], - "audience_hint": "https://data.source.coop" -} -``` - -The `well_known_claims` field provides documentation and UI hints for users creating Role bindings. The `audience_hint` is the recommended `aud` value callers should configure when requesting OIDC tokens. - -**Account IdPs** are registered by account owners (Individual or Organization): -- Must use HTTPS -- Issuer URL must not collide with any platform IdP (exact match after canonicalization) -- Must not duplicate another IdP on the same account -- Must serve a valid OIDC discovery document at `/.well-known/openid-configuration` -- Resolved IP must not be private, loopback, or link-local (SSRF protection) -- Fetch timeout: 3 seconds, response body limit: 256KB - -Account IdP stored record: -```json -{ - "id": "uuid", - "account_id": "my-org", - "issuer_url": "https://corp.okta.com/oauth2/default", - "display_name": "Our Corporate Okta", - "created_at": "2025-03-22T...", - "created_by": "user-id" -} -``` - -No JWKS is stored at registration time. JWKS is fetched and cached at STS exchange time from the OIDC discovery document's `jwks_uri`. - -**IdP deletion** is blocked if any Role references the IdP. The account must first remove the IdP binding from all Roles, then delete the IdP. - -### Roles - -Roles belong to an account (Individual or Organization), identified by URN: `source::{account_id}::role/{role_name}`. - -Each Role contains: -- **Identity constraints** — which IdPs can assume this Role, with what claim requirements -- **Permission statements** — what the Role's credentials can access -- **`max_session_duration`** — ceiling on credential TTL (default 1 hour, max 12 hours) - -Role schema: -```json -{ - "name": "github-publisher", - "display_name": "GitHub CI Publisher", - "max_session_duration": 3600, - "identity_constraints": [ - { - "idp": "github-actions", - "audience": "https://data.source.coop", - "claim_constraints": [ - {"claim": "repository", "operator": "equals", "value": "my-org/my-repo"}, - {"claim": "ref", "operator": "starts_with", "value": "refs/heads/"} - ] - }, - { - "idp": "uuid-of-account-idp", - "audience": "https://data.source.coop", - "claim_constraints": [ - {"claim": "sub", "operator": "equals", "value": "service-account-42"} - ] - } - ], - "permissions": [ - { - "actions": ["read", "write"], - "resources": ["source::my-org::product/climate-data/*"] - }, - { - "actions": ["read"], - "resources": ["source::my-org::product/reference-data/*"] - } - ] -} -``` - -**Built-in default Role:** `source::{account_id}::role/_default` -- Undeletable, always exists for every account -- Constrained to the `auth.source.coop` IdP only -- Permissions: `{"actions": ["read", "write"], "resources": ["*"]}` — unlimited ceiling, account's actual permissions are the sole constraint -- Account owners can add claim constraints but cannot change the IdP binding - -### Claim Constraint Language - -Three operators, deliberately minimal: - -| Operator | Behavior | Example | -|----------|----------|---------| -| `equals` | Exact string match | `repository` equals `my-org/my-repo` | -| `starts_with` | String prefix match | `ref` starts_with `refs/heads/` | -| `glob` | Wildcard: `*` (any chars), `?` (single char) | `repository` glob `my-org/*` | - -Rules: -- All claim values coerced to strings before comparison. Arrays and objects evaluate to false. -- All constraints within a single IdP binding are ANDed. -- Multiple IdP bindings on a Role are ORed. -- Missing claims evaluate to false (fail-closed). -- Top-level claims only — no nested path traversal. -- No regex. Glob is the most expressive operator. - -### Permission Statements - -Resource pattern format: -``` -* → all resources (unlimited ceiling) -source::{account_id}::product/* → all of an account's products -source::{account_id}::product/{product_name} → entire product -source::{account_id}::product/{product_name}/* → entire product (equivalent) -source::{account_id}::product/{product_name}/{prefix}/* → prefix-scoped -source::{account_id}::product/{product_name}/{key} → single object -``` - -Rules: -- Resource patterns can reference any account's products. A Role can delegate access to products the account has access to, even if owned by another account or org. -- `*` as the entire resource value means "all resources" — no ceiling; the account's actual permissions are the sole constraint. -- `*` at the end of a pattern matches any suffix (prefix matching). `*` is valid only as the final character or as the entire value. -- Actions are `read` and `write`. `read` maps to `GetObject`, `HeadObject`, `ListObjects`. `write` maps to `PutObject`, `DeleteObject`, and multipart operations. -- Permission statements are additive (allow-only). No explicit denies. -- Roles act as a ceiling — they can never exceed the account's own permissions. The request-time intersection `(Role permissions) ∩ (account's actual permissions)` is the sole enforcement mechanism. - -### Role Validation at Creation - -1. `name` must match `[a-z0-9][a-z0-9-]{0,62}` (lowercase, hyphens, max 63 chars) -2. Each IdP reference must exist (platform IdP by well-known ID, account IdP by UUID) -3. `max_session_duration` between 900 and 43200 seconds (15 min to 12 hours) -4. At least one identity constraint required -5. At least one permission statement required -6. Maximum 10 IdP bindings per Role, 20 claim constraints per binding, 50 permission statements per Role - -## STS Token Exchange - -### Endpoint - -``` -POST /.sts/assume-role-with-web-identity -``` - -Dot-prefixed account names are reserved as invalid, preventing routing conflicts. - -**Request format:** `application/x-www-form-urlencoded`, AWS STS-compatible: -``` -Action=AssumeRoleWithWebIdentity -&WebIdentityToken= -&RoleArn=source::my-org::role/github-publisher -&RoleSessionName=my-ci-job-42 -&DurationSeconds=3600 -``` - -**Response format:** XML, AWS STS-compatible: -```xml - - - - SCSTS... - derived-secret - eyJ... - 2025-03-22T13:00:00Z - - - source::my-org::role/github-publisher - SCSTS...:my-ci-job-42 - - - -``` - -This format enables `boto3.client('sts', endpoint_url='https://data.source.coop/.sts').assume_role_with_web_identity(...)` with a custom endpoint URL. The `RoleArn` parameter accepts Source Cooperative URN format — the AWS SDK passes the string through without client-side validation. - -### Exchange Flow - -1. Parse `RoleArn` → extract `account_id` and `role_name` -2. Load Role definition from policy store (cached, 30–60s TTL) -3. Extract `iss` from JWT (without verification) -4. Match `iss` against Role's allowed IdPs — reject immediately if no match -5. Fetch JWKS from the matched IdP (cached, 1hr TTL, 3s timeout, stale-while-revalidate on failure) -6. Verify JWT signature, `exp`, `nbf` (60s clock skew tolerance), and `aud` -7. Evaluate claim constraints for the matched IdP binding -8. Validate `DurationSeconds` ≤ Role's `max_session_duration` -9. Generate credentials and return response - -### Credential Issuance - -**AccessKeyId:** Random unique identifier prefixed `SCSTS` to distinguish from permanent keys. - -**SecretAccessKey:** Derived deterministically: `HMAC-SHA256(server_secret, AccessKeyId)`. Never stored, never transmitted in the SessionToken. The server reconstructs it on each request by re-deriving from the AccessKeyId. - -**SessionToken:** A signed JWT (ES256, asymmetric) containing: -```json -{ - "jti": "", - "sub": "source::my-org::role/github-publisher", - "account_id": "my-org", - "role_name": "github-publisher", - "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", - "assumed_by_issuer": "https://token.actions.githubusercontent.com", - "session_name": "my-ci-job-42", - "access_key_id": "SCSTS...", - "permissions": [ - {"actions": ["read", "write"], "resources": ["source::my-org::product/climate-data/*"]} - ], - "iat": 1711100000, - "exp": 1711103600, - "aud": "data.source.coop", - "kid": "" -} -``` - -Key properties: -- **Permissions embedded in the token** avoid per-request policy store lookups for Role ceiling evaluation. The account's underlying permissions are still checked dynamically. -- **`assumed_by`** preserves the original IdP subject for audit trails. -- **`access_key_id`** included so the server can derive the SecretAccessKey via HMAC. -- **SecretAccessKey is NOT in the token.** - -### Signing Key Management - -- Asymmetric signing: ES256 (ECDSA P-256) -- Private key stored in KMS, used only for token issuance -- Public key served at a JWKS endpoint for verification -- `kid` in JWT header supports key rotation: sign new tokens with new key, accept old key until all tokens signed with it expire -- Rotation procedure: generate new key → sign with new key → old key valid for `max_session_duration` after last use - -### Revocation - -- Lightweight deny-list of revoked `jti` values stored in Cloudflare KV (or equivalent) -- TTL on each KV entry matches the token's remaining lifetime (self-cleaning) -- Checked on every authenticated request (KV lookup ~1ms at edge) -- `POST /.sts/revoke-session-token` endpoint, callable by account admins - -### JWKS Caching - -- Cache key: canonicalized issuer URL -- TTL: 1 hour -- Stale-while-revalidate: if JWKS fetch fails and a cached copy exists, serve stale for up to 24 hours (with warning logged) -- If no cache and fetch fails → return `IDPCommunicationError` -- Max response body: 256KB - -## Request-Time Authorization - -### Step 1: Identify the Caller - -- **No credentials** → anonymous -- **Permanent API key** (non-`SCSTS` prefix) → legacy API key lookup via Source API -- **STS credentials** (`SCSTS` prefix) → derive SecretAccessKey via HMAC, verify SigV4, decode SessionToken JWT - -### Step 2: Role Action Check (in-memory) - -For anonymous callers, only read actions are permitted. - -For STS callers, the SessionToken's embedded permissions define the ceiling. If the requested action is not covered, deny immediately. - -### Step 3: Resource Resolution - -Map the S3 request to a Source Cooperative resource: -- Bucket name → `account_id/product_name` -- Object key → path within the product - -### Step 4: Public Resource Early Exit (cached, 60–300s TTL) - -For read requests on public products (`data_mode: open`), permit immediately. No further lookups. This is the fast path for the majority of traffic. - -### Step 5: Account Permission Lookup (cached, 30–60s TTL) - -For non-public resources or write operations: -1. Fetch the account's permissions from the policy store -2. Compute: `(Role ceiling permissions) ∩ (account's actual permissions)` -3. If the intersection includes the requested action on the requested resource → permit -4. Otherwise → deny - -### Step 6: Prefix Enforcement - -If the Role's permission statement includes a prefix constraint, verify the object key falls within that prefix. - -### Authorization Truth Table - -| Caller | Resource | Account has access? | Role permits? | Result | -|--------|----------|-------------------|--------------|--------| -| Anonymous | Public product | N/A | N/A | **Allow** (read only) | -| Anonymous | Private product | N/A | N/A | **Deny** | -| STS | Public product, read | N/A | Yes | **Allow** | -| STS | Public product, write | Yes | Yes | **Allow** | -| STS | Private product | Yes | Yes | **Allow** | -| STS | Private product | Yes | No (ceiling) | **Deny** | -| STS | Private product | No | Yes | **Deny** | - -## STS Error Responses - -Errors use AWS STS XML format for SDK compatibility: - -```xml - - - InvalidIdentityToken - JWT claim 'repository' value 'my-org/wrong-repo' does not match - constraint 'my-org/correct-repo' on role 'github-publisher' - - -``` - -| Condition | Error Code | HTTP Status | -|-----------|-----------|-------------| -| Role URN malformed | `MalformedPolicyDocument` | 400 | -| Role not found | `InvalidParameterValue` | 400 | -| JWT malformed or unparseable | `InvalidIdentityToken` | 400 | -| JWT issuer matches no IdP on Role | `InvalidIdentityToken` | 400 | -| JWT signature verification failed | `InvalidIdentityToken` | 400 | -| JWT expired | `ExpiredTokenException` | 400 | -| JWT `aud` mismatch | `InvalidIdentityToken` | 400 | -| Claim constraints not satisfied | `InvalidIdentityToken` | 400 | -| IdP JWKS endpoint unreachable | `IDPCommunicationError` | 400 | -| `DurationSeconds` exceeds max | `ValidationError` | 400 | - -Error messages include enough detail for callers to diagnose problems (which claim failed, expected vs. actual values). The Role definition is not secret — the account admin created it. - -## Observability - -### STS Exchange Logging - -Every exchange (success or failure) emits a structured log entry: -```json -{ - "event": "sts_exchange", - "timestamp": "2025-03-22T12:00:00Z", - "account_id": "my-org", - "role_name": "github-publisher", - "role_urn": "source::my-org::role/github-publisher", - "idp_issuer": "https://token.actions.githubusercontent.com", - "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", - "session_name": "my-ci-job-42", - "result": "success", - "access_key_id": "SCSTS...", - "duration_seconds": 3600, - "client_ip": "...", - "failure_reason": null -} -``` - -### Request-Time Access Logging - -Every S3 request with STS credentials logs: -```json -{ - "event": "s3_request", - "timestamp": "...", - "account_id": "my-org", - "role_name": "github-publisher", - "session_name": "my-ci-job-42", - "assumed_by": "repo:my-org/my-repo:ref:refs/heads/main", - "action": "PutObject", - "resource": "source::my-org::product/climate-data/2025/data.parquet", - "result": "allow", - "client_ip": "..." -} -``` - -## Migration & Client Tooling - -### Coexistence with Current Auth - -The STS system runs alongside existing permanent API key auth: -- AccessKeyId prefix `SCSTS` routes to STS credential path; all other prefixes route to legacy API key lookup -- No changes to existing API key auth -- No forced migration timeline — STS is additive - -### Anonymous Access - -Public data (`data_mode: open`) remains accessible with zero authentication. No STS exchange, no credentials. `--no-sign-request` keeps working. This is a hard requirement. - -### Client Tooling (v1) - -**GitHub Action: `source-cooperative/configure-credentials`** -```yaml -permissions: - id-token: write -steps: - - uses: source-cooperative/configure-credentials@v1 - with: - role-urn: source::my-org::role/github-publisher - - run: aws s3 cp data.parquet s3://data.source.coop/my-org/my-product/ -``` - -Requests a GitHub OIDC token with audience `https://data.source.coop`, calls the STS endpoint, and exports `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`. All downstream tools pick these up automatically. - -**source-coop CLI (`source-cooperative/source-coop-cli`)** - -The existing CLI already supports `source login` and `credential_process` integration. The STS work extends it so that login calls the new STS endpoint. - -Users configure `~/.aws/config` with role-specific profiles: -```ini -[profile source-read] -credential_process = source-coop creds --role-arn source::my-org::role/reader - -[profile source-write] -credential_process = source-coop creds --role-arn source::my-org::role/publisher -``` - -The `source-coop creds` command: checks for cached valid credentials → if expired, triggers OIDC login (or uses cached auth.source.coop token) → calls STS endpoint → returns credentials in `credential_process` JSON format. - -Users select the profile per tool: -```bash -aws s3 ls s3://data.source.coop/ --profile source-read -``` - -**Direct SDK usage** for programmatic integrations: -```python -sts = boto3.client('sts', endpoint_url='https://data.source.coop/.sts') -creds = sts.assume_role_with_web_identity( - RoleArn='source::my-org::role/github-publisher', - WebIdentityToken=token, - RoleSessionName='my-job' -) -``` - -## Role Management API - -``` -POST /api/accounts/{account_id}/idps -GET /api/accounts/{account_id}/idps -DELETE /api/accounts/{account_id}/idps/{idp_id} - -POST /api/accounts/{account_id}/roles -GET /api/accounts/{account_id}/roles -GET /api/accounts/{account_id}/roles/{role_name} -PUT /api/accounts/{account_id}/roles/{role_name} -DELETE /api/accounts/{account_id}/roles/{role_name} -``` - -Only account owners and org admins can manage IdPs and Roles. The `_default` Role cannot be deleted. From 990e6f932758334190ab770ca0dda1a32b8bb477 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 10:53:38 -0700 Subject: [PATCH 07/17] Apply suggestion from @tylere --- adrs/rfc-001.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 4861463..9fbeeeb 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -86,7 +86,7 @@ The initial deployment is a single global proxy on Cloudflare Workers: **Cloudflare Workers (global)** A WASM-compiled Rust service deployed across Cloudflare's global edge network. Handles all traffic. Requests are routed through the Cloudflare network to upstream object storage, reducing latency for users far from origin storage regions. Suited for read-heavy, latency-sensitive workloads. -The Workers deployment hosts an STS endpoint at `/.sts` for credential exchange. +The Workers deployment hosts an endpoint that provides short-lived credentials via token exchange, available at `/.sts`. ```mermaid flowchart TD From 4c96a5aceecf20f9132ea12bfd208134768eaeeb Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 16:29:52 -0700 Subject: [PATCH 08/17] Reduce focus on decisions related to multistore design --- adrs/002-runtimes.md | 2 +- adrs/003-rust.md | 2 +- adrs/007-middleware.md | 71 ---------------------------------- adrs/008-crate-architecture.md | 70 --------------------------------- adrs/009-configuration.md | 4 +- adrs/rfc-001.md | 64 +++++++++++------------------- 6 files changed, 27 insertions(+), 186 deletions(-) delete mode 100644 adrs/007-middleware.md delete mode 100644 adrs/008-crate-architecture.md diff --git a/adrs/002-runtimes.md b/adrs/002-runtimes.md index f940063..c815fd4 100644 --- a/adrs/002-runtimes.md +++ b/adrs/002-runtimes.md @@ -29,7 +29,7 @@ Key properties: - **WASM compatibility.** Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`). > [!NOTE] -> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — data pipelines (Spark, Databricks, Polars) running in the same cloud region as the source data — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. The trait-based architecture (ADR-008) is designed to support additional runtime targets without code divergence. This can be pursued when there is demonstrated demand. +> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — data pipelines (Spark, Databricks, Polars) running in the same cloud region as the source data — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. Multistore's trait-based architecture is designed to support additional runtime targets without code divergence. This can be pursued when there is demonstrated demand. --- diff --git a/adrs/003-rust.md b/adrs/003-rust.md index a3367d7..78b518a 100644 --- a/adrs/003-rust.md +++ b/adrs/003-rust.md @@ -25,7 +25,7 @@ We continue with **Rust** as the implementation language. **Type system and correctness.** The proxy handles authentication tokens, credential issuance, cryptographic signature verification, and access policy evaluation. Rust's type system — and in particular its trait system — encodes invariants that would be runtime errors in other languages. This is increasingly valuable in a codebase where AI-assisted development is part of the workflow: a strong type system provides a correctness harness that catches generated code that compiles but violates domain constraints. -**Trait-based extensibility.** The Rust trait system is central to the modularity goals described in ADR-008. Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. +**Trait-based extensibility.** The Rust trait system is central to multistore's modularity goals. Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. **Community familiarity.** Rust is the best fit given the actual pool of contributors. diff --git a/adrs/007-middleware.md b/adrs/007-middleware.md deleted file mode 100644 index 7652b5d..0000000 --- a/adrs/007-middleware.md +++ /dev/null @@ -1,71 +0,0 @@ -# ADR-007: Middleware Architecture - -**Date:** 2026-03-14 -**RFC:** RFC-001 §10 -**Depends on:** ADR-005, ADR-008 - ---- - -## Context - -A general-purpose data proxy needs behaviours beyond authentication and object retrieval — access logging, usage analytics, rate limiting, and cost attribution. These cross-cutting concerns are best implemented as composable middleware wrapping the core request handler. - ---- - -## Decision - -### Middleware Stack Pattern - -Cross-cutting concerns are implemented as a **composable middleware stack** wrapping the core request handler. Each middleware layer: - -- Receives the request context (resolved identity, role, resource, action) and may modify or enrich it -- May short-circuit the request with a denial response -- May record an event (e.g. to a log or metrics store) -- Passes the request to the next layer if permitted - -### Middleware as Rust Traits - -Middleware components are defined as Rust traits, making them first-class extension points. Source Cooperative ships standard implementations; operators can add their own without forking the core (see ADR-008). - -> [!NOTE] -> **Future extension: Access logging and analytics.** The middleware architecture is designed to support structured request logging for usage analytics (which products and files are most popular, which accounts drive the most traffic) and cost attribution (distinguishing open data program buckets, Source Cooperative-owned buckets, and third-party provider-hosted buckets). The log backend, schema, storage, and analytics pipeline are significant decisions that will require a dedicated ADR. -> -> **Future extension: Rate limiting, quotas, and billing.** The following capabilities are deferred until there is concrete demand and a defined operational model: -> -> - **Rate limiting** — per-identity or per-product request rate enforcement -> - **Quota enforcement** — cumulative data transfer tracking with access thresholds -> - **Billing event emission** — publishing usage events to a billing backend -> - **Audit logging** — tamper-evident request logs for compliance -> -> Each of these fits the middleware trait interface and can be added without modifying the core proxy. - -### Unresolved - -- **Middleware trait interface** — the exact trait signature, including how request context is threaded and how middleware ordering is enforced - ---- - -## Consequences - -**Benefits** - -- Cross-cutting concerns are composable and configurable, not hardcoded -- New middleware can be contributed by the community without forking the core -- Request logging provides the foundation for usage analysis and debugging from day one -- The trait-based design enforces a consistent interface across all middleware - -**Costs / Risks** - -- Middleware on the hot path adds per-request overhead (mitigated by keeping middleware lightweight) -- The middleware trait interface is unresolved — implementation cannot begin until it is defined -- Middleware ordering can introduce subtle bugs if order-dependent behaviours are not made explicit - ---- - -## Alternatives Considered - -**Hardcoded middleware in the core proxy** — rejected. Does not support the modularity and community-reuse goals. Provider-hosted datasets need different middleware stacks than Source Cooperative's own datasets. - -**Sidecar/external middleware (e.g. Envoy filters)** — considered. Offloads middleware to a separate process. Rejected: does not work in the Workers deployment target (no sidecar model), and adds latency from inter-process communication. - -**Plugin system (dynamic loading)** — considered. Would allow middleware to be loaded at runtime. Rejected for the Workers target: WASM does not support dynamic library loading. Rust traits with static dispatch are the natural fit for both targets. diff --git a/adrs/008-crate-architecture.md b/adrs/008-crate-architecture.md deleted file mode 100644 index 5cf12b0..0000000 --- a/adrs/008-crate-architecture.md +++ /dev/null @@ -1,70 +0,0 @@ -# ADR-008: Modular Crate Architecture and Community Reuse Model - -**Date:** 2026-03-14 -**RFC:** RFC-001 §11 -**Depends on:** ADR-003, ADR-004, ADR-005 - ---- - -## Context - -The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration is a thin layer on top. - -This work builds on [`multistore`](https://github.com/developmentseed/multistore), an existing effort to create a composable S3-compatible proxy in Rust. - ---- - -## Decision - -### Separation of Concerns via Crates - -The proxy is structured as separate Rust crates to promote composability. Concerns like auth, authorization, storage backend resolution, and middleware are separated behind trait boundaries so that they can be developed, tested, and reused independently. - -The exact crate boundaries will emerge during implementation. The principle is separation of concerns, not a fixed crate map. Key areas of separation: - -- **Request routing and SigV4 verification** — the core proxy mechanics -- **STS exchange and JWT validation** — inbound authentication (ADR-004) -- **Authorization and policy evaluation** — Role ceiling, account permissions (ADR-005) -- **Storage backend resolution** — mapping products to `object_store` configurations -- **Middleware** — request logging and future cross-cutting concerns (ADR-007) -- **Runtime adapters** — Cloudflare Workers (WASM) and traditional server (Hyper/Tokio) - -**Nothing Source Cooperative-specific lives in the core crates.** All Source Cooperative-specific behaviour is expressed through the same trait interfaces that any other operator would use. - -### Trait-Based Extension Points - -Each area of concern defines traits that downstream operators implement. This allows operators to provide their own IdP configurations, policy store backends, storage resolvers, and middleware without forking the core. - -### Publication and Licensing - -Core crates are intended for publication to `crates.io` under a permissive licence. - -> [!NOTE] -> **TODO:** Finalise crate boundaries, naming, and licensing as the implementation progresses. - ---- - -## Consequences - -**Benefits** - -- Community members can build their own data proxies on the same foundation -- Contributions to the core benefit all deployments -- Clean trait boundaries prevent Source Cooperative-specific concerns from leaking into the framework -- No forking required for custom deployments - -**Costs / Risks** - -- Maintaining trait stability across crate versions requires discipline and a clear semver policy -- Multiple crates increase build and release coordination overhead -- Trait boundaries must be designed carefully — changing a public trait is a breaking change - ---- - -## Alternatives Considered - -**Monolithic crate with feature flags** — considered. Simpler build, but makes it difficult for operators to depend on only the parts they need. Feature flags don't provide the same clean separation as separate crates with trait boundaries. - -**Fork-based customisation** — rejected. The current model. Leads to divergent forks that don't benefit from upstream improvements. Trait-based extension is strictly preferable. - -**Configuration file instead of trait implementations** — considered. Would allow operators to customise behaviour via YAML/TOML without writing Rust. Rejected: insufficient expressiveness for the range of customisation needed (custom auth flows, custom middleware, custom policy stores). Configuration can complement traits but cannot replace them. diff --git a/adrs/009-configuration.md b/adrs/009-configuration.md index 1ae4825..8f16860 100644 --- a/adrs/009-configuration.md +++ b/adrs/009-configuration.md @@ -1,7 +1,7 @@ # ADR-009: Configuration Layer — Policy Store Implementation and Caching Strategy **Date:** 2026-03-14 -**RFC:** RFC-001 §12 +**RFC:** RFC-001 §11 **Depends on:** ADR-004, ADR-005 --- @@ -66,7 +66,7 @@ The product metadata record includes a `backend_config` that bridges authorizati } ``` -The `credential_ref` identifies either an OIDC trust relationship or a stored credential secret (see ADR-006). The exact schema is defined by the `proxy-storage` crate's backend resolver trait (ADR-008). +The `credential_ref` identifies either an OIDC trust relationship or a stored credential secret (see ADR-006). The exact schema is defined by multistore's storage backend resolver trait. ### Implementation Approach diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 9fbeeeb..afba771 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -10,7 +10,7 @@ This document describes the proposed re-architecture of the Source Cooperative data proxy. It is written to give contributors, maintainers, and stakeholders a complete picture of the system design — including the reasoning behind each major choice — and to invite critique before decisions are ratified. Open questions are explicitly called out throughout. -This is not a final specification. It is the basis for a conversation. Decisions that emerge from review will be captured in individual Architecture Decision Records (ADRs) referenced in the [Decision Index](#decision-index). +This is not a final specification. It is the basis for a conversation. Decisions that emerge from review will be captured in individual Architecture Decision Records (ADRs) referenced in the [Decision Index](#13-decision-index). --- @@ -25,11 +25,10 @@ This is not a final specification. It is the basis for a conversation. Decisions 7. [Inbound Authentication](#7-inbound-authentication) 8. [Authorization](#8-authorization) 9. [Outbound Connectivity](#9-outbound-connectivity) -10. [Extensibility — Middleware](#10-extensibility--middleware) -11. [Modular Architecture and Community Reuse](#11-modular-architecture-and-community-reuse) -12. [Configuration and Data Layer](#12-configuration-and-data-layer) -13. [Future Work](#13-future-work) -14. [Decision Index](#14-decision-index) +10. [Multistore — Open-Source Proxy Framework](#10-multistore--open-source-proxy-framework) +11. [Configuration and Data Layer](#11-configuration-and-data-layer) +12. [Future Work](#12-future-work) +13. [Decision Index](#13-decision-index) --- @@ -214,7 +213,7 @@ We are continuing with Rust as the implementation language. The reasoning is as **Community familiarity.** The Source Cooperative contributor community has more Rust experience than Go, and more Go experience than C++. Python is more widely known, but is not suitable for the WASM target. Rust is the best fit given the actual pool of contributors. -**Trait-based extensibility.** The Rust trait system is central to the modularity goals described in [Section 11](#11-modular-architecture-and-community-reuse). Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. This is difficult to achieve cleanly in languages without a comparable abstraction. +**Trait-based extensibility.** The Rust trait system is central to the modularity goals described in [Section 10](#10-multistore--open-source-proxy-framework). Traits allow the core proxy framework to define interfaces — for auth, authz, storage backend, middleware, configuration — that downstream users implement without forking the core. This is difficult to achieve cleanly in languages without a comparable abstraction. --- @@ -452,43 +451,28 @@ This model offers data providers: --- -## 10. Extensibility — Middleware +## 10. Multistore — Open-Source Proxy Framework -### Motivation +The proxy is built on [`multistore`](https://github.com/developmentseed/multistore), an open-source composable S3-compatible proxy framework in Rust. Multistore provides the core proxy mechanics — request routing, SigV4 verification, storage backend resolution via `object_store`, and extension points for authentication, authorization, and middleware. -A general-purpose data proxy needs to support behaviours beyond simple authentication and object retrieval — access logging, usage analytics, rate limiting, and cost attribution. These cross-cutting concerns are implemented as a composable middleware stack wrapping the core request handler. +Source Cooperative's deployment is one instance of multistore. The proxy implements multistore's extension points for its specific needs: OIDC-based STS exchange (§7), Role-ceiling authorization with dynamic account permissions (§8), and configuration backed by the Source Cooperative API (§11). Nothing Source Cooperative-specific lives in the multistore codebase. -### Middleware Stack +Developing multistore as a separate open-source project provides several benefits: -Each middleware layer receives the request context (resolved identity, role, resource, action), may short-circuit with a denial, may record an event, and passes the request onward. Middleware components are defined as Rust traits (see §11), making them first-class extension points for community contributions. +- **Community reuse** — other organisations can build their own data proxies on the same foundation without forking +- **Clear boundaries** — Source Cooperative-specific behaviour is isolated from the general-purpose framework, keeping both codebases focused +- **Shared investment** — contributions to multistore (new storage backends, performance improvements, bug fixes) benefit all deployments, including Source Cooperative's -> [!NOTE] -> **Future extension: Access logging and analytics.** The middleware architecture is designed to support structured request logging for usage analytics (which products and files are most popular, which accounts drive the most traffic) and cost attribution (distinguishing open data program buckets, Source Cooperative-owned buckets, and third-party provider-hosted buckets). The log backend, schema, storage, and analytics pipeline are significant decisions that will require a dedicated ADR. -> -> **Future extension: Rate limiting, quotas, and billing.** Rate limiting, quota enforcement, billing event emission, and audit logging are deferred until there is concrete demand. Each fits the middleware trait interface and can be added without modifying the core proxy. - ---- - -## 11. Modular Architecture and Community Reuse - -### Motivation - -The current proxy is tightly coupled to Source Cooperative's specific data model, backend configuration, and operational context. The re-architecture treats Source Cooperative's deployment as *one instance* of a general-purpose S3-compatible data proxy framework. The framework is the primary artefact; Source Cooperative's configuration of that framework is a thin layer on top. - -This work builds on [`multistore`](https://github.com/developmentseed/multistore), an early-stage effort to create a composable S3-compatible proxy in Rust. - -### Design Approach - -The proxy is structured as separate Rust crates to promote composability. Concerns like auth, authorization, storage backend resolution, and middleware are separated behind trait boundaries so that they can be developed, tested, and reused independently. The exact crate boundaries will emerge during implementation — the principle is separation of concerns, not a fixed crate map. - -Source Cooperative-specific behaviour (IdP configuration, policy store implementation, deployment configuration) is expressed through the same trait interfaces that any other operator would use. Nothing Source Cooperative-specific lives in the core crates. +Multistore's internal architecture — crate boundaries, trait design, middleware composition — is governed by the multistore project and is outside the scope of this RFC. > [!NOTE] -> **TODO:** Finalise crate boundaries, naming, licensing, and governance model as the implementation progresses. +> **Future extension: Access logging and analytics.** Multistore's middleware architecture supports structured request logging for usage analytics (which products and files are most popular, which accounts drive the most traffic) and cost attribution (distinguishing open data program buckets, Source Cooperative-owned buckets, and third-party provider-hosted buckets). The log backend, schema, storage, and analytics pipeline are significant decisions that will require a dedicated ADR. +> +> **Future extension: Rate limiting, quotas, and billing.** Rate limiting, quota enforcement, billing event emission, and audit logging are deferred until there is concrete demand. --- -## 12. Configuration and Data Layer +## 11. Configuration and Data Layer ### Constraint: The Policy Store Is on the Hot Path @@ -533,19 +517,19 @@ For access control decisions, eventual consistency is generally acceptable — a --- -## 13. Future Work +## 12. Future Work The following areas are acknowledged as future work. Each is designed to be additive — the initial architecture supports them without breaking changes. Details are captured as future extension notes in the relevant ADRs. -- **Access logging, analytics, and cost attribution** — the middleware architecture (§10) supports structured request logging, but the log backend, schema, storage, and analytics pipeline warrant a dedicated ADR (ADR-007) -- **Rate limiting, quota enforcement, and billing** — deferred until there is concrete demand (ADR-007) +- **Access logging, analytics, and cost attribution** — multistore's middleware architecture supports structured request logging, but the log backend, schema, storage, and analytics pipeline warrant a dedicated ADR +- **Rate limiting, quota enforcement, and billing** — deferred until there is concrete demand - **Regional ECS deployments** — for high-throughput, in-region workflows where edge routing adds unnecessary hops and egress fees (ADR-002) - **Account-registered IdPs** — allowing accounts to register corporate OIDC issuers (Okta, Keycloak) without operator intervention (ADR-004) - **Permanent API keys** — long-lived credentials exchangeable for temporary STS credentials, for environments without OIDC support (ADR-001, ADR-005) --- -## 14. Decision Index +## 13. Decision Index The following ADRs will be produced as decisions are ratified through this RFC process. Links will be added as documents are published. @@ -557,10 +541,8 @@ The following ADRs will be produced as decisions are ratified through this RFC p | ADR-004 | Inbound authentication — OIDC federation, platform IdPs and account-owned Roles | | ADR-005 | Authorization model — Role ceiling with dynamic account permission resolution | | ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | -| ADR-007 | Middleware architecture | -| ADR-008 | Modular crate architecture and community reuse model | | ADR-009 | Configuration layer — policy store implementation and caching strategy | --- -*This RFC is open for comment. Please raise questions, objections, and alternative proposals against the open questions in Section 13 and the design decisions throughout. The goal is collective understanding and buy-in before implementation begins.* +*This RFC is open for comment. Please raise questions, objections, and alternative proposals against the design decisions throughout. The goal is collective understanding and buy-in before implementation begins.* From cc83d23535bd9cf3fa4ec0296bfc285bb67ce30b Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 16:34:25 -0700 Subject: [PATCH 09/17] Reduce focused on "trait-based" architecture --- adrs/002-runtimes.md | 4 ++-- adrs/rfc-001.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/adrs/002-runtimes.md b/adrs/002-runtimes.md index c815fd4..cb1cb59 100644 --- a/adrs/002-runtimes.md +++ b/adrs/002-runtimes.md @@ -29,7 +29,7 @@ Key properties: - **WASM compatibility.** Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`). > [!NOTE] -> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — data pipelines (Spark, Databricks, Polars) running in the same cloud region as the source data — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. Multistore's trait-based architecture is designed to support additional runtime targets without code divergence. This can be pursued when there is demonstrated demand. +> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — data pipelines (Spark, Databricks, Polars) running in the same cloud region as the source data — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. Multistore is designed to support additional runtime targets without code divergence. This can be pursued when there is demonstrated demand. --- @@ -55,6 +55,6 @@ Key properties: **CDN in front of ECS** — considered. A traditional CDN (CloudFront, Cloudflare) can cache static responses, but the proxy's responses are not cacheable in a general-purpose CDN sense (authenticated, per-user). The proxy logic must run at the edge, not just caching. -**Workers + Regional ECS** — considered as the initial deployment. Simpler to start with Workers only and add regional ECS deployments when demand materialises. The trait-based architecture supports this without requiring upfront investment in a second deployment target. +**Workers + Regional ECS** — considered as the initial deployment. Simpler to start with Workers only and add regional ECS deployments when demand materialises. Multistore's architecture supports this without requiring upfront investment in a second deployment target. **Lambda@Edge / CloudFront Functions** — considered. More limited runtime environment, tighter CPU and memory constraints, and AWS-specific. Workers offer a more capable and provider-neutral edge compute model. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index afba771..e1ab6ea 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -66,7 +66,7 @@ Several pressures have converged to make a re-architecture worthwhile rather tha - A globally distributed deployment model that improves latency for international users without data replication costs - Support for any standards-compliant OIDC identity provider as an authentication source - An authorization model expressive enough to support per-dataset, per-user, and per-organisation access control, with extensibility for future rate limiting, metering, and billing -- A modular, trait-based Rust implementation that the community can depend on, extend, and contribute to +- A modular Rust implementation, built on multistore, that the community can depend on, extend, and contribute to - First-class support for data providers hosting their own datasets through Source Cooperative's access control and distribution layer - Support for all object storage backends provided by the `object_store` crate, including AWS S3, GCS, Azure Blob Storage, Cloudflare R2, and HTTP @@ -101,7 +101,7 @@ flowchart TD ``` > [!NOTE] -> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — for example, a Databricks cluster in `us-west-2` reading large volumes from S3 in the same region — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. This introduces open questions around access restriction (ensuring regional proxies only serve in-region consumers), credential interoperability (whether credentials issued by Workers should be valid against regional proxies), and operational overhead (two deployment targets to build, test, and monitor). The shared Rust core and trait-based architecture (§11) are designed to support this without code divergence when the need arises. +> **Future extension: Regional ECS deployments.** For high-throughput, in-region workflows — for example, a Databricks cluster in `us-west-2` reading large volumes from S3 in the same region — routing through an edge node adds unnecessary hops and egress fees. Regional ECS deployments running the same Rust core could serve these workloads with lower latency and zero cross-region egress. This introduces open questions around access restriction (ensuring regional proxies only serve in-region consumers), credential interoperability (whether credentials issued by Workers should be valid against regional proxies), and operational overhead (two deployment targets to build, test, and monitor). The shared Rust core is designed to support this without code divergence when the need arises. --- From b38f7e0d91cabb8725f9f04b0a60035a9be07a24 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 17:20:10 -0700 Subject: [PATCH 10/17] Add detail on zero-copy streaming --- adrs/rfc-001.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index e1ab6ea..10050f7 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -151,6 +151,8 @@ The deployment target is Cloudflare Workers, with the proxy compiled to WebAssem **WASM compatibility.** The Workers runtime supports WASM natively. Rust compiles to WASM with mature toolchain support (`wasm-pack`, `worker-rs`), making the Workers target a natural fit for a Rust-implemented proxy. +**Zero-copy streaming.** Multistore passes request and response bodies through in unaltered form, allowing each runtime to specify its own body type. For the Workers runtime, this means bodies remain as `web_sys::ReadableStream` — data flows between the caller and upstream storage without being converted into Rust types or passing through the WASM CPU boundary. This is critical for staying within Workers' CPU time limits when proxying large objects. + [^1]: Cloudflare operates a global anycast network across 330+ cities. See [Cloudflare Network](https://www.cloudflare.com/network/). [^2]: Cloudflare's backbone is a private network interconnecting its data centres, used to route traffic between edge nodes and origin servers without traversing the public internet. See [Cloudflare Network Interconnect](https://www.cloudflare.com/network-interconnect/). [^3]: Cloudflare blog: [Eliminating Cold Starts with Cloudflare Workers](https://blog.cloudflare.com/eliminating-cold-starts-with-cloudflare-workers/) — describes the original TLS handshake pre-warming technique. From 70fb0ecce16133e0626d9cfb9452b8690807e1e7 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 17:26:29 -0700 Subject: [PATCH 11/17] fix numbering --- adrs/{009-configuration.md => 007-configuration.md} | 2 +- adrs/rfc-001.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename adrs/{009-configuration.md => 007-configuration.md} (99%) diff --git a/adrs/009-configuration.md b/adrs/007-configuration.md similarity index 99% rename from adrs/009-configuration.md rename to adrs/007-configuration.md index 8f16860..dcd2343 100644 --- a/adrs/009-configuration.md +++ b/adrs/007-configuration.md @@ -1,4 +1,4 @@ -# ADR-009: Configuration Layer — Policy Store Implementation and Caching Strategy +# ADR-007: Configuration Layer — Policy Store Implementation and Caching Strategy **Date:** 2026-03-14 **RFC:** RFC-001 §11 diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 10050f7..9d9f9b1 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -543,7 +543,7 @@ The following ADRs will be produced as decisions are ratified through this RFC p | ADR-004 | Inbound authentication — OIDC federation, platform IdPs and account-owned Roles | | ADR-005 | Authorization model — Role ceiling with dynamic account permission resolution | | ADR-006 | Outbound connectivity — OIDC issuer model, `object_store` adoption | -| ADR-009 | Configuration layer — policy store implementation and caching strategy | +| ADR-007 | Configuration layer — policy store implementation and caching strategy | --- From 0497d7fe72c891bfd06ebfb6e8e3ac90d1f7c89d Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 22:16:37 -0700 Subject: [PATCH 12/17] Add details on direct federation and brokered role access for data providers --- adrs/006-outbound-storage.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/adrs/006-outbound-storage.md b/adrs/006-outbound-storage.md index e02ce00..5059128 100644 --- a/adrs/006-outbound-storage.md +++ b/adrs/006-outbound-storage.md @@ -38,6 +38,18 @@ This model means: - The trust relationship is declarative and auditable - Key rotation at the proxy level propagates automatically without reconfiguring upstream providers +#### Direct Federation vs. Brokered Role Access + +There are two ways a third-party data provider can grant the proxy access to their storage: + +1. **Direct federation** — The data provider registers Source Cooperative as a trusted OIDC identity provider in their own cloud account and creates a role (or service account, or federated identity) that the proxy can assume directly. This gives the provider full control but requires them to configure IdP trust in their account. + +2. **Brokered role access** — Source Cooperative registers itself as an OIDC identity provider in its _own_ cloud account and assumes its own cloud role (e.g. an AWS IAM role, GCP service account, or Azure managed identity). The data provider then grants that Source Cooperative role cross-account access to their storage (e.g. via an S3 bucket policy, GCS IAM binding, or Azure role assignment). The provider never needs to register Source Cooperative as an identity provider — they only need to trust an existing cloud identity. + +The brokered model lowers the barrier for data providers: granting a cloud role access to a bucket is a familiar operation, while registering an external OIDC identity provider is not. It also centralises the OIDC configuration to a single place (Source Cooperative's own account) rather than requiring each provider to replicate it. The tradeoff is that the provider must trust Source Cooperative's intermediate role, and Source Cooperative's account becomes a choke point — any misconfiguration or compromise of that role affects all providers who rely on it. + +Both models can coexist. Providers with stricter security requirements or existing IdP federation workflows can use direct federation; providers who prefer simplicity can grant access to Source Cooperative's brokered role. + ### Outbound Authentication — Stored Credentials (Fallback) The current proxy fetches static cloud credentials (access key ID and secret access key) from the Source Cooperative API for each data connection. The API stores these credentials and serves them to the proxy on demand, cached with a short TTL. From 5f384a7823eb1bd9d558df19568ec443348ae2f2 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 31 Mar 2026 22:19:08 -0700 Subject: [PATCH 13/17] Add details on backends requiring stored credentials for OIDC integration --- adrs/006-outbound-storage.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/adrs/006-outbound-storage.md b/adrs/006-outbound-storage.md index 5059128..6dbe41e 100644 --- a/adrs/006-outbound-storage.md +++ b/adrs/006-outbound-storage.md @@ -56,6 +56,13 @@ The current proxy fetches static cloud credentials (access key ID and secret acc For upstream providers or storage systems that do not support OIDC workload identity federation, this model continues: the proxy fetches stored credentials from the API and uses them to authenticate to the upstream backend. This is not a preferred path — stored credentials must be rotated manually, create a larger blast radius if compromised, and require the platform to hold long-lived secrets on behalf of providers. Data providers should be encouraged to configure OIDC trust relationships where their cloud supports it. +Notable backends that **do not** support external OIDC identity federation for storage access (and therefore require stored credentials): + +- **Cloudflare R2** — API tokens or access key pairs only; no mechanism to trust an external OIDC issuer for storage operations +- **Backblaze B2** — Application keys only; no STS or federation mechanism +- **Wasabi** — Supports STS `AssumeRole` for its own IAM users, but OIDC integration is limited to console SSO, not storage API federation from an external identity provider +- **DigitalOcean Spaces** — No support for trusting an external OIDC issuer; workload identity is limited to DigitalOcean's own internal Droplet-issued tokens + ### Data Provider Hosting Data providers register their upstream storage (their own S3 bucket, GCS bucket, etc.) with Source Cooperative. The proxy serves as an access control, metering, and distribution layer in front of their data. From f4ff58d38138437a88b18c112176706dc6fc80fc Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 3 Apr 2026 09:58:57 -0700 Subject: [PATCH 14/17] Update adrs/004-sts.md Co-authored-by: Tyler Erickson --- adrs/004-sts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adrs/004-sts.md b/adrs/004-sts.md index f9a768c..ccff123 100644 --- a/adrs/004-sts.md +++ b/adrs/004-sts.md @@ -179,7 +179,7 @@ POST /.sts/assume-role-with-web-identity Dot-prefixed account names (`.sts`, etc.) are reserved as invalid, preventing routing conflicts with S3 API paths. -**Request format:** `application/x-www-form-urlencoded`, AWS STS-compatible: +**Request format:** `application/x-www-form-urlencoded`, [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/)-compatible: ``` Action=AssumeRoleWithWebIdentity From e349000f96485d7098a8e8aca2af09cb4c620fb3 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 7 Apr 2026 13:19:02 -0700 Subject: [PATCH 15/17] docs: update ADRs for RFC-001 and authorization Co-Authored-By: Claude Opus 4.6 --- adrs/005-authorization.md | 87 +++++++++++++++++++++++++++++++++++++++ adrs/rfc-001.md | 7 ++++ 2 files changed, 94 insertions(+) diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md index f19cf99..61e171c 100644 --- a/adrs/005-authorization.md +++ b/adrs/005-authorization.md @@ -82,6 +82,93 @@ For non-public resources or write operations: The proxy does not evaluate org membership or permission inheritance logic — the API resolves these internally. When a user belongs to an organisation, the API includes permissions inherited through that membership in the account's resolved grants. The proxy treats the API response as the authoritative set of permissions for the account. +### Proxy-to-API Authentication + +The proxy authenticates its policy store lookups by presenting itself as +the account whose permissions it needs. This avoids a separate +"service account" code path in the API — the same authorization logic +that serves the frontend serves the proxy. + +#### How `account_id` Flows into Proxy-to-API Requests + +The `account_id` that the proxy uses as `sub` originates from the Role +and travels through the credential chain: + +1. **Role lookup.** A caller presents an OIDC token to `/.sts` and + requests a specific Role. The Role is owned by an account — this + could be a **user account** or an **organisation account**. The + owning account's ID becomes the `account_id`. + +2. **SessionToken minting.** The proxy mints a SessionToken JWT + containing `account_id`, `role_name`, `permissions` (the Role's + ceiling), and `assumed_by` (the caller's original IdP subject). + +3. **STS credential issuance.** The SessionToken is encoded into STS + credentials returned to the caller. The `account_id` is embedded in + the `AccessKeyId` (used to derive the signing key) and the + SessionToken JWT is returned as the `SessionToken`. + +4. **Request-time decoding.** When the proxy receives an S3 request + signed with these credentials, it verifies the SigV4 signature, + decodes the SessionToken, and extracts `account_id`. + +5. **Policy store lookup.** The proxy mints a new short-lived JWT with + `sub: account_id` and sends it to the Source Cooperative API to + fetch that account's permissions. + +#### The `sub` Claim Represents an Account, Not Necessarily a User + +The `sub` claim in the proxy-to-API JWT is an **account ID**, which may +identify either a user or an organisation: + +- **User account.** A user authenticates via the frontend or an IdP and + assumes their own `_default` Role. `sub` = the user's account ID. The + API returns that user's permissions — identical to a direct frontend + request. + +- **Organisation account.** A CI workflow authenticates via GitHub OIDC + and assumes a Role owned by `my-org`. `sub` = `my-org` (the org's + account ID). The API returns my-org's full permissions — including + grants on products owned by other accounts that my-org has access to. + The Role ceiling (evaluated locally by the proxy) constrains what the + CI workflow can actually do within those permissions. + +The API does not need to distinguish between these cases. In both, it +receives an account ID and returns that account's resolved permissions. +The proxy handles the Role ceiling intersection locally. + +#### JWT Claims + +For each policy store request, the proxy mints a short-lived JWT signed +with its private key: + +| Claim | Value | Purpose | +|-------|-------|---------| +| `sub` | `account_id` from the SessionToken | Tells the API whose permissions to return. May be a user ID or an org ID. | +| `iss` | The proxy's OIDC issuer URL | The API verifies the signature against the proxy's `/.well-known/jwks.json`. | +| `aud` | The API's base URL | Scopes the token to the policy store API. | +| `role` | `role_name` from the SessionToken | Informational — for audit logging. Does not affect the API's response. | +| `assumed_by` | `assumed_by` from the SessionToken | The original IdP subject, for audit trail. | +| `exp` | Short-lived (≤ 60s) | Limits replay window. | + +#### Trust Model + +The API trusts the proxy to assert any `sub` value. This trust is +established by the API verifying the JWT signature against the proxy's +published OIDC discovery document. Only the proxy holds the signing key. +This is analogous to how an AWS service uses IAM role assumption — the +service is trusted to act on behalf of the principal. + +#### Why `sub` = `account_id`, Not `assumed_by` + +The API's permission model is account-centric. Grants are assigned to +Source Cooperative accounts (users and organisations), not to external +IdP subjects. When a GitHub Actions workflow assumes an org's Role, the +relevant permissions are the org's — not the workflow's. The Role +ceiling (evaluated locally by the proxy) constrains what the workflow +can do within those permissions. The `assumed_by` claim preserves +attribution for audit without affecting authorization. + **Step 6 — Prefix enforcement** If the Role's permission statement includes a prefix constraint (e.g., `sc::my-org::product/my-dataset/uploads/*`), verify the object key falls within that prefix. This enforcement is part of Step 2 and Step 5 — the prefix is evaluated when matching the resource pattern. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 9d9f9b1..74f894f 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -506,6 +506,13 @@ The proxy calls the existing Source Cooperative API for all configuration lookup The Next.js application remains the sole schema owner. The proxy does not need direct database credentials. The API enforces schema constraints before data reaches the proxy. This keeps the operational surface small and avoids the schema governance problems that arise when multiple systems access the same database directly. +The proxy authenticates to the API by minting short-lived JWTs with +`sub` set to the `account_id` from the caller's SessionToken. This +account ID may represent a user or an organisation, depending on which +account owns the Role being assumed. The API uses the same permission +resolution code path regardless of origin — frontend or proxy. See +ADR-005 §Proxy-to-API Authentication for the full credential flow. + The REST API is an availability dependency on the hot path for cache misses. In-process caching absorbs the majority of lookups, so the API is only hit on cold starts and TTL expiry. If profiling reveals the API as a latency bottleneck, direct DynamoDB access can be introduced for the highest-frequency lookups (product flags, account grants) while keeping management operations on the API. ### Configuration in Workers From 7321e69f9fb3a38266a2c5fa22bb7ec783279989 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 17 Apr 2026 09:52:50 -0700 Subject: [PATCH 16/17] docs(adr): API Keys for Environments Without OIDC (#133) ## What I'm changing ## How I did it ## How to test it ## PR Checklist - [ ] This PR has **no** breaking changes. - [ ] I have updated or added new tests to cover the changes in this PR. - [ ] This PR affects the [Source Cooperative Frontend & API](https://github.com/source-cooperative/source.coop), and I have opened issue/PR #XXX to track the change. ## Related Issues --- adrs/001-s3-credentials.md | 4 +- adrs/004-sts.md | 2 +- adrs/008-api-keys.md | 183 +++++++++++++++++++++++++++++++++++++ 3 files changed, 185 insertions(+), 4 deletions(-) create mode 100644 adrs/008-api-keys.md diff --git a/adrs/001-s3-credentials.md b/adrs/001-s3-credentials.md index 8e3ec56..16a91e7 100644 --- a/adrs/001-s3-credentials.md +++ b/adrs/001-s3-credentials.md @@ -135,9 +135,7 @@ No external database lookup is required to verify a request or reconstruct the s ## Permanent API Keys > [!NOTE] -> **Not included in the initial implementation.** The proxy supports only STS-issued session credentials and anonymous access. Long-lived API keys may be added in the future for workflows where neither workload identity federation nor interactive authentication via `auth.source.coop` is feasible — for example, on-premises instruments, legacy ETL systems, or environments without OIDC support. -> -> API keys would be exchanged for temporary STS credentials at the `/.sts` endpoint — the same way OIDC tokens are exchanged today. This keeps the proxy's request-time verification uniform: only short-lived STS credentials are accepted on S3 API calls. No second authorization path is needed. +> **Not included in the initial implementation.** The proxy supports only STS-issued session credentials and anonymous access. See ADR-008 for the API key design: long-lived JWTs signed by the proxy's own OIDC issuer, exchanged at `/.sts` for short-lived STS credentials like any other token. Covers environments without ambient OIDC tokens or browser access (university HPC clusters, on-premises instruments, legacy ETL systems). --- diff --git a/adrs/004-sts.md b/adrs/004-sts.md index ccff123..2851e2a 100644 --- a/adrs/004-sts.md +++ b/adrs/004-sts.md @@ -380,6 +380,6 @@ creds = sts.assume_role_with_web_identity( **Claim constraints on IdP registration (not on Role)** — rejected. Claim constraints belong on the Role because different Roles for the same account may need different constraints for the same IdP. For example, one Role for GitHub Actions might constrain to `refs/heads/main` while another allows any branch. -**SC Credential Tokens (long-lived, up to 90 days)** — not included. For workflows that need persistent access without ambient OIDC, the established pattern is direct access to the underlying storage bucket. The STS design focuses on short-lived, federated credentials. +**API keys for environments without OIDC** — see ADR-008. Long-lived JWTs signed by the proxy's own OIDC issuer, exchanged at `/.sts` like any other token. Covers university HPC clusters, on-premises instruments, and other environments without ambient OIDC tokens or browser access. **RFC 8693 token exchange ("act-as") for Next.js server-side requests** — considered. Rejected for initial implementation: passing the user's Ory token through client-side is simpler and achieves the same access metrics goal. Can be revisited if audit logs need to distinguish "user accessed directly" from "Next.js accessed on user's behalf." diff --git a/adrs/008-api-keys.md b/adrs/008-api-keys.md new file mode 100644 index 0000000..16f853b --- /dev/null +++ b/adrs/008-api-keys.md @@ -0,0 +1,183 @@ +# ADR-008: API Keys for Environments Without OIDC + +**Date:** 2026-04-01 +**RFC:** RFC-001 +**Depends on:** ADR-001, ADR-004, ADR-006 + +--- + +## Context + +ADR-004 defines inbound authentication via OIDC federation: callers present a JWT from a trusted identity provider and exchange it at `/.sts` for short-lived STS credentials. This works well for CI/CD platforms with ambient OIDC tokens (GitHub Actions, GitLab CI, etc.) and for interactive users who can complete a browser-based login via `auth.source.coop`. + +However, a significant class of users has neither: + +- Researchers running recurring batch jobs or cronjobs on university HPC clusters (SLURM, PBS, traditional login nodes) +- On-premises instruments or data loggers that push observations on a schedule +- Legacy ETL systems in environments without a supported OIDC issuer + +These users have Source Cooperative accounts but operate in compute environments that do not issue OIDC tokens and cannot perform interactive browser authentication at runtime. ADR-001 and ADR-004 both identify this gap as future work. + +--- + +## Decision + +### API Keys as Long-Lived JWTs + +Source Cooperative issues API keys as long-lived JWTs signed by the data proxy's own signing key — the same key the proxy uses as an OIDC issuer for outbound storage authentication (ADR-006). The proxy already publishes its JWKS and `/.well-known/openid-configuration`; API key JWTs are verifiable against the same key material. + +An API key JWT contains: + +```json +{ + "iss": "https://data.source.coop", + "sub": "", + "jti": "", + "iat": 1711929600, + "exp": 1743465600, + "type": "api_key" +} +``` + +- `iss` is the proxy's own issuer URL, not `auth.source.coop` (which is Ory Network and outside Source Cooperative's control for token minting) +- `sub` identifies the Source Cooperative account that owns the key +- `jti` is a unique key identifier used for revocation checks +- `exp` is optional — keys without an expiry are valid until explicitly revoked +- `type` distinguishes API key JWTs from other tokens the proxy may issue (e.g. outbound federation tokens) + +### Key Lifecycle + +**Creation:** + +Users create API keys via the Source Cooperative UI or CLI: + +``` +source keys create --label "ncar-cronjob" --role sc::my-org::role/publisher +``` + +The system: +1. Generates a unique `jti` +2. Stores key metadata in the policy store: `jti`, account ID, label, bound Role (optional), created-at, expires-at (nullable) +3. Mints and signs the JWT +4. Returns the raw JWT to the user — displayed once, never stored by the platform + +**Revocation:** + +Users revoke keys via the UI or CLI: + +``` +source keys revoke +``` + +Revocation marks the key's `jti` as revoked in the policy store. The revocation takes effect within the `jti` validation cache TTL (see below). + +**Management API:** + +``` +POST /api/accounts/{account_id}/keys +GET /api/accounts/{account_id}/keys +DELETE /api/accounts/{account_id}/keys/{key_id} +``` + +The `GET` endpoint returns key metadata (ID, label, created-at, expires-at, last-used-at) but never the JWT itself. Only account owners and org admins can manage keys. + +### STS Exchange + +API key JWTs are exchanged at `/.sts/assume-role-with-web-identity` using the same flow as any other OIDC token (ADR-004): + +``` +Action=AssumeRoleWithWebIdentity +&WebIdentityToken= +&RoleArn=sc::my-org::role/publisher +&RoleSessionName=ncar-daily-sync +``` + +The STS exchange flow proceeds as defined in ADR-004 with one additional step: + +1. Parse `RoleArn` → extract `account_id` and `role_name` +2. Load Role definition (cached) +3. Extract `iss` from JWT → matches `https://data.source.coop` +4. Verify JWT signature against the proxy's own JWKS +5. Verify `exp` (if present), `nbf`, `iat` +6. **Validate `jti` against the policy store** — confirm the key has not been revoked (cached, 30–60s TTL) +7. Evaluate claim constraints for the matched IdP binding +8. Validate `DurationSeconds` ≤ Role's `max_session_duration` +9. Generate credentials and return response + +Step 6 is the only addition to the existing STS flow. For non-API-key tokens (those without `"type": "api_key"`), this step is skipped. + +### Platform IdP Registration + +The proxy's own issuer is registered as a platform IdP: + +```json +{ + "id": "source-coop-api-key", + "issuer_url": "https://data.source.coop", + "display_name": "Source Cooperative API Key", + "well_known_claims": ["type"], + "audience_hint": "https://data.source.coop" +} +``` + +Roles that should be assumable via API key must include an identity constraint binding for this IdP: + +```json +{ + "idp": "source-coop-api-key", + "claim_constraints": [ + {"claim": "type", "operator": "equals", "value": "api_key"} + ] +} +``` + +This reuses the existing Role and identity constraint model from ADR-004 without modification. Account owners explicitly opt in to API key access per Role — a Role without a `source-coop-api-key` binding cannot be assumed with an API key. + +### Role Binding + +API keys can optionally be bound to a specific Role at creation time. A bound key can only be used to assume that Role. An unbound key can assume any Role the account owns that has a `source-coop-api-key` identity constraint. + +Bound keys reduce blast radius: if leaked, the key can only access what that specific Role permits. For high-value automated workflows, bound keys are recommended. + +### Caching and Revocation Latency + +The `jti` validity check uses the same caching infrastructure as other policy store lookups (ADR-007): + +- In-process cache with 30–60s TTL +- Workers KV as a shared cache tier + +This means revocation takes effect within 30–60 seconds. For the target use case (long-running cronjobs, batch pipelines), this latency is acceptable. If faster revocation is needed, the HMAC server secret rotation mechanism from ADR-001 invalidates all active STS sessions immediately — a more disruptive but available emergency response. + +--- + +## Consequences + +**Benefits** + +- Covers the authentication gap for environments without OIDC or browser access +- No new auth path at the proxy layer — API key JWTs flow through the existing `/.sts` exchange +- Reuses the proxy's existing OIDC issuer infrastructure (signing key, JWKS) from ADR-006 +- Reuses the existing Role and identity constraint model from ADR-004 +- Revocation is explicit and auditable via `jti` lookup +- Optional Role binding limits blast radius of leaked keys + +**Costs / Risks** + +- API key JWTs are bearer tokens — anyone with the raw JWT can use it. Users must treat them like passwords (store in environment variables or secret files, not in source control) +- The `jti` revocation check adds a policy store dependency to the STS exchange path for API key tokens. Cache misses add latency. +- Keys without expiry are valid indefinitely until revoked. If a user loses access to the management UI (e.g. leaves a university), orphaned keys persist unless an org admin revokes them. +- The proxy's signing key is now used for two purposes: outbound federation tokens (ADR-006) and API key JWTs. A signing key compromise affects both. Key rotation must account for both uses. + +--- + +## Alternatives Considered + +**Ory-issued long-lived tokens** — not feasible. `auth.source.coop` is Ory Network, which controls its own signing keys. Source Cooperative cannot mint arbitrary long-lived JWTs from Ory's issuer. + +**OAuth2 client credentials grant** — considered. The client credentials grant authenticates an application, not a user — the resulting token's `sub` is the client ID, not a user identity. Mapping OAuth2 clients back to Source Cooperative accounts would require a bespoke service account system built on top of OAuth2. + +**Ory personal access tokens** — investigated. Ory Network's PAT/API key concept (`ory_pat_`) is for project admin API access, not end-user authentication. User-scoped PATs are an [open feature request](https://github.com/ory/kratos/issues/1106) on Ory Kratos but not available. + +**Opaque API keys with hash-based validation** — considered. The platform generates a random secret, stores a hash, and validates by re-hashing. This works but requires a dedicated validation endpoint or a new auth path at `/.sts`. The JWT approach avoids this by making API keys indistinguishable from other OIDC tokens at the STS layer — no new endpoint, no new validation logic beyond the `jti` check. + +**Long-lived Ory refresh tokens** — considered as a near-term workaround. The user performs a one-time `source login` (device flow) and stores the refresh token. Cronjobs silently refresh access tokens. This works without new infrastructure but refresh tokens expire eventually, causing silent failures in unattended workflows. Suitable as an interim measure but not a durable solution for indefinitely recurring workloads. From a1b6082a48e4d31eaa053fcc018de7d328b4e564 Mon Sep 17 00:00:00 2001 From: Tyler Erickson Date: Mon, 16 Mar 2026 15:30:56 -0700 Subject: [PATCH 17/17] Add trailing spaces for newline --- adrs/rfc-001.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index 74f894f..17b6114 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -1,7 +1,7 @@ # RFC-001: Source Cooperative Data Proxy Re-Architecture **Date:** 2026-03-14 -**Authors:** @alukach +**Authors:** @alukach **Replaces:** Current data proxy (ECS, Rust, long-lived credentials) ---