Audit logging for CALM Hub mutating operations

[markscott-ms]

Feature Proposal

Target Project:

calm-hub

Description of Feature:

Add an audit trail to CALM Hub that records every mutating operation — who created a namespace, who uploaded a new architecture/pattern/control version, who granted or revoked access, etc. — across all ~30+ mutating REST endpoints spread over 14 resource classes, including the legacy name-based MappingControllerResource. Each record captures who performed the action, when, on which entity, and whether it succeeded or was denied for lack of permission. This gives operators a durable, tamper-resistant "who did what, when" log without requiring per-endpoint boilerplate that's easy to forget on future endpoints.

User Stories:

• As a CALM Hub operator, I want a durable record of who created or modified a namespace/architecture/pattern/control/etc. and when, so that I can answer "who changed X and when" without relying on database timestamps alone.
• As a CALM Hub operator, I want denied/unauthorized mutation attempts recorded, so that I can detect misconfigured clients or attempted unauthorized access to a namespace or domain.
• As an operator running the self-contained standalone/read-only deployment, I want the audit trail to work without requiring an external logging pipeline, while still being able to ship audit events to my existing log aggregation if I have one.
• As a maintainer adding a new mutating endpoint in the future, I want it to be covered by the audit trail automatically, without having to remember to add anything, as long as it follows the existing URL conventions.

Current Limitations:

CALM Hub has no audit logging today — a grep -ri "audit" across the codebase turns up nothing but a documentation string. There is no way to determine who performed a given mutation or when, beyond whatever creationDateTime/updateDateTime fields exist on individual entities (which don't capture the actor or record denied attempts at all).

Proposed Implementation:

Capture mechanism. A new post-matching ContainerResponseFilter (AuditRequestFilter, in org.finos.calm.security) runs on every request and derives audit data generically, rather than instrumenting every store method by hand. This is deliberately not modeled on the existing ReadOnlyRequestFilter's @PreMatching pattern — it needs UriInfo.getPathParameters() resolved, which only happens after JAX-RS routing.

Nearly every mutating endpoint already encodes namespace/domain + entity ID + version as path template params (e.g. POST {namespace}/architectures/{architectureId}/versions/{version}), so the filter reads these directly with zero per-endpoint code. For the handful of server-generated-ID create endpoints where the ID isn't in the request path (NamespaceResource.createNamespace, DomainResource.createDomain, CoreSchemaResource.createSchemaVersion, MappingControllerResource's two generic /calm writes), every one of them returns Response.created(URI) on success with a Location header that encodes the new entity's identity — the filter falls back to parsing that header, again with no resource-class changes needed for the success path.

The one real gap is denied (401/403) outcomes on body-derived endpoints, since no store call (and thus no Location header) happens on denial. Of the five body-derived endpoints, three deny via a manual in-method check (NamespaceResource.createNamespace; MappingControllerResource.createResourceFromDocument/updateResourceFromDocument) and can stage a small "audit context" property immediately after parsing (reusing the existing CalmDocumentParser.parseCanonicalId/parseDomainControlId output — no new parsing code) before the permission check runs. The other two (DomainResource.createDomain, CoreSchemaResource.createSchemaVersion) deny via the @PermissionsAllowed CDI interceptor, which rejects before the method body executes, so no in-method hook can run early enough — these get full fidelity on success and a documented entityId=null on denial (an accepted, narrow gap, both being low-frequency GLOBAL_ADMIN-only operations).

Outcomes recorded: SUCCESS (2xx) and DENIED (401/403) only. Other failures (400/404/409/500) are routine/operational and already covered by normal application logs.

Storage: a new dual-backend AuditLogStore (Mongo + Nitrite, following the exact existing pattern every other entity uses — interface + MongoAuditLogStore + NitriteAuditLogStore + AuditLogStoreProducer, new collection auditLogs), plus a structured log line via the existing logger under a dedicated org.finos.calm.audit category. Both independently toggleable via calm.audit.store.enabled / calm.audit.log.enabled, so the self-contained standalone/read-only deployment stays dependency-free by default while still giving operators with an existing SIEM a free integration path.

Tamper resistance: append-only at the interface level — AuditLogStore exposes only record()/query(), no update()/delete() anywhere.

Record schema: timestamp, actor (username, from SecurityIdentity.getPrincipal().getName()), action (CREATE/UPDATE/GRANT/REVOKE), entityType, namespace/domain (nullable, mutually exclusive), entityId, version (nullable), outcome (SUCCESS/DENIED), sourceIp (nullable, its own toggle: calm.audit.capture-source-ip).

Technical design considerations

• No query REST API in this iteration — AuditLogStore.query() exists on the interface for future/internal use only; reading the trail today means direct DB access or ops tooling.
• calm.readonly=true needs no special-casing: ReadOnlyRequestFilter (@PreMatching @Priority(0)) rejects mutating verbs with 405 before routing, so no mutation ever reaches the post-matching audit filter in that mode.
• In no-auth mode, actor is always the literal string "no-auth" for every caller — a known limitation of that deployment mode, not something this feature can fix, and worth a startup warning if calm.audit.store.enabled=true alongside calm.auth.enabled=false.

API changes

None — no new REST endpoints in this iteration (see "No query REST API" above).

Data model changes

New domain/audit/ package: AuditLogEntry, AuditAction, AuditEntityType, AuditOutcome, AuditLogQuery — following the existing UserAccess.java POJO/Builder/Jackson-timestamp conventions. New Mongo collection auditLogs with non-unique indexes on {namespace,entityType,entityId,timestamp}, {domain,entityType,entityId,timestamp}, {actor,timestamp}, {timestamp}.

Dependencies on other components

Reuses CalmDocumentParser (parseCanonicalId, parseDomainControlId, TYPE_MAP) for the MappingControllerResource hooks — no new $id-parsing logic. Follows the existing store/producer dual-backend pattern and ReadOnlyRequestFilter's native-image-safe config-reading pattern.

Alternatives Considered:

• Explicit per-store instrumentation instead of a filter — rejected: every entity has parallel Mongo/Nitrite implementations, so this would mean ~60 call sites with no structural guarantee a future endpoint remembers to add it, and it's structurally unable to see denied requests at all (those are rejected before the store is reached).
• Structured logging only, no DB store — rejected: would leave the self-contained standalone/read-only deployment with no queryable audit trail at all if no external log pipeline is configured.
• Hash-chained/tamper-evident storage — considered and rejected for v1: forces a global write-serialization point (every other store here has independent, uncontended writes) and only protects against partial tampering, not a privileged attacker with direct DB access. Flagged as a possible future enhancement.
• In-app REST query API for the audit trail — deferred: adds a real access-control design question (which scope gates cross-namespace audit data) that isn't needed for the core capture mechanism to be useful via direct DB access.
• Capturing all 4xx/5xx as audit-worthy, not just 401/403 — rejected: routine client/validation errors aren't security-relevant and would dilute the trail; ordinary application logs already cover them.
• Recording before/after content diffs per mutation — rejected: requires fetching and diffing the prior document on every write (non-trivial for raw-JSON entities like Decorator), disproportionate to a "who did what" trail, and largely redundant with entities' own version history.

Testing Strategy:

• Spike first: confirm empirically that UriInfo.getPathParameters() is populated in a post-matching ContainerResponseFilter when the response was produced by an @PermissionsAllowed denial (the one real technical risk in this design). If path params aren't reliably available on denial, fall back to recording entityType+actor+timestamp+outcome only for those DENIED entries.
• Unit tests: TestAuditRequestFilterShould (method×status matrix, per-resource-class entity-type resolution, Location-header parsing per entity type, staged-context priority over path params), TestAuditServiceShould (toggle matrix, exception isolation so a store failure never blocks the actual response), TestMongoAuditLogStoreShould/TestNitriteAuditLogStoreShould/TestAuditLogStoreProducerShould (mirroring their Namespace equivalents), TestLocationSegmentParserShould.
  • As implemented: rather than extending TestNamespaceResourceShould/TestMappingControllerResourceShould with staging-hook assertions as originally sketched here, TestAuditRequestFilterShould's staged-context tests plus the DENIED-path integration test below cover the same behavior more directly — a real end-to-end HTTP call asserting the persisted record, not a mock-verification that a staging call happened.
• Integration tests (-P integration, TestContainers, mirroring MongoArchitectureIntegration.java): MongoAuditLogIntegration (end-to-end mutating call producing the expected auditLogs document) and MongoAuditLogDeniedIntegration — critically a DENIED-path test (call without sufficient permission, assert 403, assert a correctly-populated DENIED audit record with namespace still resolved) — the concrete proof that resolves the spike's risk. Also empirically confirms a distinct, real behavior worth noting: a @PermissionsAllowed 403 reaches this filter (fully attributable), but a fully unauthenticated 401 is rejected by the OIDC layer before JAX-RS dispatch and never reaches it — no audit record is written for that specific case, since AuditRequestFilter is a JAX-RS filter and only sees requests that reach resource matching.
• Manual smoke test (superseded): the automated DENIED-path integration test above exercises the same "create via HTTP, inspect auditLogs directly" flow with real assertions on every CI run, which is strictly stronger than a one-time manual curl check — so the manual step was dropped rather than performed separately.
• Full ../mvnw clean verify -Ddependency-check.skip=true run to confirm the existing 90%-per-class JaCoCo gate passes on every new/touched class.

Documentation Requirements:

• Update calm-hub/AGENTS.md's "Adding a New REST Endpoint" checklist to note that new mutating endpoints get audit coverage automatically as long as they follow the existing {namespace|domain}/.../{id}[/versions/{version}] path convention and return Response.created(...) with a Location that embeds the new ID — and flag AuditRequestFilter's entity-type map and Location-parsing helper as the two places needing a one-line addition when a genuinely new resource type (not just a new endpoint) is introduced.
• Document the calm.audit.* config properties alongside the existing calm.readonly/calm.mcp.enabled documentation.
• Class-level javadoc on AuditRequestFilter explaining the read-only-mode non-interaction and the no-auth-mode actor limitation, so future readers don't wonder why there's no special-casing.

Implementation Checklist:

• Design reviewed and approved
• Implementation completed
• Tests written and passing
• Documentation updated
• Relevant workflows updated (if needed)
• Performance impact assessed

Additional Context:

Design decisions were reached through an extended discussion covering: capture mechanism (filter vs. per-store instrumentation), which outcomes to record (denials vs. all failures), storage destination (DB store vs. structured logging vs. both), tamper resistance (append-only vs. hash-chaining), query access (in-app API vs. deferred), and record schema (including source IP as an independently-toggleable field). See "Alternatives Considered" above for the reasoning behind each rejected option.

Implementation status: built on feature/audit-log. The design above holds; the Testing Strategy section was updated post-implementation to reflect two deliberate deviations (test coverage achieved via different, more direct means; the manual smoke test superseded by an automated equivalent) — see strikethrough/inline notes above. A code-review pass also caught and fixed a few implementation-level issues along the way (an ADR revision-number field not being populated on update, and unsanitized values reaching the audit log line) — the design itself was unaffected.