Scaffolding: Federation gateway setup

[klpoland]

Note

Medium Risk
Introduces a new internal export surface and changes global API-key auth/permissions; misconfiguration of CIDRs or operational flags could block sync or widen access, though export is off by default and sync keys are route-scoped.

Overview
Adds gateway-side federation scaffolding so a local federation-sync service can pull public metadata and react to changes, controlled by FEDERATION_ENABLED and related env settings (documented in example django.env).

Internal export API is registered under /api/.../federation/ with list/detail endpoints for public finalized datasets and public captures. Responses use new DatasetFederationSerializer / CaptureFederationSerializer payloads (including site_name from FEDERATION_SITE_NAME). Access requires a FederationSync API key, federation operational checks (sync health URL, Redis ping, presence of sync key), and client IP in FEDERATION_EXPORT_ALLOWED_CIDRS (defaults to private ranges).

Change notifications: post_save signals on datasets/captures publish created/updated/deleted events to Redis (FEDERATION_EVENTS_CHANNEL) when federation is operational and the record is (or ceases to be) exportable.

API key behavior: APIKeyAuthentication now sets request.auth to the UserAPIKey object. A global DisallowFederationSyncKey permission blocks sync keys from normal asset APIs; export routes use the inverse IsFederationSyncKey. New create_federation_sync_api_key management command and DB migration extend KeySources with federation_sync.

Also includes contract tests against sibling sds_federation Pydantic models, hardening/operational tests, and small dev-tool tweaks (fallow-cross-file-dupes.sh, create_test_files import paths).

^{Reviewed by Cursor Bugbot for commit 3393c0f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
gateway/sds_gateway/api_methods/views/dataset_endpoints.py	42% smaller
gateway/sds_gateway/api_methods/tests/test_authenticate.py	26% smaller
gateway/config/settings/base.py	7% smaller
gateway/.envs/example/django.env	Unsupported file format
gateway/config/api_router.py	0% smaller
gateway/pyproject.toml	Unsupported file format
gateway/scripts/fallow-cross-file-dupes.sh	Unsupported file format
gateway/sds_gateway/api_methods/apps.py	0% smaller
gateway/sds_gateway/api_methods/authentication.py	0% smaller
gateway/sds_gateway/api_methods/federation/__init__.py	0% smaller
gateway/sds_gateway/api_methods/federation/availability.py	0% smaller
gateway/sds_gateway/api_methods/federation/events.py	0% smaller
gateway/sds_gateway/api_methods/federation/export_contract.py	0% smaller
gateway/sds_gateway/api_methods/federation/permissions.py	0% smaller
gateway/sds_gateway/api_methods/federation/signals.py	0% smaller
gateway/sds_gateway/api_methods/helpers/compile_federated_data.py	0% smaller
gateway/sds_gateway/api_methods/models.py	0% smaller
gateway/sds_gateway/api_methods/permissions.py	0% smaller
gateway/sds_gateway/api_methods/serializers/capture_serializers.py	0% smaller
gateway/sds_gateway/api_methods/serializers/dataset_serializers.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export_contract.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_hardening.py	0% smaller
gateway/sds_gateway/api_methods/views/capture_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/views/federation_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/views/file_endpoints.py	0% smaller
gateway/sds_gateway/users/management/commands/create_federation_sync_api_key.py	0% smaller
gateway/sds_gateway/users/management/commands/create_test_files.py	0% smaller
gateway/sds_gateway/users/migrations/0012_alter_userapikey_source_federation_sync.py	0% smaller
gateway/sds_gateway/users/migrations/max_migration.txt	Unsupported file format
gateway/sds_gateway/users/models.py	0% smaller

[klpoland]

Added some dedicated API endpoints, serializers, and signals for federation and configuration that defaults to disabling federation sync.

[cursor]

Test command calls missing upload method

Low Severity

This commit switches the dev command to get_minio_client(), which returns ObjectStoreFacade with MinIO-compatible methods like fput_object, but the upload loop still calls upload_file. That method is not defined on the facade or delegated MinIO client, so the command fails at runtime when uploading fixtures.

 

^{Reviewed by Cursor Bugbot for commit 2c09fae. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3393c0f. Configure here.}

[cursor]

Command always rotates sync keys

Medium Severity

Each run of create_federation_sync_api_key deletes every existing FederationSync API key for the sync user before creating a new one. Re-running the command invalidates keys already configured in federation-sync without an explicit rotate flag.

 

^{Reviewed by Cursor Bugbot for commit 3393c0f. Configure here.}