Skip to content

Feat/irokle: Replace automerge and gossip with a performance optimized version of irokle#296

Open
St4NNi wants to merge 86 commits into
mainfrom
feat/irokle_perf
Open

Feat/irokle: Replace automerge and gossip with a performance optimized version of irokle#296
St4NNi wants to merge 86 commits into
mainfrom
feat/irokle_perf

Conversation

@St4NNi

@St4NNi St4NNi commented Jun 15, 2026

Copy link
Copy Markdown
Member

This PR replaces the legacy Automerge/gossip-based metadata synchronization path with an irokle-backed document sync pipeline. It also hardens metadata replication with durable task/outbox persistence, improves cold-start recovery, and adds performance work across metadata materialization, and storage read / list operations.

Changes

  • Replace the legacy Automerge and gossip runtime with irokle-backed document synchronization for core metadata documents, realm configuration, and lifecycle events.
  • Add durable document sync outboxes, persistent task scheduling, failed-task rearming, and startup restoration so metadata sync work survives restarts and transient failures.
  • Introduce realm-scoped sync placements, peer refresh/backfill logic, and node/topic diagnostics to make replication targets explicit and easier to inspect.
  • Rework metadata handling around event-sourced updates, deterministic actors, materialization queues, visible registry caching, and cache warming to reduce read-time work and improve convergence behavior.
  • Improve listing and storage performance with batched point reads, inclusive iterator start bounds, faster hash path handling, visible registry lookups, and ListObjectsV2 pagination/common-prefix optimizations.
  • Add stage, latency, queue-lag, DHT lookup, and replication telemetry to make sync throughput, tail latency, and backlog behavior observable during operation and tests.
  • Drastically improve performance by batching operations and reducing IO churn
  • Expand API support and tests around users, groups, info endpoints, onboarding, metadata CRUD, replication, cold starts, throughput, convergence tails, and query concurrency.
  • Update dependencies, doctor tooling, startup wiring, and diagnostics to support the new irokle-based sync architecture end to end.

St4NNi added 30 commits June 1, 2026 16:36
@St4NNi St4NNi force-pushed the feat/irokle_perf branch from f403ba4 to e8614ab Compare June 16, 2026 18:17

@das-Abroxas das-Abroxas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, it is not possible to review a pull request of this size in all its extent within a reasonable amount of time. However, I hope that the tests implemented in Aruna with this (as well as the tests in Craqle and Irokle) provide sufficient evidence of the stability and consistency of this implementation. For the most part, I have to trust your judgement here and assume that you know what you're doing and that hopefully you'll take responsibility whatever the outcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants