-
Notifications
You must be signed in to change notification settings - Fork 17
perf(ensindexer): unblock Ponder prefetch on hot tables #2016
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 4 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
a88a715
perf(ensindexer): unblock Ponder prefetch on hot tables
shrugs dcc8ca8
update comment
shrugs f85feb0
fix: bot notes (loop 1)
shrugs cbf8bec
fix: bot notes (loop 2)
shrugs 5a79192
test(integration-test-env): set devnet chain id to match datasources
shrugs ce176bd
Merge remote-tracking branch 'origin/main' into perf/delimited-ids
shrugs ab4a94e
fix(integration-tests): wipe state, ephemeral postgres port, lowercas…
shrugs b36ec5d
fix: normalize address at readContract boundary, restore main() invoc…
shrugs e4eef01
refactor: cast event.log.address as NormalizedAddress in getThisAccou…
shrugs 04f081c
docs(ensdb): update schema docs for migrated_nodes_by_(parent|node) s…
shrugs 3db9b6d
docs(enssdk): genericize id type comments and point at ids.ts
shrugs cc93429
refactor(orchestrator): drop unnecessary pre-up wipe step
shrugs 97b96fd
refactor(orchestrator): drop container_name on ensdb, fix lookup
shrugs 952c514
docs(orchestrator): tighten compose file comments
shrugs 01f2fa3
fix(enssdk): drop AccountIdString brand on composite id types
shrugs File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "@ensnode/ensdb-sdk": minor | ||
| --- | ||
|
|
||
| `migrated_nodes` renamed to `migrated_nodes_by_parent` and re-keyed by composite `(parentNode, labelHash)` to match the payload of `ENSv1Registry(Old)#NewOwner` events. New sibling `migrated_nodes_by_node` keyed solely by `node` for the three `ENSv1RegistryOld` handlers (`Transfer` / `NewTTL` / `NewResolver`) that emit only `node`. Both rows are written together by the migration helper so each read site addresses whichever key matches its event payload. Schema definitions live in a new `migrated-nodes.schema.ts`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| "enssdk": minor | ||
| --- | ||
|
|
||
| Switch composite ids to dash-delimited tuples so Ponder's profile-pattern matcher can decompose them and prefetch hot tables. | ||
|
|
||
| Every id constructor (`makeENSv1RegistryId`, `makeENSv2RegistryId`, `makeENSv1VirtualRegistryId`, `makeConcreteRegistryId`, `makeResolverId`, `makeENSv1DomainId`, `makeENSv2DomainId`, `makePermissionsId`, `makePermissionsResourceId`, `makePermissionsUserId`, `makeResolverRecordsId`, `makeRegistrationId`, `makeRenewalId`) now joins its components with `-` instead of CAIP-style mixed `:` / `/` delimiters. `makeENSv2DomainId` no longer wraps the registry contract in CAIP-19 ERC1155 form since the registry already namespaces it. Ponder's matcher only does single-level string-delimiter splits, so the unified `-` tuple is the shape it can decompose to derive prefetch lookup keys from event args. | ||
101 changes: 101 additions & 0 deletions
101
apps/ensindexer/src/lib/protocol-acceleration/migrated-node-db-helpers.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| import config from "@/config"; | ||
|
|
||
| import { type LabelHash, makeSubdomainNode, type Node } from "enssdk"; | ||
|
|
||
| import { getENSRootChainId } from "@ensnode/datasources"; | ||
|
|
||
| import { ensIndexerSchema, type IndexingEngineContext } from "@/lib/indexing-engines/ponder"; | ||
|
|
||
| /** | ||
| * Why two tables for one logical "is this node migrated?" check. | ||
| * | ||
| * The check fires from many Registry handlers, but the event payload differs between them: | ||
| * - ENSv1Registry(Old)#NewOwner emits `parentNode` and `labelHash` as separate args. | ||
| * - ENSv1RegistryOld#Transfer / NewTTL / NewResolver emit only the post-namehash `node` | ||
| * | ||
| * Ponder's indexing-cache prefetch path predicts hot-table reads ahead of each event by deriving | ||
| * the lookup key from the event's args — but its profile-pattern matcher can only do direct equality | ||
| * and single-level string-delimiter splits. It can NOT invert keccak. So a table keyed by the | ||
| * post-namehash `node` is unprofileable from a NewOwner event (where `node` is a computed namehash | ||
| * of `(parentNode, labelHash)`), and a table keyed by `(parentNode, labelHash)` is unprofileable | ||
| * from a Transfer/NewTTL/NewResolver event (which doesn't carry those fields). | ||
| * | ||
| * Either single-table choice surrenders prefetch on other handlers. Keying solely by | ||
| * `(parentNode, labelHash)` would help the NewOwner hot path but disable prefetching on the other | ||
| * three handlers, which can't reconstruct that pair from `node` without a reverse-index whose lookup | ||
| * key is itself a un-prefetchable namehash. | ||
| * | ||
| * The two-table layout sidesteps both problems: write _both_ rows on every migration, then have each | ||
| * read site address the table whose key matches its event payload. Both reads stay on the prefetch | ||
| * hot-path. The cost is one extra "insert on conflict do nothing" per migration, and the storage of | ||
| * that information, naturally, doubles. As of 2026-04-29, the size of the migrated_nodes_by_parent | ||
| * table is ~1GB, meaning that this optimization will consume an additional ~1GB of storage but | ||
|
shrugs marked this conversation as resolved.
|
||
| * will result in significantly faster indexing for the ENSv1Registry(Old) events. | ||
| * | ||
| * See {@link migratedNodeByParent} and {@link migratedNodeByNode} in the ensdb-sdk schema. | ||
| */ | ||
|
|
||
| const invariant_isENSRootChain = (context: IndexingEngineContext) => { | ||
| if (context.chain.id === getENSRootChainId(config.namespace)) return; | ||
|
shrugs marked this conversation as resolved.
|
||
|
|
||
| throw new Error( | ||
| `Invariant: Node migration status is only relevant on the ENS Root Chain, and this function was called in the context of ${context.chain.id}.`, | ||
| ); | ||
|
shrugs marked this conversation as resolved.
|
||
| }; | ||
|
|
||
| /** | ||
| * Returns whether `(parentNode, labelHash)` has migrated to the new Registry contract. Used by | ||
| * ENSv1RegistryOld#NewOwner where both fields are emitted as event args directly — keyed access | ||
| * keeps the read on Ponder's prefetch hot-path. | ||
| */ | ||
| export async function nodeIsMigratedByParentAndLabel( | ||
| context: IndexingEngineContext, | ||
| parentNode: Node, | ||
| labelHash: LabelHash, | ||
| ) { | ||
| invariant_isENSRootChain(context); | ||
|
|
||
| const record = await context.ensDb.find(ensIndexerSchema.migratedNodeByParent, { | ||
| parentNode, | ||
| labelHash, | ||
| }); | ||
| return record !== null; | ||
| } | ||
|
shrugs marked this conversation as resolved.
|
||
|
|
||
| /** | ||
| * Returns whether `node` has migrated to the new Registry contract. Used by | ||
| * ENSv1RegistryOld#Transfer/NewTTL/NewResolver where only `node` is emitted as an event arg — | ||
| * keyed access on the sibling {@link migratedNodeByNode} table keeps the read on the prefetch | ||
| * hot-path even though the composite-key {@link migratedNodeByParent} table can't be addressed | ||
| * without a reverse lookup. | ||
| */ | ||
| export async function nodeIsMigrated(context: IndexingEngineContext, node: Node) { | ||
| invariant_isENSRootChain(context); | ||
|
|
||
| const record = await context.ensDb.find(ensIndexerSchema.migratedNodeByNode, { node }); | ||
| return record !== null; | ||
| } | ||
|
|
||
| /** | ||
| * Record that `(parentNode, labelHash)` has migrated to the new Registry contract. Writes both | ||
| * the composite-key {@link migratedNodeByParent} row and its sibling {@link migratedNodeByNode} | ||
| * index so each downstream read site can address whichever key it can profile against event args. | ||
| */ | ||
| export async function migrateNode( | ||
| context: IndexingEngineContext, | ||
| parentNode: Node, | ||
| labelHash: LabelHash, | ||
| ) { | ||
| invariant_isENSRootChain(context); | ||
|
|
||
| await context.ensDb | ||
| .insert(ensIndexerSchema.migratedNodeByParent) | ||
| .values({ parentNode, labelHash }) | ||
| .onConflictDoNothing(); | ||
|
|
||
| const node = makeSubdomainNode(labelHash, parentNode); | ||
| await context.ensDb | ||
| .insert(ensIndexerSchema.migratedNodeByNode) | ||
| .values({ node }) | ||
| .onConflictDoNothing(); | ||
|
shrugs marked this conversation as resolved.
shrugs marked this conversation as resolved.
|
||
| } | ||
|
shrugs marked this conversation as resolved.
|
||
36 changes: 0 additions & 36 deletions
36
apps/ensindexer/src/lib/protocol-acceleration/registry-migration-status.ts
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
59 changes: 59 additions & 0 deletions
59
packages/ensdb-sdk/src/ensindexer-abstract/migrated-nodes.schema.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| /** | ||
| * Schema Definitions that track ENS Registry migration status for Protocol Acceleration. | ||
| */ | ||
|
|
||
| import type { LabelHash, Node } from "enssdk"; | ||
| import { onchainTable, primaryKey } from "ponder"; | ||
|
|
||
| /** | ||
| * Tracks the migration status of a node. | ||
| * | ||
| * Due to a security issue, ENS migrated from the RegistryOld contract to a new Registry | ||
| * contract. When indexing events, the indexer must ignore any events on the RegistryOld for domains | ||
| * that have since been migrated to the new Registry. | ||
| * | ||
| * To store the necessary information required to implement this behavior, we track the set of nodes | ||
| * that have been registered in the (new) Registry contract on the ENS Root Chain. When an event is | ||
| * encountered on the RegistryOld contract, if the relevant node exists in this set, the event should | ||
| * be ignored, as the node is considered migrated. | ||
| * | ||
| * Note that this logic is only necessary for the ENS Root Chain, the only chain that includes the | ||
| * Registry migration: we do not track nodes in the Basenames and Lineanames deployments of the | ||
| * Registry on their respective chains, for example. | ||
| * | ||
| * Note also that this Registry migration tracking is isolated to the Protocol Acceleration schema/plugin. | ||
| * That is, the subgraph plugin implements its own Registry migration logic. By isolating this logic | ||
| * to the Protocol Acceleration plugin, we allow the Protocol Acceleration plugin to be run | ||
| * independently of other plugins. | ||
| * | ||
| * Note also that we key this record by (parentNode, labelHash) to stay on Ponder's prefetch hot-path, | ||
|
shrugs marked this conversation as resolved.
|
||
| * which requires that the key of the entity be trivially derived from event arguments. Because this | ||
| * record is consulted in the context of the ENSv1RegistryOld#NewOwner event (which emits both | ||
| * `parentNode` and `labelHash` directly), keying by (parentNode, labelHash) lets Ponder's profile | ||
| * pattern matcher recover the key from event args. See the helper module's block comment for the | ||
| * full rationale. | ||
| * | ||
| * The ensv2 plugin depends on the Protocol Acceleration plugin in order to piggyback on this | ||
| * Registry migration logic. | ||
| */ | ||
| export const migratedNodeByParent = onchainTable( | ||
|
shrugs marked this conversation as resolved.
|
||
| "migrated_nodes_by_parent", | ||
| (t) => ({ | ||
| // keyed by (parentNode, labelHash) | ||
| parentNode: t.hex().notNull().$type<Node>(), | ||
| labelHash: t.hex().notNull().$type<LabelHash>(), | ||
| }), | ||
| (t) => ({ | ||
| pk: primaryKey({ columns: [t.parentNode, t.labelHash] }), | ||
| }), | ||
| ); | ||
|
|
||
| /** | ||
| * Sibling lookup-by-namehash table for {@link migratedNodeByParent}. Indexed by `node` so that | ||
| * ENSv1RegistryOld#Transfer/NewTTL/NewResolver — which emit only `node` — can read migration | ||
| * status on Ponder's prefetch hot-path. Existence in this table is equivalent to existence in | ||
| * {@link migratedNodeByParent}; both are written together by the migration helper. | ||
| */ | ||
| export const migratedNodeByNode = onchainTable("migrated_nodes_by_node", (t) => ({ | ||
| node: t.hex().notNull().primaryKey().$type<Node>(), | ||
|
shrugs marked this conversation as resolved.
Outdated
|
||
| })); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.