Skip to content

[Feature][Connector-V2] Expose CDC source identifier metadata#11401

Open
wipheg wants to merge 2 commits into
apache:devfrom
wipheg:feature/11042-cdc-source-identifiers
Open

[Feature][Connector-V2] Expose CDC source identifier metadata#11401
wipheg wants to merge 2 commits into
apache:devfrom
wipheg:feature/11042-cdc-source-identifiers

Conversation

@wipheg

@wipheg wipheg commented Jul 10, 2026

Copy link
Copy Markdown

Purpose of this pull request

Closes #11042.

This PR exposes structured CDC source identifiers as stable metadata fields.

It adds Schema, Collection, and Namespace metadata keys and wires them into the existing CDC metadata projection path.

Connector coverage:

  • MySQL-CDC / TiDB-CDC: Database, Table
  • PostgreSQL-CDC / Opengauss-CDC / Oracle-CDC / SQLServer-CDC: Database, Schema, Table
  • MongoDB-CDC: Database, Collection, Namespace

Missing identifiers remain absent instead of being filled with placeholders or duplicated values.

Does this PR introduce any user-facing change?

Yes.

CDC users can now project structured source identifiers through the Metadata transform.

Relational CDC example:

metadata_fields {
  Database = source_database
  Schema = source_schema
  Table = source_table
}

MongoDB-CDC example:

metadata_fields {
  Database = source_database
  Collection = source_collection
  Namespace = source_namespace
}

The metadata contract and connector coverage matrix are documented in both English and Chinese docs.

How was this patch tested?

Added focused tests:

  • MetadataTransformTest
  • SeaTunnelRowDebeziumDeserializeSchemaTest
  • MongoDBConnectorDeserializationSchemaTest

Ran:

./mvnw spotless:apply
./mvnw -pl seatunnel-transforms-v2,seatunnel-connectors-v2/connector-cdc/connector-cdc-base,seatunnel-connectors-v2/connector-cdc/connector-cdc-mongodb "-Dtest=MetadataTransformTest,SeaTunnelRowDebeziumDeserializeSchemaTest,MongoDBConnectorDeserializationSchemaTest" "-DskipIT" "-DskipE2E" "-DfailIfNoTests=false" "-Dsurefire.failIfNoSpecifiedTests=false" test

Full ./mvnw -q -DskipTests verify was attempted locally, but it failed in unrelated existing E2E/JDBC modules.

Check list

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job.

Thanks a lot for your valuable contribution to Apache SeaTunnel!

I reviewed the full current diff from the CDC source metadata-schema injection down to the Debezium and MongoDB row deserializers.

What this PR fixes

  • User pain: today Metadata can expose Database and Table, but CDC users still cannot project source identifiers like relational schema or owner, or MongoDB collection or namespace, as first-class fields.
  • Fix approach: add stable Schema, Collection, and Namespace metadata keys, populate them on the relevant CDC deserialization paths, and document the connector coverage in EN/ZH metadata docs.
  • One-line summary: the identifier metadata contract is now more explicit and I do not see a source-side blocker on the current head.

Runtime path checked

relational CDC path
  -> IncrementalSource.updateCatalogTableMetadata()
      -> inject source-identifier metadata columns for connectors that expose them
  -> SeaTunnelRowDebeziumDeserializeSchema.deserializeDataChangeRecord()
      -> set tableId
      -> set Schema metadata when the source record has a relational schema or owner

MongoDB CDC path
  -> MongoDBConnectorDeserializationSchema.deserialize()
      -> derive database and collection from the source namespace
      -> keep tableId on the row
      -> populate Collection and Namespace metadata

Findings

  • I do not see a reopened source-level blocker on the latest head.
  • The new CDC deserializer tests and metadata-transform tests cover the main new identifiers on the current diff.
  • The current remaining gate is CI: the Build workflow is still action_required on this first-time-contributor PR:
    https://github.com/apache/seatunnel/runs/86429299781

Merge conclusion

Conclusion: can merge after CI is enabled and passes

Blocking items:

  1. Enable the PR workflow in the fork and rerun Build.

Suggested follow-up:

  • None from Daniel's current source review.

If you run into any trouble enabling CI or hit a follow-up workflow error after that, feel free to leave a comment here. Happy to take another look once the checks are running.

@DanielLeens

Copy link
Copy Markdown
Contributor

Rechecked the current head 25008db065d8 against the latest dev after the CI / merge-gate fact changed.

Runtime path rechecked

relational CDC path
  -> IncrementalSource.updateCatalogTableMetadata()
      -> inject source-identifier metadata columns for connectors that expose them
  -> SeaTunnelRowDebeziumDeserializeSchema.deserializeDataChangeRecord()
      -> set tableId
      -> set Schema metadata when the source record has a relational schema or owner

MongoDB CDC path
  -> MongoDBConnectorDeserializationSchema.deserialize()
      -> derive database and collection from the source namespace
      -> keep tableId on the row
      -> populate Collection and Namespace metadata

Merge conclusion

Conclusion: source blocker cleared from Daniel's side; sync latest dev and rerun CI first

  1. Blocking items
  1. Lowest-cost next step
  • Please sync with the latest dev and rerun the failing gate first.
  • If it still fails on the updated head, I can help narrow down the new failure from there.

Daniel only has READ permission on this repository, so a write-capable maintainer may still need to handle the final approval / dismissal step once the gate is clear.

@wipheg
wipheg force-pushed the feature/11042-cdc-source-identifiers branch from 25008db to 5eb5f54 Compare July 14, 2026 16:03
@DanielLeens

Copy link
Copy Markdown
Contributor

Rechecked the latest head again from the CDC metadata injection path.

From Daniel's side, I do not see a current code-level blocker in this metadata expansion. The new fields are additive, the existing metadata semantics are not being redefined, and the source/deserializer updates line up with that scope.

The current remaining gate is the red Build, not a reopened source-side issue from this review pass. Because this branch is already up to date with dev, I would not treat this as a "sync latest dev" case first. The lowest-cost next step is to rerun / inspect the current failing CI lanes directly and see whether they reproduce on the same head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Connector-V2][CDC] Expose structured source identifiers as metadata fields

3 participants