[Feature][Connector-V2] Expose CDC source identifier metadata#11401
[Feature][Connector-V2] Expose CDC source identifier metadata#11401wipheg wants to merge 2 commits into
Conversation
3ff6587 to
25008db
Compare
DanielLeens
left a comment
There was a problem hiding this comment.
Good job.
Thanks a lot for your valuable contribution to Apache SeaTunnel!
I reviewed the full current diff from the CDC source metadata-schema injection down to the Debezium and MongoDB row deserializers.
What this PR fixes
- User pain: today
Metadatacan exposeDatabaseandTable, but CDC users still cannot project source identifiers like relational schema or owner, or MongoDB collection or namespace, as first-class fields. - Fix approach: add stable
Schema,Collection, andNamespacemetadata keys, populate them on the relevant CDC deserialization paths, and document the connector coverage in EN/ZH metadata docs. - One-line summary: the identifier metadata contract is now more explicit and I do not see a source-side blocker on the current head.
Runtime path checked
relational CDC path
-> IncrementalSource.updateCatalogTableMetadata()
-> inject source-identifier metadata columns for connectors that expose them
-> SeaTunnelRowDebeziumDeserializeSchema.deserializeDataChangeRecord()
-> set tableId
-> set Schema metadata when the source record has a relational schema or owner
MongoDB CDC path
-> MongoDBConnectorDeserializationSchema.deserialize()
-> derive database and collection from the source namespace
-> keep tableId on the row
-> populate Collection and Namespace metadata
Findings
- I do not see a reopened source-level blocker on the latest head.
- The new CDC deserializer tests and metadata-transform tests cover the main new identifiers on the current diff.
- The current remaining gate is CI: the
Buildworkflow is stillaction_requiredon this first-time-contributor PR:
https://github.com/apache/seatunnel/runs/86429299781
Merge conclusion
Conclusion: can merge after CI is enabled and passes
Blocking items:
- Enable the PR workflow in the fork and rerun
Build.
Suggested follow-up:
- None from Daniel's current source review.
If you run into any trouble enabling CI or hit a follow-up workflow error after that, feel free to leave a comment here. Happy to take another look once the checks are running.
|
Rechecked the current head Runtime path recheckedMerge conclusionConclusion: source blocker cleared from Daniel's side; sync latest
|
25008db to
5eb5f54
Compare
|
Rechecked the latest head again from the CDC metadata injection path. From Daniel's side, I do not see a current code-level blocker in this metadata expansion. The new fields are additive, the existing metadata semantics are not being redefined, and the source/deserializer updates line up with that scope. The current remaining gate is the red |
Purpose of this pull request
Closes #11042.
This PR exposes structured CDC source identifiers as stable metadata fields.
It adds
Schema,Collection, andNamespacemetadata keys and wires them into the existing CDC metadata projection path.Connector coverage:
Database,TableDatabase,Schema,TableDatabase,Collection,NamespaceMissing identifiers remain absent instead of being filled with placeholders or duplicated values.
Does this PR introduce any user-facing change?
Yes.
CDC users can now project structured source identifiers through the
Metadatatransform.Relational CDC example:
MongoDB-CDC example:
The metadata contract and connector coverage matrix are documented in both English and Chinese docs.
How was this patch tested?
Added focused tests:
MetadataTransformTestSeaTunnelRowDebeziumDeserializeSchemaTestMongoDBConnectorDeserializationSchemaTestRan:
Full
./mvnw -q -DskipTests verifywas attempted locally, but it failed in unrelated existing E2E/JDBC modules.Check list
New License Guide
incompatible-changes.mdto describe the incompatibility caused by this PR.