HDDS-15193. Move the "atomic key creation" logic from output stream to S3 endpoints#10202
HDDS-15193. Move the "atomic key creation" logic from output stream to S3 endpoints#10202peterxcli wants to merge 8 commits intoapache:masterfrom
Conversation
peterxcli
changed the title
|
cc @xichen01, @ivandika3 Could you please take a look? Thanks! |
|
Thanks @peterxcli for the patch. However, I went back to review the PR for these lines, and it appears they are not related to conditional requests. They were actually added to fix a commit overwrite issue that occurs when multiple S3G instances are writing the same key. I am concerned that removing them, as I don't think that specific problem has been resolved yet. Could you please provide a bit more detail on why these lines need to be deleted? Thanks! |
Signed-off-by: peterxcli <peterxcli@gmail.com>
peterxcli
changed the title
peterxcli
changed the title
peterxcli
changed the title
peterxcli
changed the title
peterxcli
changed the title
There was a problem hiding this comment.
Thanks @peterxcli , overall LGTM. Using the new pre-commit hook is a good idea. Left a few nits.
| if (isS3GRequest.get() && size == 0) { | ||
| openKey.getKeyInfo().setDataSize(0); | ||
| } |
There was a problem hiding this comment.
Is isS3GRequest attributes still used? If not, let's remove it.
Removing the setDataSize(0) seems fine since we are not dependent on the DataSize returned by OpenKey. However, just to be sure, please help check whether the OBJECT_STORE empty file case is already test is covered.
| preCommits.add(validateContentLength( | ||
| length, putLength, keyPath)); |
There was a problem hiding this comment.
Nit: Let's not wrap this in a new line. Also apply in the other similar places.
| @@ -158,7 +160,6 @@ public void close() throws IOException { | |||
| UserGroupInformation.getCurrentUser().getShortUserName(), | |||
| tags | |||
| )); | |||
| super.close(); | |||
There was a problem hiding this comment.
Could you help briefly explain this?
There was a problem hiding this comment.
super.close() runs the output stream pre-commit hooks. Previously the stub recorded the key in keyContents/keyDetails before those hooks ran, so a failing pre-commit could still leave the key visible in the test stub.
…date ClientProtocolStub accordingly. Signed-off-by: peterxcli <peterxcli@gmail.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>
|
I cherry-picked the refactor/fix for the test failure to #10224. the error log is: 2026-05-09 07:21:08,033 [qtp1907241392-109] WARN server.HttpChannel: /bucket-qiyxgiosny/ozone-test-5105985444/ecmultipartKey32
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.ozone.client.io.KeyDataStreamOutput.setPreCommits(java.util.List)" because the return value of "org.apache.hadoop.ozone.client.io.OzoneDataStreamOutput.getKeyDataStreamOutput()" is nullThe root cause of the EC mpu failure is |
3 participants
What changes were proposed in this pull request?
this is a no-op change
the "atomic key creation" logic in Ozone client output stream is actually only for s3g to do the length check.
removes the "atomic key creation" logic from the Ozone client output stream classes and shifts the responsibility for validating S3 object upload content length from the client layer to the S3 gateway layer. The main effect is that S3-specific length checks are now enforced in the S3 gateway, not in the general Ozone client code, leading to a cleaner separation of concerns and more maintainable code.
the name "atomic key creation" is also misleading people to think it's related to atomicity, but it's actually a integrity check for the written block data for s3g. If users want the real atomicity, they must use conditional request:
relate to: #5524
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15193
How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)