[#10805] Support bulk access control operations by jarredhj0214 · Pull Request #11906 · apache/gravitino

jarredhj0214 · 2026-07-06T02:43:37Z

What changes were proposed in this pull request?

This PR adds bulk access control APIs for users, groups, and roles under a metalake:

POST /api/bulk/metalakes/{metalake}/users/add
POST /api/bulk/metalakes/{metalake}/users/remove
POST /api/bulk/metalakes/{metalake}/groups/add
POST /api/bulk/metalakes/{metalake}/groups/remove
POST /api/bulk/metalakes/{metalake}/roles/add
POST /api/bulk/metalakes/{metalake}/roles/remove

The implementation includes:

New REST resources for bulk user, group, and role operations.
Request DTOs for user names, group names, role names, and bulk role creation.
A shared bulk operation response with succeeded and failed results.
Request validation for empty input and duplicated names.
OpenAPI documentation and examples for the new APIs.
Unit tests for request validation and REST behavior.

Why are the changes needed?

Currently, users need to call single-resource APIs repeatedly to add or remove multiple users, groups, or roles. These new APIs provide a batch operation interface for access control management and reduce repeated client-side API calls.

This PR follows the behavior discussed in #10805:

Validate the whole request before processing.
Reject invalid requests, such as duplicated names, without processing any item.
After validation succeeds, process each item independently.
Return successful items in succeeded and per-item business failures in failed.

Closes #10805.

Does this PR introduce any user-facing change?

Yes. This PR introduces new REST APIs for bulk access control operations.

How was this patch tested?

./gradlew :common:test --tests org.apache.gravitino.dto.requests.TestBulkRequests
./gradlew :server:test --tests org.apache.gravitino.server.web.rest.TestBulkOperations -PskipITs
./gradlew :docs:build

roryqi · 2026-07-06T02:47:41Z

@lasdf1234 Could u help me review this pull request?

lasdf1234 · 2026-07-06T02:49:11Z

@lasdf1234 Could u help me review this pull request?

No problem.

lasdf1234 · 2026-07-06T09:27:11Z

Could you add some content to the doc document to explain the usage scenarios of these APIs?

lasdf1234 · 2026-07-06T09:29:28Z

Is the bulk interface better when processing data one by one or in batches? Could you please share your research findings?

jarredhj0214 · 2026-07-07T01:20:12Z

Is the bulk interface better when processing data one by one or in batches? Could you please share your research findings?

Thanks for the review. @lasdf1234

For the first question:

I added a new section named "Bulk Access Control Operations" in docs/security/access-control.md.

The new documentation explains:

Usage scenarios for these APIs, such as team onboarding/offboarding, synchronizing users and groups from an external identity provider, bootstrapping roles for a new environment, and cleaning up obsolete principals.
Required privileges for each bulk API.
Request body fields for users, groups, and roles.
Shell examples for all six bulk APIs.
The response format with succeeded and failed.
The processing semantics: validate the whole request first, then process each item independently after validation succeeds.

For the second question:

I did some research on common batch/bulk API designs. There does not seem to be a single universal rule, but there are several common patterns.

Per-item result with partial success

Microsoft Graph JSON batching returns one top-level batch response, but each individual request has its own status. A top-level 200 only means the batch payload was accepted; clients still need to inspect each item result.
https://learn.microsoft.com/en-us/graph/json-batching
Microsoft Graph throttling documentation also says requests in a batch are evaluated individually, and failed requests should be retried individually.
https://learn.microsoft.com/en-us/graph/throttling
Elasticsearch Bulk API returns a top-level errors flag and an items array. Each operation has its own result, and successful operations are not rolled back when other items fail.
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk
Amazon SQS SendMessageBatch returns Successful and Failed lists. AWS explicitly says the batch can contain both successful and unsuccessful actions, so clients should check batch errors even when HTTP status is 200.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html
Criteo documents a similar model: authentication, authorization, or deserialization errors return 4xx; otherwise the bulk call can return 200 regardless of individual operation success or failure, and clients should inspect the response errors.
https://developers.criteo.com/marketing-solutions/docs/partial-success-for-bulk-operations

Batch request accepted, failed items returned for retry

Amazon DynamoDB BatchWriteItem says each individual put/delete operation is atomic, but the batch as a whole is not. Failed operations are returned as UnprocessedItems, and clients should retry those items, usually with exponential backoff.
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html

Atomic bulk operation

Some API guidelines recommend atomic bulk operations. For example, Adidas API guidelines say a bulk operation should be atomic and should not commit partial changes if any item fails.
https://github.com/adidas/api-guidelines/blob/master/rest-api-guidelines/execution/batch-operations.md

For Gravitino access control APIs, I think the current design is closer to the first pattern and fits this feature better:

We validate the whole request before processing, including empty input, duplicated names, and authorization.
If request-level validation fails, no item is processed.
After the request is valid, each user/group/role operation is processed independently.
The response returns successful items in succeeded and per-item business failures in failed.

This design avoids partial execution for malformed or unauthorized requests, while still allowing clients to make progress and retry only failed items for business-level failures, such as removing a non-existent user.

The current implementation batches at the API contract level and reuses existing per-item access-control managers internally. This keeps the behavior consistent with existing single-item APIs, including existing authorization checks, events, and cache invalidation behavior. Internal storage-level batching can still be optimized later without changing the public API response contract.

github-actions · 2026-07-07T12:28:05Z

Code Coverage Report

Overall Project	67.55% `+0.17%`	🟢
Files changed	80.83%	🟢

Module	Coverage
aliyun	1.72%	🔴
api	46.49%	🟢
authorization-common	85.96%	🟢
aws	42.04%	🟢
azure	2.47%	🔴
catalog-common	9.92%	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.91%	🟢
catalog-hive	79.42%	🟢
catalog-jdbc-clickhouse	80.55%	🟢
catalog-jdbc-common	44.22%	🟢
catalog-jdbc-doris	80.28%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	80.91%	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	58.53%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.94%	🟢
catalog-lakehouse-paimon	84.25%	🟢
catalog-model	77.72%	🟢
cli	44.49%	🟢
client-java	78.01% `+2.68%`	🟢
common	51.74% `-0.4%`	🟢
core	82.58%	🟢
filesystem-hadoop3	77.3%	🟢
flink	0.0%	🔴
flink-common	47.09%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-auth	66.67%	🟢
hadoop-common	12.7%	🔴
hive-metastore-common	53.29%	🟢
iceberg-common	58.3%	🟢
iceberg-rest-server	73.94%	🟢
idp-basic	85.71%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.67%	🔴
lance-rest-server	64.84%	🟢
lineage	53.02%	🟢
optimizer	83.17%	🟢
optimizer-api	21.95%	🔴
server	85.73% `-0.49%`	🟢
server-common	74.62%	🟢
spark	28.57%	🔴
spark-common	46.01%	🟢
tencent	69.84%	🟢
trino-connector	40.29%	🟢

Files

Module	File	Coverage
client-java	GravitinoMetalake.java	92.31%	🟢
	GravitinoClient.java	70.89%	🟢
common	BulkRequestValidator.java	80.0%	🟢
	BulkRoleCreateRequest.java	63.16%	🟢
	GroupNamesRequest.java	52.94%	🔴
	RoleNamesRequest.java	52.94%	🔴
	UserNamesRequest.java	52.94%	🔴
	GroupAddRequest.java	0.0%	🔴
	UserAddRequest.java	0.0%	🔴
	BulkOperationFailureDTO.java	0.0%	🔴
	BulkOperationResponse.java	0.0%	🔴
server	GroupOperations.java	100.0%	🟢
	UserOperations.java	96.08%	🟢
	BulkOperationUtils.java	93.33%	🟢
	BulkRoleOperations.java	84.13%	🟢
	BulkUserOperations.java	82.86%	🟢
	BulkGroupOperations.java	77.14%	🟢
	GravitinoInterceptionService.java	66.37%	🟢

jerryshao · 2026-07-07T12:50:13Z

@jarredhj0214 can you split into several small PRs? The current PR is too large to review.

lasdf1234 · 2026-07-07T13:03:40Z

@roryqi Should the "bulk" scenario also consider the scenario where "externalId" is used as the association key?

roryqi · 2026-07-08T00:37:41Z

@roryqi Should the "bulk" scenario also consider the scenario where "externalId" is used as the association key?

I think yes.

lasdf1234 · 2026-07-08T00:42:00Z

@roryqi Should the "bulk" scenario also consider the scenario where "externalId" is used as the association key?

I think yes.

@jarredhj0214 Coluld you split PR? In addition, the implementation of the externalID scenario needs to be considered.

jarredhj0214 · 2026-07-08T02:13:51Z

@roryqi Should the "bulk" scenario also consider the scenario where "externalId" is used as the association key?

I think yes.

@jarredhj0214 Coluld you split PR? In addition, the implementation of the externalID scenario needs to be considered.

Currently, the existing single user/group/role access-control APIs are name-based and do not support externalId as the association key. This PR follows the current single-operation API contract and keeps the bulk APIs name-based as well.

If we want to support externalId, I think it should be discussed as a separate API/model design because it may require changes to the user/group identity model, request DTOs, storage lookup, and OpenAPI contract, not only the bulk APIs.

jarredhj0214 · 2026-07-08T02:38:50Z

@jarredhj0214 can you split into several small PRs? The current PR is too large to review.

I have split the original large PR into smaller PRs.

The first split PR will include bulk user operations and the common bulk infrastructure. The bulk group and bulk role operations will be submitted as follow-up PRs after the first one is reviewed/merged, so each PR remains easier to review.

For the externalId point: the existing single user/group/role access-control APIs are name-based and do not support externalId as the association key. The split PRs will stay aligned with the current name-based API contract. I think externalId support may require a separate API/model design and can be discussed as a follow-up.

lasdf1234 · 2026-07-08T02:42:00Z

@jarredhj0214 can you split into several small PRs? The current PR is too large to review.

I have split the original large PR into smaller PRs.

The first split PR will include bulk user operations and the common bulk infrastructure. The bulk group and bulk role operations will be submitted as follow-up PRs after the first one is reviewed/merged, so each PR remains easier to review.

For the externalId point: the existing single user/group/role access-control APIs are name-based and do not support externalId as the association key. The split PRs will stay aligned with the current name-based API contract. I think externalId support may require a separate API/model design and can be discussed as a follow-up.

You can update the code. The externalId method has already been implemented.

jarredhj0214 · 2026-07-08T02:43:50Z

The first split PR has been created:

[#10805] Support bulk user operations #11930

It includes bulk user operations and the common bulk infrastructure. I will submit the bulk group and bulk role operations as follow-up PRs after this one is reviewed/merged.

jarredhj0214 · 2026-07-08T03:21:33Z

@jarredhj0214 can you split into several small PRs? The current PR is too large to review.

I have split the original large PR into smaller PRs.
The first split PR will include bulk user operations and the common bulk infrastructure. The bulk group and bulk role operations will be submitted as follow-up PRs after the first one is reviewed/merged, so each PR remains easier to review.
For the externalId point: the existing single user/group/role access-control APIs are name-based and do not support externalId as the association key. The split PRs will stay aligned with the current name-based API contract. I think externalId support may require a separate API/model design and can be discussed as a follow-up.

You can update the code. The externalId method has already been implemented.

I have updated this PR based on the latest externalId and enabled support. Bulk user add now uses UserAddRequest[], so it supports name, externalId, and enabled, consistent with the single user add API. The OpenAPI docs, access-control docs, examples, and tests have also been updated.

The bulk remove API remains name-based, aligned with the existing single delete user API.

The latest changes have been pushed.

jarred0214 added 2 commits July 6, 2026 10:26

[apache#10805] Support bulk access control operations

142027a

Merge remote-tracking branch 'origin/main' into support-bulk-operation

6063b28

[apache#10805] Document bulk access control APIs

5c4f698

jarredhj0214 added 2 commits July 7, 2026 15:14

Merge branch 'main' into support-bulk-operation

db09dfe

Merge branch 'main' into support-bulk-operation

30d5b10

Uh oh!

Conversation

jarredhj0214 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

roryqi commented Jul 6, 2026

Uh oh!

lasdf1234 commented Jul 6, 2026

Uh oh!

lasdf1234 commented Jul 6, 2026

Uh oh!

lasdf1234 commented Jul 6, 2026

Uh oh!

jarredhj0214 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Code Coverage Report

Uh oh!

jerryshao commented Jul 7, 2026

Uh oh!

lasdf1234 commented Jul 7, 2026

Uh oh!

roryqi commented Jul 8, 2026

Uh oh!

lasdf1234 commented Jul 8, 2026

Uh oh!

jarredhj0214 commented Jul 8, 2026

Uh oh!

jarredhj0214 commented Jul 8, 2026

Uh oh!

lasdf1234 commented Jul 8, 2026

Uh oh!

jarredhj0214 commented Jul 8, 2026

Uh oh!

jarredhj0214 commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jarredhj0214 commented Jul 6, 2026 •

edited

Loading

jarredhj0214 commented Jul 7, 2026 •

edited

Loading