DAOS-18928 dfuse: increase MAX_DAOS_MT to 32 by mchaarawi · Pull Request #18526 · daos-stack/daos

mchaarawi · 2026-06-23T20:19:30Z

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

daosbuild3 · 2026-06-23T21:14:06Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18526/2/execution/node/582/log

github-actions · 2026-06-23T22:05:35Z

Ticket title is 'pil4dfs: increase MAX_DAOS_MT to 32'
Status is 'In Progress'
Labels: 'triaged'
https://daosio.atlassian.net/browse/DAOS-18928

Also prevent a crash if we go beyond MAX_DAOS_MT by just disabling interception in that case. Features: pil4dfs Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

daosbuild3 · 2026-06-24T04:03:43Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18526/4/execution/node/1027/log

daosbuild3 · 2026-06-24T09:35:21Z

Test stage Functional on EL 9 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18526/5/testReport/

daosbuild3 · 2026-06-24T11:17:08Z

Test stage Functional Hardware Large MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18526/5/testReport/

daosbuild3 · 2026-06-24T13:20:20Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18526/5/execution/node/1345/log

daosbuild3 · 2026-06-24T19:19:01Z

Test stage Functional on EL 9 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18526/6/testReport/

daltonbohning

I pushed most of my feedback here if you want to use it
https://github.com/daos-stack/daos/compare/mschaara/18928...dbohning/mschaara/18928?expand=1

daltonbohning · 2026-06-24T19:58:49Z

+        finally:
+            # Unmount this case's dfuse instances so the next case only sees its own mounts.
+            for dfuse in dfuses:
+                dfuse.stop()


Since we know this runs with 10, 32, then 33 instances, do we want to optimize this to reuse the containers and mounts? I.e.

Create and mount 10 containers

Verify behavior

Create and mount additional containers up to 32

Verify behavior

Create and mount additional containers up to 33

Verify behavior

I could make that change if you agree

good call.. i will make the change.
will wait for all testing to complete first and will post following PR with test only changes to address your feedback.

kevin-harms · 2026-06-25T09:04:09Z


 /* The max number of mount points for DAOS mounted simultaneously */
-#define MAX_DAOS_MT         (8)
+#define MAX_DAOS_MT         (32)


Can we not make this 128 or 256? The cost of this is what, a few KB? 32 is still in the realm of possibility of mounts on a common login.

I suggest increasing the size to something that will likely not happen. Otherwise you'll just hit this in the future and be annoyed again.

discussed with Kevin offline. the change made to just disable interception once > 32 mounts are there is sufficient and this won't abort apps as before and cause problems.

Quick-Functional: true Test-tag: test_pil4dfs_many_mounts Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi · 2026-06-25T15:47:04Z

only failure in https://jenkins-3.daos.hpc.amslabs.hpecorp.net/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-18526/6/tests was the known issue with the dfuse build test.

repushed with only test change using test-tag to avoid running all CI

daltonbohning · 2026-06-25T20:11:54Z

+hosts:
+  test_servers: 1
+  test_clients: 1
+timeout: 900


Only if you need to push again: could reduce this to 600 since the actual execution time was ~6 minutes

Suggested change

timeout: 900

timeout: 600

sigh.. i did repush but forgot about this

knard38

From my understanding, the behaviour of discover_daos_mount_with_env() should also be changed to be consistent with the change done to discover_dfuse_mounts(). The same num_dfs >= MAX_DAOS_MT test returns an error that triggers an abort() via the caller init_myhook(). If the previous call to discover_dfuse_mounts() fills the last slot, the call to discover_daos_mount_with_env() will abort.

There is an additional ordering issue: the overflow guard fires before the dedup check (query_dfs_mount at L603). This means that even when D_IL_MOUNT_POINT points to a mount already registered by discover_dfuse_mounts() — so no new slot would be needed — the function aborts before it can detect that.

If I am correct, do you think the fix below (dedup first, then graceful warn-and-skip) would address this properly?

/* move dedup before the overflow guard */
idx = query_dfs_mount(fs_root);
if (idx >= 0)
    D_GOTO(out, rc = 0);   /* already registered — no new slot needed */

if (num_dfs >= MAX_DAOS_MT) {
    D_WARN("D_IL_MOUNT_POINT ignored: table full (%d mounts). "
           "Increase MAX_DAOS_MT or reduce simultaneous mounts.\n",
           MAX_DAOS_MT);
    D_GOTO(out, rc = 0);   /* graceful skip, interception disabled by caller */
}

knard38 · 2026-06-26T09:05:26Z

-			abort();
-		}
-		pt_dfs_mt = &dfs_list[num_dfs];
 		if (memcmp(fs_entry->mnt_type, STR_AND_SIZE(MNT_TYPE_FUSE)) == 0) {


Nit

Suggested change

if (memcmp(fs_entry->mnt_type, STR_AND_SIZE(MNT_TYPE_FUSE)) == 0) {

if (memcmp(fs_entry->mnt_type, STR_AND_SIZE(MNT_TYPE_FUSE)) != 0)

continue;

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi · 2026-06-26T13:34:52Z

From my understanding, the behaviour of discover_daos_mount_with_env() should also be changed to be consistent with the change done to discover_dfuse_mounts(). The same num_dfs >= MAX_DAOS_MT test returns an error that triggers an abort() via the caller init_myhook(). If the previous call to discover_dfuse_mounts() fills the last slot, the call to discover_daos_mount_with_env() will abort.

There is an additional ordering issue: the overflow guard fires before the dedup check (query_dfs_mount at L603). This means that even when D_IL_MOUNT_POINT points to a mount already registered by discover_dfuse_mounts() — so no new slot would be needed — the function aborts before it can detect that.

If I am correct, do you think the fix below (dedup first, then graceful warn-and-skip) would address this properly?
/* move dedup before the overflow guard */
idx = query_dfs_mount(fs_root);
if (idx >= 0)
    D_GOTO(out, rc = 0);   /* already registered — no new slot needed */

if (num_dfs >= MAX_DAOS_MT) {
    D_WARN("D_IL_MOUNT_POINT ignored: table full (%d mounts). "
           "Increase MAX_DAOS_MT or reduce simultaneous mounts.\n",
           MAX_DAOS_MT);
    D_GOTO(out, rc = 0);   /* graceful skip, interception disabled by caller */
}

while i totally agree with you, the env path is a test only scenario that i had long asked Lei to remove since it should never be used in production. but for the sake of consistency, ill push the change since it is quite straighforward.

mchaarawi force-pushed the mschaara/18928 branch from ece2201 to 60600f5 Compare June 23, 2026 20:21

mchaarawi force-pushed the mschaara/18928 branch from 60600f5 to b038095 Compare June 23, 2026 22:11

DAOS-18928 dfuse: increase MAX_DAOS_MT to 32

740d6ce

Also prevent a crash if we go beyond MAX_DAOS_MT by just disabling interception in that case. Features: pil4dfs Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi force-pushed the mschaara/18928 branch from b038095 to 740d6ce Compare June 23, 2026 23:42

mchaarawi marked this pull request as ready for review June 24, 2026 19:27

mchaarawi requested review from a team as code owners June 24, 2026 19:27

daltonbohning reviewed Jun 24, 2026

View reviewed changes

kevin-harms reviewed Jun 25, 2026

View reviewed changes

address review feedback

e66f329

Quick-Functional: true Test-tag: test_pil4dfs_many_mounts Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi requested review from daltonbohning and knard38 June 25, 2026 17:30

daltonbohning previously approved these changes Jun 25, 2026

View reviewed changes

knard38 requested changes Jun 26, 2026

View reviewed changes

mchaarawi added 2 commits June 26, 2026 13:19

Merge branch 'master' into mschaara/18928

94bb7ec

address feedback

40ec478

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi dismissed daltonbohning’s stale review via 40ec478 June 26, 2026 13:34

mchaarawi requested review from daltonbohning and knard38 June 26, 2026 20:08

	if (memcmp(fs_entry->mnt_type, STR_AND_SIZE(MNT_TYPE_FUSE)) == 0) {
	if (memcmp(fs_entry->mnt_type, STR_AND_SIZE(MNT_TYPE_FUSE)) != 0)
	continue;

Uh oh!

Conversation

mchaarawi commented Jun 23, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jun 24, 2026

Uh oh!

daosbuild3 commented Jun 24, 2026

Uh oh!

daosbuild3 commented Jun 24, 2026

Uh oh!

daosbuild3 commented Jun 24, 2026

Uh oh!

daosbuild3 commented Jun 24, 2026

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchaarawi commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchaarawi commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented Jun 23, 2026 •

edited

Loading