desing question ceph tentacl 20 with nvme/tcp with vmware

[zouhairnador]

We are planning to deploy a new Ceph cluster for block storage workloads using NVMe/TCP, and we would like to validate whether our design is production-ready.

Below are the planned technical design aspects along with our questions.

Hardware Specifications (Target Bill of Materials)

For a 6-node NVMe-optimized cluster, we are leveraging the high PCIe lane count of AMD EPYC CPUs.

The cluster will be dedicated to block storage workloads (NVMe/TCP) for a VMware vSphere environment.

• OS: Ubuntu 24.04.4 LTS

• Ceph Version: Tentacle 20.2.0

Component | Specification (Per Node) | Reasoning -- | -- | -- CPU | AMD EPYC 9535 (64 cores, 128 PCIe Gen5 lanes/socket) | Enough headroom for 12 OSDs + 8 reactor cores + possible deduplication overhead RAM | 256GB DDR5-6400 ECC | Targeting ~10GB RAM per OSD for performance Storage | 12 × 7.68TB NVMe Gen5 | Enterprise-grade SSDs (DWPD ≥ 1) Network | 2 × Dual-Port 25GbE SFP28 | Separation of Public and Cluster traffic

Questions

9. What is your feedback regarding these tuning choices?
   Are there any risks, production caveats, or kernel stability implications with these settings?

NVMe/TCP Multipathing & Active/Active Paths

Compared to traditional enterprise storage arrays, Ceph NVMe/TCP currently appears limited regarding true active/active path utilization for the same namespace/LUN (ANA group behavior).

Questions

10. From the Ceph side, how can we achieve better active/active path utilization for the same namespace before tuning the VMware vSphere side?
    Are there recommended NVMe-oF gateway designs, ANA configurations, or load-balancing approaches for this?

Traffic Separation for NVMe/TCP

For the NVMe/TCP target interfaces exposed to ESXi hosts, we are using dedicated listener IPs on a separate VLAN reserved only for storage traffic, isolated from MON/Gateway public IPs.

Questions

11. Is this architecture aligned with current best practices for Ceph NVMe/TCP environments?
    Have you seen similar production deployments, and would you recommend any additional isolation or routing considerations?

[caroav]

Regarding 'active/active path utilization' I don't believe it imposes any significant limitations. When multiple namespaces are used, traffic is distributed across the gateways because the namespaces are assigned evenly across ANA groups. Based on our testing, the performance achieved on a single path is more than sufficient. While any active/active solution must eventually serialize I/O through one of the controllers (nodes) to maintain data integrity, we do plan to implement full active/active support - primarily to rule out concerns of this nature.

HW specification sounds very good.

We have some best practices for ESX:

• Allocate more of the smaller datastores instead of a few large datastores. This approach does not carry capacity cost as Ceph has thin provisions by default and helps ensure that Ceph I/O contexts are available for the gateway.
• Consider isolating applications with high I/O demands to use a dedicated datastore to ensure adequate host adapter queue depth.
• Use jumbo frames when possible for ESXi to Gateway network links. Using jumbo frames is beneficial for workloads that use larger I/O block sizes

[zouhairnador]

thank you for replaying,
whene talking about the active/active is the behavioure from the vmware multipathing perspective that need to roudrobing or loadbased between pathes to the same Lun (say we have only one namespace) , so same LUN (subsysteme or RBD image) will be requested from diferente VMs or ESXi each time (each session) from diferente path to distribute the load to support multi Read write with better performence .
and the second question about the A/A needs its the HA of the GW , say we have only 3 NVME/TCP GW and we need to use theme all by roudrobind between them from differents esxi with having each time a standby path unusable. so this can help on reducing down time if GW goes down .
is this possible ? (3 GWs one ANA group , one GW group)

[caroav]

In case one (or more) of the GWs goes down, failover is done within seconds. Assuming this is not a scenario that happens very often, I don't think it should be a big concern. Even if lower levels in ceph fail (e.g. one of the OSDs is down), you might wait few seconds - regardless to A/A in nvmeof.
As to the other concern you raised here. It is better ofcourse to be able to always have access via all paths, but in practice, I don't think that it will make much difference. Did you try and actually saw this issue?