/kind bug
What happened?
While enabling KEP-4876: Mutable CSINode Allocatable Property on a self-managed Kubernetes cluster (RKE2), the CSINode allocatable volume count does not update dynamically as expected.
The AWS EBS CSI driver (helm.sh/chart: aws-ebs-csi-driver-2.55.1) is configured with:
nodeAllocatableUpdatePeriodSeconds: 10
MutableCSINodeAllocatableCount feature gate enabled on both kube-apiserver and kubelet components
We have reserved volume count reservedVolumeAttachments: 3
The node metadata hop limit is set to 2.
The instance type we are using is - m5.4xlarge
Although NodeGetInfo is being invoked periodically, it always returns a static value:
maxVolumesPerNode=23
Even when additional out-of-band EBS volumes are attached to the node (outside Kubernetes), this value does not change.
Observations:
The driver only accounts for ENI-based limits and does not consider externally attached EBS volumes.
As a result, CSINode.spec.allocatable.count remains fixed at 23.
The metadata labeler updates the node label:
ebs.csi.aws.com/non-csi-ebs-volumes-count
which correctly reflects external volume usage, but:
This label is not used by NodeGetInfo
Its updates are not periodic/deterministic, so it cannot be relied upon for scheduler decisions.
This prevents effective use of the dynamic allocatable feature introduced in the KEP.
What you expected to happen?
NodeGetInfo should return a dynamic maxVolumesPerNode value that accounts for:
Kubernetes-managed EBS volumes
Out-of-band (externally attached) EBS volumes
Reserved volume attachments
CSINode.spec.allocatable.count should be updated periodically based on actual node capacity.
The scheduler should make placement decisions based on real-time volume capacity, as intended by KEP-4876.
How to reproduce it (as minimally and precisely as possible)?
Set up a self-managed Kubernetes cluster (RKE2) with Kubernetes v1.35.
Install AWS EBS CSI driver:
Chart: aws-ebs-csi-driver-2.55.1
Configure:
nodeAllocatableUpdatePeriodSeconds: 10
reservedVolumeAttachments: 3
Ensure IMDS hop limit is set to 2.
Attach additional EBS volumes to the node outside Kubernetes (out-of-band).
Monitor logs:
kubectl -n kube-system logs -l app=ebs-csi-node -c ebs-plugin | grep -Ei "NodeGetInfo|allocatable"
Observe:
NodeGetInfo is called periodically
maxVolumesPerNode remains constant (e.g., 23)
Check:
CSINode.spec.allocatable.count → remains unchanged
Node label ebs.csi.aws.com/non-csi-ebs-volumes-count → updates inconsistently and is not used
Anything else we need to know?
The current implementation appears to not integrate external volume awareness into NodeGetInfo, which limits the usefulness of KEP-4876 for AWS EBS.
The metadata labeler provides partial visibility but:
Is not synchronized with NodeGetInfo
Does not update at a fixed interval
This creates a gap where:
Dynamic capacity exists in theory (via KEP)
But is not practically usable with AWS EBS CSI driver
Question:
Is there any planned support or recommended approach to:
Incorporate out-of-band EBS volumes (attached outside Kubernetes) into NodeGetInfo?
Use metadata labeler or node-level signals for dynamic allocatable calculation?
We observed that IMDS (http://169.254.169.254/latest/meta-data/block-device-mapping/) only returns launch-time mappings (root/ami/ephemeral) and does not reflect runtime-attached EBS volumes.
In contrast, EC2 API shows the actual attached volumes (e.g., 26), so IMDS cannot be used for real-time volume count.
Question:
What is the recommended approach for CSI drivers (AWS EBS CSI) to handle dynamic volume state for mutable CSINode allocatable (KEP-4876)?
Is querying the EC2 API the only supported approach?
Are there any plans to support real-time attachment awareness in the driver?
Environment
Kubernetes version (use kubectl version):
v1.35.x (self-managed via RKE2)
Driver version:
aws-ebs-csi-driver-2.55.1
/kind bug
What happened?
While enabling KEP-4876: Mutable CSINode Allocatable Property on a self-managed Kubernetes cluster (RKE2), the CSINode allocatable volume count does not update dynamically as expected.
The AWS EBS CSI driver (helm.sh/chart: aws-ebs-csi-driver-2.55.1) is configured with:
nodeAllocatableUpdatePeriodSeconds: 10
MutableCSINodeAllocatableCount feature gate enabled on both kube-apiserver and kubelet components
We have reserved volume count reservedVolumeAttachments: 3
The node metadata hop limit is set to 2.
The instance type we are using is - m5.4xlarge
Although NodeGetInfo is being invoked periodically, it always returns a static value:
maxVolumesPerNode=23
Even when additional out-of-band EBS volumes are attached to the node (outside Kubernetes), this value does not change.
Observations:
The driver only accounts for ENI-based limits and does not consider externally attached EBS volumes.
As a result, CSINode.spec.allocatable.count remains fixed at 23.
The metadata labeler updates the node label:
ebs.csi.aws.com/non-csi-ebs-volumes-count
which correctly reflects external volume usage, but:
This label is not used by NodeGetInfo
Its updates are not periodic/deterministic, so it cannot be relied upon for scheduler decisions.
This prevents effective use of the dynamic allocatable feature introduced in the KEP.
What you expected to happen?
NodeGetInfo should return a dynamic maxVolumesPerNode value that accounts for:
Kubernetes-managed EBS volumes
Out-of-band (externally attached) EBS volumes
Reserved volume attachments
CSINode.spec.allocatable.count should be updated periodically based on actual node capacity.
The scheduler should make placement decisions based on real-time volume capacity, as intended by KEP-4876.
How to reproduce it (as minimally and precisely as possible)?
Set up a self-managed Kubernetes cluster (RKE2) with Kubernetes v1.35.
Install AWS EBS CSI driver:
Chart: aws-ebs-csi-driver-2.55.1
Configure:
nodeAllocatableUpdatePeriodSeconds: 10
reservedVolumeAttachments: 3
Ensure IMDS hop limit is set to 2.
Attach additional EBS volumes to the node outside Kubernetes (out-of-band).
Monitor logs:
kubectl -n kube-system logs -l app=ebs-csi-node -c ebs-plugin | grep -Ei "NodeGetInfo|allocatable"
Observe:
NodeGetInfo is called periodically
maxVolumesPerNode remains constant (e.g., 23)
Check:
CSINode.spec.allocatable.count → remains unchanged
Node label ebs.csi.aws.com/non-csi-ebs-volumes-count → updates inconsistently and is not used
Anything else we need to know?
The current implementation appears to not integrate external volume awareness into NodeGetInfo, which limits the usefulness of KEP-4876 for AWS EBS.
The metadata labeler provides partial visibility but:
Is not synchronized with NodeGetInfo
Does not update at a fixed interval
This creates a gap where:
Dynamic capacity exists in theory (via KEP)
But is not practically usable with AWS EBS CSI driver
Question:
Is there any planned support or recommended approach to:
Incorporate out-of-band EBS volumes (attached outside Kubernetes) into NodeGetInfo?
Use metadata labeler or node-level signals for dynamic allocatable calculation?
We observed that IMDS (http://169.254.169.254/latest/meta-data/block-device-mapping/) only returns launch-time mappings (root/ami/ephemeral) and does not reflect runtime-attached EBS volumes.
In contrast, EC2 API shows the actual attached volumes (e.g., 26), so IMDS cannot be used for real-time volume count.
Question:
What is the recommended approach for CSI drivers (AWS EBS CSI) to handle dynamic volume state for mutable CSINode allocatable (KEP-4876)?
Is querying the EC2 API the only supported approach?
Are there any plans to support real-time attachment awareness in the driver?
Environment
Kubernetes version (use kubectl version):
v1.35.x (self-managed via RKE2)
Driver version:
aws-ebs-csi-driver-2.55.1