/kind bug
What happened?
NodeGetVolumeStats RPC can fail on Windows with an unexpected "The system cannot find the path specified" error:
E1006 17:17:10.351024 5192 driver.go:112] "GRPC error" err="rpc error: code = Internal desc = failed to get fs info on path c:\var\lib\kubelet\pods\89096d8e-51de-4fec-b892-e413a5ee1349\volumes\kubernetes.io~csi\pvc-2a818845-d016-4896-9b78-4106dcd13a04\mount: The system cannot find the path specified."
Example logs: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_aws-ebs-csi-driver/2666/pull-aws-ebs-csi-driver-test-e2e-external-eks-windows/1975242407234506752
What you expected to happen?
NodeGetVolumeStats should successfully retrieve volume statistics, or return gracefully.
How to reproduce it (as minimally and precisely as possible)?
- Create a pod with a dynamically provisioned Windows volume.
- Expand the PVC.
- Wait for controller resize to complete.
- Delete the pod.
- Creating a new pod with the same volume on the same node.
Anything else we need to know?:
The current implementation uses windows.GetDiskFreeSpaceEx directly:
func (m *NodeMounter) GetVolumeStats(volumePath string) (VolumeStats, error) {
stats := VolumeStats{}
var availableBytes, totalBytes, totalFreeBytes uint64
err := windows.GetDiskFreeSpaceEx(
windows.StringToUTF16Ptr(volumePath),
&availableBytes,
&totalBytes,
&totalFreeBytes,
)
if err != nil {
return VolumeStats{}, err
}
// ...
}This code path seems fragile and there is a TODO. As part of this fix, I suggest considering refactoring it to use the official CSI proxy GetVolumeStats API instead, ie:
func (m *NodeMounter) getVolumeStatsV1(proxyMounter *CSIProxyMounter, volumePath string) (VolumeStats, error) {
return getVolumeStatsFromCSIProxy(func() (int64, int64, error) {
idReq := &volumev1.GetVolumeIDFromTargetPathRequest{TargetPath: util.NormalizeWindowsPath(volumePath)}
idResp, err := proxyMounter.VolumeClient.GetVolumeIDFromTargetPath(context.Background(), idReq)
if err != nil {
return 0, 0, err
}
statsReq := &volumev1.GetVolumeStatsRequest{VolumeId: idResp.VolumeId}
statsResp, err := proxyMounter.VolumeClient.GetVolumeStats(context.Background(), statsReq)
if err != nil {
return 0, 0, err
}
return statsResp.TotalBytes, statsResp.UsedBytes, nil
})
}
/kind bug
What happened?
NodeGetVolumeStats RPC can fail on Windows with an unexpected "The system cannot find the path specified" error:
Example logs: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_aws-ebs-csi-driver/2666/pull-aws-ebs-csi-driver-test-e2e-external-eks-windows/1975242407234506752
What you expected to happen?
NodeGetVolumeStatsshould successfully retrieve volume statistics, or return gracefully.How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?:
The current implementation uses
windows.GetDiskFreeSpaceExdirectly:This code path seems fragile and there is a TODO. As part of this fix, I suggest considering refactoring it to use the official CSI proxy
GetVolumeStatsAPI instead, ie: