Azure Files SMB / Blobfuse / gcsfuse: restore silently loses file ownership — chown/chmod fake success, no error exists to surface

[kaovilai]

Note

Responses generated with Claude

Important

Resolution scope: documentation. Attribute storage on these filesystems is impossible by design — no Velero/kopia code change can make ownership/mode restore work here. All detection work (post-restore outcome verification and upfront mount-type warning) is consolidated under #10043. This issue closes via a docs PR capturing the verified guidance below.

Summary

On filesystems with mount-constant identity (Azure Files SMB/CIFS, Blob via blobfuse), kopia restore's chown returns success while changing nothing — for root callers, the data-mover case:

# as root, on cifs (...,uid=0,forceuid,gid=0,forcegid,...):
chown 1001:1001 /data/apps/a.txt
chown exit=0            ← success reported
stat: uid=0 gid=0       ← nothing changed

Restored files take the mount's forced owner; the restore reports Completed with zero warnings; the data-path pod log contains zero error lines. On blobfuse, chmod is also a silent no-op — mode bits are lost too. Non-root callers get a real EPERM from the same mounts, which is why app entrypoints log Operation not permitted while Velero stays silent.

Same symptom as #10040 but a distinct mechanism: no error ever exists, so IgnorePermissionErrors (that issue's root cause) is not involved, and a fail-loud build reports a clean restore here too. Error-path fixes cannot catch this class.

Verified scope (probes 2026-07-20, AKS 1.33.13, kernel 5.15)

Volume type	mount	chown as root	chmod as root
Azure Files SMB, uid=/gid= mountOptions	cifs, forceuid/forcegid, nounix	exit 0, no-op	forced by file_mode=/dir_mode=
Azure Files SMB, no uid options	cifs, uid=0,noforceuid,... (driver defaults)	exit 0, no-op — no cached in-session change either	same
Azure Files SMB, idsfromsid,modefromsid	ids/modes stored in NTFS security descriptors	works, persists server-side, survives remount	works, persists
Blob blobfuse (protocol: fuse)	blobfuse2 FUSE	exit 0, no-op	exit 0, no-op — chmod 640 left 755
Blob NFSv3 (protocol: nfs, Premium)	nfs vers=3	works (probed)	works (probed)
Azure Disk (block)	real POSIX	works	works
Azure Files NFS 4.1 (protocol: nfs, Premium)	nfs4 vers=4.1	works — full Velero e2e fidelity verified	works
Azure Files NFS 4.1 + rootSquashType: RootSquash	same	real EPERM; creates land as anon uid 4294967294 — reproduces #10040 variant 1 on Azure (and Velero would swallow it)	works

CIFS behavior is identical with and without explicit uid=/gid= (driver defaults to uid=0,gid=0), so every Azure Files SMB PVC without idsfromsid,modefromsid is affected. No "sticks until remount" mode exists.

Verified SMB-native remediation: idsfromsid,modefromsid

An SC with these two mountOptions (and no uid/gid/mode forcing) gives real per-file fidelity — verified end-to-end with Velero fs-backup: uid-1001 workload with mixed modes (770/755/640/644) backed up, deleted, restored — every uid/gid/mode exact. Caveats:

1. Fresh share root gets a quirky SD translation (uid=0 gid=<fsGroup> mode=1707 — empty group bits beat the permissive other bits): non-root workloads need one root init-container chown/chmod of the volume root (which genuinely works on this mount).
2. Real fidelity means the restored volume root's restrictive owner/mode locks out Velero's restore-wait init container unless its identity matches the workload — see fs-backup restore deadlocks at Init:0/1 on owner-restricted volumes — restore-wait init container ignores pod-level securityContext, falls back to hardcoded runAsUser 1000 #10046 (hit twice during verification; the global fs-restore-action-config workaround had to be flipped per workload uid).

Same class on other platforms (documented, not probed)

Storage (CSI driver)	Owner stored?	chown / chmod	Notes
GCP Cloud Storage FUSE (gcsfuse.csi.storage.gke.io)	No — mount-time uid/gid/modes	not supported — silent-loss class	docs: "chmod/chown/chgrp are not supported". The blobfuse twin; common GKE RWX target
AWS FSx for Windows (SMB)	NTFS ACLs	chown/chmod don't map	the Azure Files CIFS twin
AWS EFS Access Points w/ PosixUser (efs.csi.aws.com)	Yes, but AP overrides uid/gid for all ops	chown neutralized server-side	docs; common Velero target on EKS. Plain EFS is real POSIX
AWS Mountpoint for S3 (s3.csi.aws.com)	No	explicit error by design (SEMANTICS.md)	Loud — but an EPERM/EACCES errno would still be swallowed by #10040's flag, converging back to silent loss
GCP Filestore (NFS)	Yes	works; root_squash (default) → real EPERM	#10040's family
goofys	No	silent no-op	s3fs-fuse and JuiceFS persist attributes; unaffected

Mountpoint-S3 shows the two bugs compounding — a filesystem erroring correctly still ends silent if the errno matches os.IsPermission. One more argument for outcome verification (#10043).

Real-world impact (verified with PostgreSQL 16 on AKS)

The app never depended on stored attributes — only on mount configuration fabricating what its SC options say. The backup captures those fabricated values; on restore, the destination mount presents its own constants regardless of what kopia does. A perfect restore tool would fail identically: "restore corrupted my permissions" is really "destination SC differs from source".

• Same-SC restore: harmless. postgres:16 on a tuned SMB SC (uid=999,gid=999,dir_mode=0700,file_mode=0600,nobrl) → backup → delete → restore → starts, data intact, PGDATA presents 999/700 again.
• Cross-mount-identity restore: hard failure, silent restore. Same backup remapped (change-storage-class-config) onto a uid=0,0777 SC → restore Completed, zero warnings — pod crash-loops:

  FATAL:  data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
  HINT:   The server must be started by the user that owns the data directory.
  

  The recognizable field signature: stat-checking apps (postgres, mysql, sshd) fail; access()-only apps keep running under permissive forced modes.
• Backups from these volumes never contained real ownership. Fabricated identity is what kopia snapshots; restoring to real-POSIX storage writes it literally. Fidelity is lost at backup time. (Exception: idsfromsid,modefromsid SCs.)
• CSI data mover verified identical. --snapshot-move-data backup restored via DataDownload onto a forced-root SC: 1001 → 0:0, restore Completed, no volume warnings — both paths share the affected Restore().

Resolution (documentation)

Site docs (fs-backup / data-mover limitations + Azure guidance) should capture:

1. Limitation statement: on mount-constant-identity volumes (Azure Files SMB without idsfromsid,modefromsid, blobfuse, gcsfuse, FSx-Windows, EFS Access Points with PosixUser), file ownership — and on FUSE variants, mode bits — cannot be restored by any backup tool; chown/chmod are silent no-ops for root. Backups taken from such volumes carry fabricated attributes.
2. Azure Files SMB remediation (verified): idsfromsid,modefromsid mountOptions for full fidelity (note the share-root init-chown caveat and fs-backup restore deadlocks at Init:0/1 on owner-restricted volumes — restore-wait init container ignores pod-level securityContext, falls back to hardcoded runAsUser 1000 #10046 interaction), or protocol: nfs Premium (avoid rootSquashType: RootSquash — that lands in NFS: restore silently loses file ownership — server rejects chown (root_squash / idmap EPERM) and hardcoded IgnorePermissionErrors swallows the error #10040's EPERM class), or block storage.
3. No remediation exists for blobfuse/gcsfuse — state it plainly; recommend against fidelity-sensitive data there.
4. Triage guidance: mount | grep cifs (or fuse) with uid=/gid= options in the pod identifies this class; the postgres "wrong ownership" crash after a clean restore is its signature.

Detection is out of scope here — both the post-restore outcome verification and the upfront per-volume mount-type warning are consolidated in #10043, which references this issue for the fake-success evidence.

Environment

• Velero upstream/main d59e7d4eb, kopia uploader — fs-backup and CSI data mover both verified end-to-end, sharing Restore() in pkg/uploader/kopia/snapshot.go
• AKS 1.33.13; file.csi.azure.com SMB 3.1.1 + NFS 4.1; blob.csi.azure.com blobfuse2 + NFSv3; kernel 5.15
• Related: NFS: restore silently loses file ownership — server rejects chown (root_squash / idmap EPERM) and hardcoded IgnorePermissionErrors swallows the error #10040 (NFS swallowed-EPERM), Verify ownership/mode after kopia restore and surface mismatches as restore warnings — detects both NFS EPERM-swallow (#10040) and SMB/FUSE fake-success (#10044) #10043 (all detection work), fs-backup restore deadlocks at Init:0/1 on owner-restricted volumes — restore-wait init container ignores pod-level securityContext, falls back to hardcoded runAsUser 1000 #10046 (restore-wait identity deadlock, found during verification), Velero Restore Does Not Preserve Original File Permissions on PVCs on AZURE #9398/Velero Restore Does Not Preserve Original File Permissions on PVCs on AWS EKS Cluster #9399 (Azure reports likely this variant)