Cephfs XAttr missing after Clone Volume (restore), opencloud errors

[suse-coder]

Describe the bug

We use opencloud (latest version) and have cephfs (squid 19.2.0) with 3x replication. When we do a veeam k10 backup and then a restore we see that the files are there but the xattr are missing:

Before the backup and restore with veeam (csi snapshot of cephfs), i see the xattr:

file: var/lib/opencloud/storage/users/users/e7b6cc3f-e407-462f-a972-b183318b92a4/test.txt
user.oc.blobid="fb6a1bd5-eac4-4261-a4da-e3d4879d7a70"
user.oc.blobsize="6"
user.oc.cs.adler32=0sCMsCew==
user.oc.cs.md5=0s1Y41gq+pkEDie5KxPI8igA==
user.oc.cs.sha1=0sVTrn2pL1UFqSu7jJ1Hvnarn2W8I=
user.oc.id="1aa81512-0a69-4a72-9335-4837d8b6adcd"
user.oc.mtime="2025-11-21T19:53:36.111133433Z"
user.oc.name="test.txt"
user.oc.parentid="ecb2d4cf-0ecf-446f-8f11-665273450c01"
user.oc.scandate="2025-11-21T19:53:36.258760181Z"
user.oc.scanstatus=""
user.oc.type="1"

But after the snapshot restore with veeam all are gone (file still there).

! When I do a ceph snapshot clone (restore it) and mount it into ubuntu I can also see that the xattr are missing. That would rule out an opencloud bug or veeam k10 bug. It seems more that ceph is with the snapshot clone (restore) not having the xattr also copied. Maybe there is an easy fix missing that would do that.

When I use rbd (instead of cephfs) backup and restore works fine.

{"level":"debug","service":"storage-users","host.name":"storageusers-597fc957c9-s8ltj","pkg":"rgrpc","driver":"posix","path":"/var/lib/opencloud/storage/users/users/fe451606-1eef-41b7-bf70-5baddd034d92/.personal_data_export.json","recurse":false,"time":"2025-11-21T16:50:17Z","line":"github.com/opencloud-eu/reva/v2@v2.39.1/pkg/storage/fs/posix/tree/assimilation.go:409","message":"assimilate"}
{"level":"error","service":"storage-users","host.name":"storageusers-597fc957c9-s8ltj","pkg":"rgrpc","driver":"posix","error":"could not find space for path /var/lib/opencloud/storage/users/users/fe451606-1eef-41b7-bf70-5baddd034d92","path":"/var/lib/opencloud/storage/users/users/fe451606-1eef-41b7-bf70-5baddd034d92/.personal_data_export.json","time":"2025-11-21T16:50:17Z","line":"github.com/opencloud-eu/reva/v2@v2.39.1/pkg/storage/fs/posix/tree/assimilation.go:897","message":"could not assimilate item"}

Steps to reproduce

1. Cephfs 3x Coded Pool
2. Backup Veeam k10 or normal snapshot clone in ceph
3. Restore csi snapshot

[suse-coder]

As you also have cephfs (we have 19.2.1) I guess you also do a snapshot restore and would have got that the xattr are getting removed. Little confused why that happens (or if cephfs just does not support it yet?)

[suse-coder]

I would like to also export it to S3 (full backup) (with veeam k10 s3 export). But the restore is also not restoring the xattr. Is there a way to make it that opencloud could restore these automatically if they are lost?

[suse-coder]

Is there maybe a way to disable that opencloud uses xattr?

[micbar]

OpenCloud posixfs uses the storage and the xattrs as a database.

When snapshotting and restoring the data, make sure that you backup and restore the xattrs also. Losing the xattrs due to the backup restore would mean metadata loss.

[micbar]

You can use decomposedFS, which doesn’t use xattrs. But the storage layout is not human readable because it is decomposed using sharded id patterns and symlinks. With the decomposedFS, you need to include symlinks in the backup.

[suse-coder]

OpenCloud posixfs uses the storage and the xattrs as a database.

When snapshotting and restoring the data, make sure that you backup and restore the xattrs also. Losing the xattrs due to the backup restore would mean metadata loss.

Do you know why the normal ceph snapshot clone does not have these restored? Do you know if there is some setting one would need to set?

[micbar]

OpenCloud posixfs uses the storage and the xattrs as a database.
When snapshotting and restoring the data, make sure that you backup and restore the xattrs also. Losing the xattrs due to the backup restore would mean metadata loss.

Do you know why the normal ceph snapshot clone does not have these restored? Do you know if there is some setting one would need to set?

Not yet, that needs research.

[suse-coder]

So for migration from one cluster to another had you tried to use Export to S3 (veeam or else) and then the cephfs xattr metadata are also restored?

[suse-coder]

I use now decompose (fs) on cephfs instead of posix, veeam restore from s3 also worked flawless. Guess will go with that option.

[Koli0842]

I am experimenting with running OpenCloud myself, on a storage backed by NFS. Collabora integrations ometimes dies on me, and today was such time, so I attempted to upgrade my containers after not touching it since november or so. I am seeing the same error occur some times like above, and I'm not able to start up the serevr. I am running an traefikless setup, as I'm using a system-wide caddy to deal with certificated and routing. I have not worked with NFS before, so far from an expert on this matter. Any pointers what might have gone wrong and how could I fix my deployment? Much appreciated!

jan 13 01:14:40 arch opencloud_opencloud_1[3841569]: {"level":"error","service":"storage-users","host.name":"2b8347444629","pkg":"rgrpc","driver":"posix","error":"could not find space for path /var/lib/opencloud/storage/users","path":"/var/lib/opencloud/storage/users/posixfs-xattr-check-3276222753","time":"2026-01-13T00:14:40Z","message":"could not assimilate item"}
jan 13 01:14:40 arch opencloud_opencloud_1[3841569]: {"level":"error","service":"storage-users","host.name":"2b8347444629","pkg":"rgrpc","driver":"posix","error":"could not find space for path /var/lib/opencloud/storage/users","path":"/var/lib/opencloud/storage/users/posixfs-xattr-check-3276672168","time":"2026-01-13T00:14:40Z","message":"could not assimilate item"}
jan 13 01:14:40 arch opencloud_opencloud_1[3841569]: {"level":"error","service":"storage-users","host.name":"2b8347444629","pkg":"rgrpc","driver":"posix","error":"could not find space for path /var/lib/opencloud/storage/users","path":"/var/lib/opencloud/storage/users/posixfs-xattr-check-3297758589","time":"2026-01-13T00:14:40Z","message":"could not assimilate item"}
jan 13 01:14:40 arch opencloud_opencloud_1[3841569]: {"level":"error","service":"storage-users","host.name":"2b8347444629","pkg":"rgrpc","driver":"posix","error":"could not find space for path /var/lib/opencloud/storage/users","path":"/var/lib/opencloud/storage/users/posixfs-xattr-check-3305480910","time":"2026-01-13T00:14:40Z","message":"could not assimilate item"}