fuse: add no_create fallback and offer ATOMIC_O_TRUNC/BIG_WRITES in INIT - #14447
fuse: add no_create fallback and offer ATOMIC_O_TRUNC/BIG_WRITES in INIT#14447nicolaslara wants to merge 3 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
nicolaslara
changed the title
FUSE_CREATE is optional: a server may answer it with ENOSYS, in which case Linux latches fc->no_create and creates the file with FUSE_MKNOD followed by FUSE_OPEN (fs/fuse/dir.c:fuse_atomic_open()). The sentry returns the ENOSYS to the application instead, so open(O_CREAT) fails on any such server (mountpoint-s3, for one) even though mknod(2) followed by open(2) works. Latch the ENOSYS in conn.noCreate, mirroring conn.noOpen, and issue FUSE_MKNOD for that and every later create. The inode returned from the MKNOD path carries no file handle, so the Open() kernfs performs next sends a FUSE_OPEN for it.
The sentry already handles the flag on the reply side: when the server echoes FUSE_ATOMIC_O_TRUNC, O_TRUNC rides on the FUSE_OPEN request instead of costing a separate SETATTR round trip. A server can only echo a flag the kernel offered, so without offering it the atomic path is never taken, and servers that require it refuse to start: mountpoint-s3 with --allow-overwrite panics at INIT, leaving the mount unusable. Offering the flag makes two latent bugs reachable, fixed here first: - Open() delegates truncation to the FUSE_OPEN request when the server has negotiated the flag. If the server has also ENOSYSed FUSE_OPEN (conn.noOpen), neither the manual truncate nor an open carrying O_TRUNC is issued and O_TRUNC is silently dropped. Compute whether an open will be sent once, and delegate only when it will. - The first FUSE_OPEN carrying a delegated O_TRUNC can itself return the ENOSYS that latches noOpen, leaving the file untruncated. Truncate with SETATTR in that branch. Linux has offered the flag since FUSE 7.9.
The sentry already handles the flag on the reply side: it clamps every FUSE_WRITE to one page unless the server echoes FUSE_BIG_WRITES. A server can only echo a flag the kernel offered, so without offering it every write is limited to 4 KiB regardless of the negotiated max_write. Offering it lets writes go out at up to min(max_pages * 4096, max_write) per request. Two prerequisites are fixed here first: - The write fragmentation loop bounded its byte count by the absolute file offset plus the length. Correct only at offset zero; otherwise it sent one spurious zero-length FUSE_WRITE per write() and, once the page clamp is lifted, over-sized the first request by the offset. Bound the loop by the request length and use the offset only to address each fragment. - The daemon read-buffer minimum was computed with the reply header rather than FUSEWriteIn, 24 bytes short of the largest request and of what Linux requires. This was unreachable while every request fit in FUSE_MIN_READ_BUFFER. Linux has offered the flag since FUSE 7.9.
674bd7b to
e817498
Compare
|
@manninglucas This change LGTM, could you also look since you're the fsimpl/fuse expert. This unblocks mountpoint-s3 in gVisor. |
| // The FUSE_INIT_IN flags sent to the daemon. | ||
| // TODO(gvisor.dev/issue/3199): complete the flags. | ||
| fuseDefaultInitFlags = linux.FUSE_MAX_PAGES | ||
| fuseDefaultInitFlags = linux.FUSE_MAX_PAGES | linux.FUSE_ATOMIC_O_TRUNC | linux.FUSE_BIG_WRITES |
There was a problem hiding this comment.
fuseDefaultInitFlags is shared by both connection.InitSend and hostConnection.InitSend. But I think this will break hostConnection. That connection mode uses a SOCK_SEQPACKET unix socketpair. Each request must fit in one datagram and in the peer's single read buffer.
Most stock libfuse3 server negotiates max_write ≈ 1 MiB (they echo FUSE_MAX_PAGES=256, so filesystem.Write() now emits single ~1 MiB requests). In Linux, SOCK_SEQPACKET unix sockets default to a datagram size of ~208 KiB. Trying to send a 1 MiB datagram will fail with EMSGSIZE. writeRequest's partial-write loop can't recover, so the app's write(2) fails with EMSGSIZE...
Maybe only add FUSE_BIG_WRITES for connection, and not hostConnection.
| // TODO(gvisor.dev/issue/3237): Add cache support: | ||
| // buffer cache. Ideally we write from src to our buffer cache first. | ||
| // The slice passed to fs.Write() should be a slice from buffer cache. | ||
| data := make([]byte, writeSize) |
There was a problem hiding this comment.
Now that we support 1 MiB writes, this will allocate 1 MiB throwaway buffers in a loop. Let's move it out and allocate one min(maxWrite, remaining) buffer and reuse it?
2 participants
Three independent fixes to the FUSE client, found while getting mountpoint-s3 to work inside a sandbox. Each applies to any FUSE server.
1. Fall back to
FUSE_MKNODwhen the server does not implementFUSE_CREATE.FUSE_CREATEis optional; a server may answer it with ENOSYS. Linux then latchesfc->no_createand creates the file withFUSE_MKNODfollowed byFUSE_OPEN(fs/fuse/dir.c:fuse_atomic_open()). The sentry returns the ENOSYS to the application, soopen(O_CREAT)fails on any such server whilemknod(2)thenopen(2)succeeds. mountpoint-s3 is one such server: it implementsmknodandopenbut notcreate, so nothing that creates files the ordinary way —open(path, "w"), shell>,cp— works on it under gVisor. This addsconn.noCreate, mirroring the existingconn.noOpen.2. Offer
FUSE_ATOMIC_O_TRUNCinFUSE_INIT.The sentry already implements the flag on the reply side but never offers it, so no server can enable it and
O_TRUNCis always emulated with a separateSETATTR. Servers that require atomic truncate refuse to start: mountpoint-s3 with--allow-overwritepanics at INIT, which leaves the mount unusable. Two latent bugs become reachable once the flag is offered and are fixed first: a server that has ENOSYSedFUSE_OPENand negotiated the flag hadO_TRUNCsilently dropped, and the firstFUSE_OPENcarrying a delegatedO_TRUNCcould itself return ENOSYS and leave the file untruncated.3. Offer
FUSE_BIG_WRITESinFUSE_INIT.Also already implemented on the reply side but never offered, so every
FUSE_WRITEis clamped to one page. With the flag, writes go out at up tomin(max_pages * 4096, max_write). Two prerequisites are fixed first: the write loop bounded a byte count by an absolute file offset (one spurious zero-lengthFUSE_WRITEperwrite(), and over-sized allocations once the clamp is gone), and the daemon read-buffer minimum used the reply header instead ofFUSEWriteIn, 24 bytes short of what Linux requires. Note that each in-flight write is held twice in sentry memory until the daemon reads it, so the worst-case memory pinned by writes to a stalled daemon grows with the larger request size (up tomaxActiveRequests× 2 MiB).Linux has offered both flags since FUSE 7.9.
Measured with mountpoint-s3 1.24.0 inside a sandbox, before → after:
open(O_CREAT)ENOSYS → succeeds;--allow-overwritepanics at INIT → mounts and overwrites; INIT flags offered0x400000→0x400028(mount-s3 echoes0x400028with--allow-overwrite,0x400020without);dd bs=1Mwrite throughput 138 → 854 MB/s (median of 3; this measuresFUSE_WRITEround trips, the S3 upload happens on close).bazel test //pkg/sentry/fsimpl/fuse:fuse_testpasses.Updates #3199
Assisted-by: Claude Code