Mount versioned LakeFS repositories into computing-unit pods (FUSE / GeeseFS)

[aicam]

Feature Summary

Mount a versioned LakeFS repository directly into a computing-unit pod's file system, so operator code can read a repository's files from a local path instead of downloading them over HTTP first. LakeFS backs both datasets and models in Texera, so this works for either.

There is no explicit mount action. A Python UDF declares a parameter whose value names a model or dataset, and the mount follows from that:

class ProcessTupleOperator(UDFOperatorV2):
    def open(self):
        model_dir = self.UiParameter("IRIS_MODEL", AttributeType.STRING, value=Resource.MODEL)
        self.model = joblib.load(f"{model_dir}/iris.pkl")

The parameter is an ordinary string; only where the string comes from differs. The property panel offers that resource's browser instead of a text box, the user picks a version, and what the UDF receives at run time is the local directory that version is mounted at. Choosing the version and mounting it are settled when the workflow runs, not while it is being edited.

Problem it solves: today, if an operator needs a repository's files (e.g. a large ML model with multiple shards), the options are to stream them through the storage API or download the whole repository into the pod before use. For large repositories (multi-GB models) this is slow, uses a lot of pod disk, and forces a full transfer even when the code only reads part of the data.

Proposed Solution or Design

Texera stores datasets and models as LakeFS repositories on MinIO/S3. LakeFS exposes an S3-compatible gateway, so a repository at a specific commit can be mounted as a read-only file system using a FUSE S3 client. We use GeeseFS for this.

Key properties:

• Lazy / on-demand — the mount transfers zero bytes; file contents are fetched with ranged reads only when the code actually read()s them, with read-ahead for throughput.
• Version-pinned & immutable — a repository version maps to a LakeFS commit, so the mounted content is content-addressed and never changes underneath the reader.
• Storage-backend agnostic — GeeseFS always talks to the LakeFS S3 gateway, so it works identically whether LakeFS is backed by in-cluster MinIO or an external S3 bucket.

Architecture (Kubernetes) — the mount is performed out-of-pod. The computing-unit pod runs untrusted user code, so it stays unprivileged. The FUSE mount is performed by a per-node texera-mounter privileged DaemonSet that we own and audit; the resulting read-only mount is exposed back into the CU pod through Kubernetes mount propagation, scoped to that CU. This shrinks the privileged surface from "every UDF" to one small reviewed program.

Authorization (no global credentials in the pod). The mount reuses the per-user JWT the pod already holds and the authorization file-service already applies to repository reads (userHasReadAccess). A JWT-authenticated S3 proxy in file-service sits in front of the LakeFS S3 gateway: GeeseFS is given the pod's JWT as its S3 access key, the proxy verifies the JWT, checks the user's read access to the requested repository, and re-signs to LakeFS with the global credentials held only server-side. No per-mount credential is minted or stored; file-service holds no per-mount state and stays horizontally scalable.

Only access-control-service may ask for a mount. The mounter is the one privileged component — root on every node, listening on a hostPort — so it authenticates its caller with the Kubernetes TokenReview API against a token bound to its own audience, and the allow-list holds one identity: access-control-service, which is already where the deployment decides whether a user may act on a computing unit. A computing-unit pod running user code holds no such token, so a mount request forged from a UDF fails regardless of what it asks for.

End-to-end flow:

1. A Python UDF names a model or dataset version through a parameter; the user picks the version in the property panel.
2. When the execution starts, the operator's deferred binding resolves that version path to a repository:commitHash locator and rewrites the parameter to the directory the version will be readable at.
3. The region scheduler gathers the deduplicated set of locators its operators name and, for each, asks access-control-service to mount it — presenting the computing unit's own user JWT.
4. Access-control-service validates that JWT, confirms the user's access to the computing unit, finds the node its pod runs on, and asks that node's texera-mounter to run GeeseFS against the file-service S3 proxy, read-only.
5. The mount propagates into the CU pod, and the UDF opens files under the directory it was handed.

This has been prototyped and validated end-to-end: a ~2 GB sharded PyTorch model in one repository, loaded inside a Python UDF via torch.load() from the mounted path, with bit-exact outputs — against both in-cluster MinIO and a remote AWS S3 backing store.

Implementation / Sub-issues

Delivered as three stacked PRs to main, each kept small and self-contained:

• [x] Out-of-pod mount infrastructure (Out-of-pod LakeFS repository mount infrastructure #6862 → PR feat(computing-unit): out-of-pod LakeFS repository mount infrastructure #6866, merged) — the per-node privileged texera-mounter, the file-service JWT S3 proxy, and the unprivileged CU pod's mount-propagation wiring. Performs and authorizes a mount; nothing triggers one yet.
• [ ] Authorize and perform a repository mount (Authorize and perform a repository mount #6863 → PR feat(dataset-mount): authorize and perform a repository mount #6896) — the access-control-service mount endpoint that the mounter admits, which decides everything the mounter cannot: request shape, the caller's write access to the computing unit, their read access to the repository, and that the commit belongs to it. No user-facing surface.
• [ ] Name a model or dataset from a Python UDF parameter (Name a model or dataset from a Python UDF parameter #6895 → PR feat(dataset-mount): use mounted datasets in Python UDFs #6897) — value=Resource.MODEL / Resource.DATASET in pytexera, the property-panel resource browser, resolving a version path to its repository and commit, the engine client that asks for the mount, the operator's execution-time binding, and mounting once per region in the scheduler.

Affected Area: Deployment / Infrastructure, Workflow Engine (Amber), Storage / Metadata, Workflow UI