[sundapeng]

Purpose

A REST catalog can vend OSS cache endpoints with the table token, for example an OSS accelerator or a cache cluster in front of the bucket. Today JindoFileIO ignores them unless a JindoCache RPC address is set: it logs "fallback to no-cache" and reads everything from OSS. This PR sends each request to a cache endpoint or to the origin, by operation and file type.

How a request is routed

Every request goes either to a cache target or to the origin, io-cache.origin.endpoint (by default fs.oss.endpoint):

1. Through a table, only paths under the table's own location may use a target; anything else, such as a file under data-file.external-paths or an object a BLOB descriptor points to, goes to the origin. A FileIO that serves no table reads such row-data URIs from the origin too. Paths that are not oss://, operations not listed in io-cache.policy (read, meta for file status, exists, write), temporary files, files rewritten in place and files named by a sequence id (snapshot-N, schema-N, changelog/changelog-N) go to the origin.
2. The file type comes from the whole name, in the shape Paimon writes it. A data file is a data or changelog file prefix, including the data-file.prefix and changelog-file.prefix of the table's current and earlier schemas, followed by {uuid}-{count}.{extension}, the same after the postpone writer's -u-{commit user}-s-{write id}-w-, or {uuid}.managed.blob for a paimon-rust managed BLOB pack; its file index adds .index. Metadata is manifest-, manifest-list-, index-manifest- or stat- followed by {uuid}-{count}, a changelog manifest written by paimon-rust is manifest-{uuid}-changelog-{count}, and a manifest may add .avro.sidecar; a bucket index is index-{uuid}-{count}; a global index is <kind>-global-index-{uuid}.index. Any other name, such as an external Format Table file called manifest.parquet, goes to the origin.
3. The type must be in io-cache.whitelist. With io-cache.routes, the first rule naming the type picks the target; without it, the first target with an endpoint takes every type. A type no rule names goes to the origin.
4. Listings, deletes, renames, atomic writes and the existence check a writer makes before writing always go to the origin. Errors from a target are returned as they are and never retried on the origin, so each endpoint is expected to be highly available.

A target only ever sees files written once under a unique name, so it can be a read cache that keeps metadata. exists is the exception: Paimon uses it to ask whether a file is still there, for example before replaying a commit on recovery, and a cache that keeps the metadata of deleted objects answers yes after the file is gone. So exists uses a target only when the policy lists it, which suits a target that answers existence as OSS does. The origin sees everything else, so it must answer every request as OSS does: OSS itself, or an endpoint that keeps no metadata cache, such as an OSS accelerator. Format, Object, Iceberg and Lance tables never use a target: other writers may replace their files in place under the same name, whatever the name looks like. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin.

Endpoint	As origin	As target
OSS	Yes	Yes, with any policy
An endpoint that keeps no metadata cache and forwards every request to OSS, such as an OSS accelerator	Yes	Yes, with meta,read,exists,write
A read cache that keeps metadata, including that of deleted objects	No	Yes, with meta,read

Usage

A read cache in front of the bucket

io-cache.enabled                          = true
io-cache.endpoint                         = http://10.0.0.1:8080
io-cache.target.default.path-style-access = true
io-cache.policy                           = meta,read
io-cache.whitelist                        = *

Reads and file status of immutable files use the cache; existence checks and everything else use OSS.

An endpoint that also takes writes

io-cache.enabled               = true
io-cache.endpoint              = https://cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
io-cache.target.default.region = cn-hangzhou
io-cache.policy                = meta,read,exists,write
io-cache.whitelist             = data

Data files are read, written and checked through the OSS accelerator, whose read, write and request paths to OSS add to the direct quotas; everything else uses OSS. exists is listed because the accelerator keeps no metadata cache.

The accelerator as origin, a read cache for data

io-cache.enabled                          = true
io-cache.origin.endpoint                  = https://cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
io-cache.endpoint                         = http://10.0.0.1:8080
io-cache.target.default.path-style-access = true
io-cache.policy                           = meta,read
io-cache.whitelist                        = data,global-index

Data files and global indexes are read through the cache. Snapshot probes, hint files, manifests, listings and all writes go through the accelerator.

The accelerator as origin, data straight to OSS

io-cache.enabled         = true
io-cache.origin.endpoint = https://cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
io-cache.endpoint        = https://oss-cn-hangzhou-internal.aliyuncs.com
io-cache.policy          = meta,read,exists,write
io-cache.whitelist       = data

Data files are read and written on the OSS endpoint; metadata and the other request-heavy traffic go through the accelerator, so the two quotas add up.

Two targets by file type

io-cache.enabled                           = true
io-cache.targets                           = accel,cluster
io-cache.target.accel.endpoint             = https://cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
io-cache.target.accel.region               = cn-hangzhou
io-cache.target.cluster.endpoint           = http://10.0.0.1:8080
io-cache.target.cluster.path-style-access  = true
io-cache.routes                            = meta=accel;data,global-index=cluster
io-cache.policy                            = meta,read

Manifests and statistics are read through the accelerator, data files and global indexes through the cache, and the rest, existence checks included, uses OSS.

Options

Vended with the table token or set in the catalog options.

Option	New or existing	Meaning
io-cache.enabled	Existing	Turns the cache on, as before; endpoint routing needs it too.
io-cache.policy	Existing; exists is a new value	The operations that may use a target: read, meta (file status), exists, write; none turns it off. Endpoint routing matches whole tokens; JindoCache keeps its substring match and exists under meta.
io-cache.whitelist-path	Existing, unchanged	Path patterns for JindoCache only; endpoint routing does not read it.
io-cache.whitelist	New	The file types that may use a target: meta, data, bucket-index, global-index, file-index, or * (the default).
io-cache.endpoint	New	A single target, named default.
io-cache.targets with io-cache.target.<name>.endpoint	New	Several named targets.
io-cache.target.<name>.region, io-cache.target.<name>.path-style-access	New	Signing region and path-style addressing of one target, default included.
io-cache.routes	New	Maps file types to targets; the first rule naming a type wins.
io-cache.origin.endpoint	New	Where requests that use no target go; by default fs.oss.endpoint, which may point at a cache for clients without routing.

fs.jindocache.namespace.rpc.address (JindoCache mode), dlf.oss-endpoint (turns routing off) and the table options data-file.prefix / changelog-file.prefix (read to recognise data files) are existing options this PR only reads.

In JindoFileIO

• newInputStream, getFileStatus and exists follow the read, meta and exists rules, and newOutputStream and two-phase multipart writes follow the write rule. Lance and Vortex build their own clients from hadoopOptions(path, op), which cannot route per request, so under routing it gives them the origin's options.
• Each table applies a view to its FileIO in the AbstractFileStoreTable constructor, which every copy, time travel, branch and fallback branch goes through: the view routes only paths under the table's location and knows the prefixes of the table's schema; the prefixes of the branch's earlier schemas are listed once, through the origin, on the first request whose data file name has none of the current prefixes, and only when routing is on. Format, Object, Iceberg and Lance tables take an origin-only view. RESTTokenFileIO, ResolvingFileIO and CachingFileIO pass the view to the FileIO they wrap when it supports views; a JindoFileIO view shares the file systems of the FileIO it was made from, and closing it closes nothing.
• The FileIO UriReaderFactory creates for a BLOB descriptor serves no table, so it reads from the origin; a reader over a table's FileIO follows that table's view.
• copyFile, deleteQuietly and readFileUtf8 keep the FileIO defaults; each read, status lookup and write inside them is routed like any other.
• RESTTokenFileIO caches one FileIO per token and catalog options. Routing may be configured in the catalog options alone, so catalogs that share a vended token but set different origin or targets keep their own endpoints, while copies of one catalog's RESTTokenFileIO, including deserialized ones, still share one FileIO.
• One file system per target and bucket. A target that fails to initialize throws the same checked IOException as the origin, so callers such as OrphanFilesClean and ManifestSidecar.read keep their retry and fallback.
• With fs.jindocache.namespace.rpc.address set, the existing JindoCache mode is used, unchanged. A client-side dlf.oss-endpoint turns routing off.

#10351 (pypaimon) and apache/paimon-rust#1014 implement the same rules.

Tests

• IoCacheRoutingTest: the endpoint JindoFileIO.hadoopOptions picks for each request with one and two targets, the policy (exists on its own included), whitelist, routes and endpoint options, and the type of each file name, including custom prefixes that collide with reserved names, entropy and postpone directories, sidecars (a ROW sidecar is sized and read through the target of its data file), temporary names, metadata directories, and external names that only look like metadata or indexes (manifest.parquet, stat-2024.parquet, foo.index). It also covers target names, policy parsing and serialization.
• JindoIoCacheRoutingTest: options of each target, which operations use a target with and without write and exists in the policy, the two-phase write checking the origin, target errors and target initialization errors thrown as checked IOExceptions, JindoCache RPC mode taking precedence, one file system per target and bucket, and close releasing them.
• TableFileIOViewTest loads tables from a FileSystemCatalog over a FileIO that records its views: copies, branches, external paths, and Format, Object, Iceberg and Lance tables; a FileIO without views, including a forwarding Proxy, stays the same instance in a Format Table, an Object Table and a catalog table.
• IoCacheTableRoutingTest writes and reads real tables through a FileSystemCatalog whose FileIO routes as JindoFileIO does, with a target that keeps the first copy it served while the origin is replaced: blob-descriptor-field values pointing outside the table (primary-key and append tables), BLOB sources copied into the table and data files under data-file.external-paths come from the origin; files under an older data-file.prefix after ALTER TABLE and a fresh load, branch files under the branch's prefix, and the files a branch created from a tag inherited under main's older prefix use the target.
• UriReaderFactoryTest, RESTTokenFileIOTest, ResolvingFileIOTest and CachingFileIOTest cover the origin-only descriptor FileIO, the views the wrappers forward and the token FileIO cached per catalog options; JindoIoCacheRoutingTest builds two RESTTokenFileIOs with one token and different catalog-only origin and target endpoints through the real REST cache and JindoFileIO configuration, and checks that each sends reads and writes to its own endpoints and gives Lance its own origin; IoCacheRoutingTest and JindoIoCacheRoutingTest also cover table views (roots, prefixes, earlier schemas loaded once and a failure to load them, shared file systems), postpone names and managed BLOB packs.
• TestJindoCacheEnable checks that the JindoCache policy matching is unchanged; JindoFileIOTest, TestJindoUserAgent and TestHadoopCompliantFileIODeleteHook still pass.

Verified end to end

All three clients (Java, Python, Rust) created, wrote, read, time-travelled and dropped partitions of tables against local OSS mock endpoints that log every request, with one target, two targets by file type, write routing, and an OSS accelerator as origin with a read cache or OSS as the target. Every request a target saw was one these rules send to it. The pypaimon side (#10351) also ran its cases through a real OSS accelerator, as target and as origin.

API and Format

No storage format change; routing is configured through options. No new types in org.apache.paimon.fs, and the FileIO interface is unchanged. The views live on one internal interface, org.apache.paimon.fs.iocache.IoCacheViewableFileIO (originOnly(), withTableView(TableView)), whose nested TableView and PrefixHistory describe a view; it is internal to io-cache routing, not a public API. JindoFileIO, RESTTokenFileIO, ResolvingFileIO and CachingFileIO implement it; callers go through its static originOnly(FileIO) and withTableView(FileIO, TableView), which return any other FileIO as given, so a FileIO that forwards every method, such as a Proxy in a test, keeps its place.

Documentation

No docs page yet; the options are listed above.

[JingsongLi]

Requirement fit: SUPPORTED. Per-file-type cache routing has practical value for REST-vended OSS endpoints. Implementation review found one P2 issue in initialization failure handling.

Validation: 177/188 selected local tests passed; 11 could not initialize Jindo native on macOS ARM. The supported subset passed the normal Maven verification profiles. Same-head Linux CI passed IoCacheRoutingTest (156), JindoIoCacheRoutingTest (10), and JindoFileIOTest (5). A source-free production-class fault-injection probe shows an initialization IOException remains checked on the origin path but becomes UncheckedIOException on the new target path, bypassing ManifestSidecar.read fallback. Invoking the actual OrphanFilesClean.retryReadingFiles helper with a one-shot initialization failure succeeds on the origin path in two attempts; the target path exits in one attempt. Unwrapping the target-path exception in a source-free control restores that fallback. The endpoint public API was also compared against the original production classes (six boundaries each). No live OSS/cache cluster was available.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: FINDINGS.

Rechecked revision dd23040, including the new exists routing and origin-only pre-write check. The previously reported P2 is still reproducible: cache-target initialization leaves SocketTimeoutException wrapped in UncheckedIOException, so OrphanFilesClean does not retry and ManifestSidecar.read cannot fall back. The equivalent origin path retries successfully in two attempts and preserves its checked exception. Please address the target computeIfAbsent exception unwrapping in addition to the routing changes.

Validation: normal Maven install and all 176 locally supported routing/cache-policy/user-agent/delete-hook tests passed (159 routing cases). The ten JindoIoCacheRoutingTest cases could not initialize the Jindo native library on macOS ARM; this is the previously identified runtime limitation. Same-head initialization fault-injection probes reconfirmed both public IOException handling and the actual orphan retry/sidecar fallback paths. No live OSS/cache endpoint was available. No additional actionable defect was established in the routing revision.

[JingsongLi]

The cache-routing use case has end-to-end value. I ran 36 focused Java tests with the standard Maven checks successfully. Another 14 local cases could not initialize Jindo Native on macOS/aarch64, so I do not count them as verified locally. The exact-head Linux CI checks are green. A production-class probe of the native-format handoff reproduces the configuration mismatch below; the probe exercises the actual Jindo, LanceUtils and VortexUtils read/write option translators without requiring native library initialization.

[JingsongLi]

Requirement fit: SUPPORTED. Per-request OSS endpoint routing has end-to-end value. The previous native-format target handoff is repaired by returning origin options to those consumers. Implementation: FINDINGS.

A clean JDK 8 reactor package passed. Locally, 124 focused tests passed, including actual table read/write/view controls; 21 more could not initialize Jindo Native on macOS/aarch64. I inspected the exact-head Linux JDK 8 CI log: JindoIoCacheRoutingTest (20), JindoFileIOTest (5), and IoCacheRoutingTest (23) all passed, and the full current-head CI matrix is green. A production REST cache/Jindo configuration probe independently reproduces the catalog-only endpoint configuration gap below; it checks actual routing parameters without executing Jindo HTTP.