docs: fix drifted statements, feature lists and code examples

README.md

@@ -31,12 +31,12 @@ Background and documentation are available at [paimon.apache.org](https://paimon
 Paimon C++ currently provides:
 
 - **Write**: append table and primary key table write support with compaction.
-- **Commit**: append table commit support for simple append-only tables.
+- **Commit**: append table and primary key table commit support. Direct file-system commits work on non-object-store paths; object-store paths require a catalog-mediated commit.
 - **Scan**: batch and stream scan for append tables and primary key tables without changelog.
 - **Read**: append table read, primary key table read with deletion vector, and primary key table merge-on-read.
 - **Arrow integration**: batch read and write interfaces based on the [Arrow Columnar In-Memory Format](https://arrow.apache.org).
 - **File systems**: file system abstraction with built-in local, Jindo, OSS, and S3 file system support.
-- **File formats**: file format abstraction with built-in ORC, Parquet, and Avro support.
+- **File formats**: file format abstraction with built-in Parquet, Blob, ORC, and Avro support, plus optional Lance and Mosaic formats.
 - **Runtime utilities**: memory pool and thread pool abstractions with default implementations.
 - **AI-Oriented Features**: supports RowTracking and DataEvolution mode and provides Global Index
   capabilities including B-tree index, DiskANN-based vector search with Lumina, and Lucene-based

docs/code-style.md

@@ -404,6 +404,11 @@ Before writing new helper code, check whether the project already provides what
 | `UniqueTestDirectory` | RAII helper that creates a unique temp directory and cleans it up on destruction |
 | `TimezoneGuard` | RAII guard that sets `TZ` environment variable and restores it on destruction |
 | `IOExceptionHelper` | Inject I/O errors via `IOHook` for fault-injection testing |
+| `CountingRoutingCache` | `Cache` routing by `CacheKind` to a per-kind `LruCache`, counting gets and supplier calls overall and per kind |
+| `DeletionVectorTestHelper` | Express row-level deletes the way an external engine does: write a deletion vector index file and commit it as an index-only commit message |
+| `GatedAsyncInputStream` | `InputStream` wrapper holding `ReadAsync` callbacks until `ReleaseAll()`, so a test can observe a cache while its fetches are in flight |
+| `BuildTestSnapshot()` | Build a `Snapshot` from an id and optional uuid, filling in every other field |
+| `VariantTestData` | Build variant test data, including an Arrow `StructArray` batch |
 | `testharness.h` | Test macros: `ASSERT_OK`, `ASSERT_NOK`, `ASSERT_NOK_WITH_MSG`, `ASSERT_RAISES` |
 
 ### Mock Objects (`src/paimon/testing/mock/`)
@@ -417,6 +422,13 @@ Before writing new helper code, check whether the project already provides what
 | `MockIndexPathFactory` | Mock index path factory |
 | `MockKeyValueDataFileRecordReader` | Mock key-value data file reader holding in-memory data |
 | `MockStatsExtractor` | Mock statistics extractor |
+| `MockInputStream` / `MockOutputStream` | No-op I/O doubles: reads and writes report zero bytes and `ReadAsync` completes inline |
+| `MockVersionManagedCatalog` | Mock `Catalog` that also implements `VersionManagedCatalog` and records every commit call |
+| `MockGlobalIndexReader` | Mock `GlobalIndexReader`, constructed with or without search support |
+| `MockSearchFileIndex` | File index backend with configured matches in search order; applies pre-filters before limits and returns vector scores aligned with ascending file-local row positions |
+| `MockSearchFileIndexer` / `MockSearchFileIndexReader` | Mock `FileIndexer` and `FileIndexReader` serving those configured scored rows |
+| `MockFullTextSearchFileIndexFactory` / `MockVectorSearchFileIndexFactory` | Mock `FileIndexerFactory` for the full-text and vector search index types |
+| `OptionsCheckingMockSearchFileIndexFactory` | Reuses the mock search backend while requiring the exact options supplied to its factory |
 
 ---
 
@@ -453,6 +465,8 @@ This installs Git hooks that automatically run on every commit:
 | `trailing-whitespace` | Trailing whitespace |
 | `end-of-file-fixer` | Missing newline at end of file |
 | `check-added-large-files` | Files > 5 MB |
+| `check-yaml` | YAML syntax |
+| `sphinx-lint` (alias `docs`) | reStructuredText syntax under `docs/source/` |
 
 To run all hooks on all files manually:
 
@@ -464,10 +478,12 @@ pre-commit run --all-files
 
 Static analysis is configured in `.clang-tidy`. Key enabled check groups:
 
-- `bugprone-*` — Common bug patterns
-- `google-*` — Google style checks
-- `modernize-*` — C++ modernization suggestions
-- `clang-analyzer-*` — Clang static analyzer
+- `bugprone-*` — Common bug patterns. These are listed one check at a time rather than enabled as a wildcard, so a new upstream `bugprone` check never turns itself on.
+- `misc-misleading-identifier`, `misc-homoglyph` — The only two `misc` checks enabled; the rest of `misc-*` is off.
+- `google-*` — Google style checks, except `google-default-arguments`.
+- `modernize-*` — C++ modernization suggestions, except `modernize-return-braced-init-list`, `modernize-avoid-c-arrays`, `modernize-use-trailing-return-type`, `modernize-use-nodiscard` and `modernize-pass-by-value`.
+- `clang-analyzer-*` — Clang static analyzer, except the `clang-analyzer-alpha*` checkers and `clang-analyzer-cplusplus.NewDeleteLeaks`.
+- `clang-diagnostic-*` — Compiler diagnostics, except `clang-diagnostic-global-constructors` and `clang-diagnostic-sign-compare`.
 
 ### Building & Testing
 

docs/source/api/catalog.rst

@@ -30,3 +30,12 @@ Interface
 .. doxygenclass:: paimon::Identifier
    :members:
    :undoc-members:
+
+.. doxygenclass:: paimon::Database
+   :members:
+
+.. doxygenclass:: paimon::Table
+   :members:
+
+.. doxygenstruct:: paimon::CatalogOptions
+   :members:

docs/source/build_system.rst

@@ -102,6 +102,8 @@ Paimon provides a set of built-in optional plugins that you can link to as neede
   - ``Paimon::paimon_avro_file_format_shared``
   - ``Paimon::paimon_blob_file_format_shared``
   - ``Paimon::paimon_lance_file_format_shared``
+  - ``Paimon::paimon_mosaic_file_format_shared``
+  - ``Paimon::paimon_vortex_file_format_shared``
 
 - File system plugins:
 
@@ -113,8 +115,17 @@ Paimon provides a set of built-in optional plugins that you can link to as neede
 - Index plugins:
 
   - ``Paimon::paimon_file_index_shared``
+  - ``Paimon::paimon_global_index_shared``
   - ``Paimon::paimon_lumina_index_shared``
   - ``Paimon::paimon_lucene_index_shared``
+  - ``Paimon::paimon_full_text_index_shared``
+
+.. note::
+
+  A plugin target only exists when its component is enabled. ``parquet``,
+  ``blob``, ``local``, ``file_index`` and ``global_index`` are always built; the
+  others require the matching ``PAIMON_ENABLE_*`` flag described in
+  :ref:`cpp_build_optional_components`.
 
 .. note::
 

docs/source/building.rst

@@ -173,12 +173,20 @@ Optional Components
 
 By default, the C++ build system creates a fairly minimal build. We have
 several optional system components which you can opt into building by passing
-boolean flags to ``cmake``.
+boolean flags to ``cmake``. ORC and Avro are the exceptions: both already default
+to ``ON``, while every other component below defaults to ``OFF``.
 
 * ``-DPAIMON_ENABLE_ORC=ON``: Paimon integration with Apache ORC
 * ``-DPAIMON_ENABLE_AVRO=ON``: Apache Avro libraries and Paimon integration
 * ``-DPAIMON_ENABLE_LANCE=ON``: Lance 0.39 file format support through a Rust FFI; requires
   Rust 1.88 or newer, cbindgen, and protoc 3.12 or newer
+* ``-DPAIMON_ENABLE_MOSAIC=ON``: Apache Paimon Mosaic file format support through a Rust FFI;
+  requires cargo on ``PATH``
+* ``-DPAIMON_ENABLE_VORTEX=ON``: Vortex file format support through a Rust FFI; requires cargo,
+  the stable Rust toolchain (Vortex pins ``rust-version`` 1.95 or newer and is built with
+  ``RUSTUP_TOOLCHAIN=stable``), and a ``flatc`` whose version matches the flatbuffers crate that
+  Vortex uses. Distribution ``flatc`` packages are typically too old; set ``FLATC`` to the path of
+  a suitable binary when it is not on ``PATH``.
 * ``-DPAIMON_ENABLE_JINDO=ON``: Support for Alibaba Jindo filesystems
 * ``-DPAIMON_ENABLE_OSS=ON``: Support for Alibaba Cloud OSS through OSS SDK V2
 * ``-DPAIMON_ENABLE_S3=ON``: Support for Amazon S3-compatible filesystems
@@ -278,6 +286,8 @@ For development builds, you will often want to enable additional targets in
 enable to exercise your changes, using the following ``cmake`` options.
 
 * ``-DPAIMON_BUILD_TESTS=ON``: Build executable unit tests.
+* ``-DPAIMON_ENABLE_NETWORK_TESTS=ON``: Enable tests that access real remote services over the
+  network. These are disabled by default.
 
 Optional Checks
 ~~~~~~~~~~~~~~~

docs/source/user_guide/catalog.rst

@@ -67,8 +67,11 @@ registered on the REST server. The catalog is configured through the
   the role from the metadata endpoint when it is omitted.
 * ``dlf.signing-algorithm``: ``default`` selects DLF4-HMAC-SHA256 for DLF VPC
   endpoints and ``openapi`` selects ROA HMAC-SHA1 for DlfNext OpenAPI endpoints.
-  When omitted, an endpoint containing ``dlfnext`` selects ``openapi`` and other
-  endpoints select ``default``.
+  ``openapi-v4`` selects ACS4-HMAC-SHA256 for DlfNext OpenAPI endpoints. When a
+  request matches a registered DlfNext operation, it also names that operation's
+  API in the ``x-acs-action`` header; other requests are signed without it.
+  Without this option, an endpoint containing ``dlfnext`` selects ``openapi`` and
+  other endpoints select ``default``; ``openapi-v4`` is only used when configured.
 * ``table-default.<key>``: table option defaults applied when a created table
   left ``<key>`` unset.
 * ``header.<name>``: sent as the ``<name>`` http header on every request to the
@@ -103,6 +106,8 @@ are reloaded when less than one hour of validity remains.
        {"dlf.access-key-secret", "<access-key-secret>"},
        // Optional for temporary credentials:
        {"dlf.security-token", "<security-token>"},
+       // Optional for endpoints that require ACS4-HMAC-SHA256 signing:
+       {"dlf.signing-algorithm", "openapi-v4"},
    };
 
 On creation the catalog queries the server's ``/v1/config`` endpoint and merges
@@ -153,6 +158,13 @@ offsets when a real-time writer is created:
    PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::FileStoreWrite> writer,
                           paimon::FileStoreWrite::Create(std::move(write_context)));
 
+The ``Table`` handle is an equivalent entry point: ``table->NewCommit()``,
+``table->NewWrite()``, ``table->NewRead()`` and ``table->NewScan()`` each take the
+same finished context and return the same operator as ``FileStoreCommit::Create()``,
+``FileStoreWrite::Create()``, ``TableRead::Create()`` and ``TableScan::Create()``.
+Pick whichever reads better; the context still carries the catalog when one is
+needed. See the :ref:`Table API <cpp-api-catalog>`.
+
 For real-time writes, also set ``WithStreamingMode(true)`` and
 ``WithRealtimeContext(...)`` with ``realtime.enabled=true``. After
 ``CommitWithProgress`` succeeds, pass its returned snapshot id to
@@ -374,12 +386,22 @@ file system of your own when the credentials expire.
 
    class MyCredentialProvider : public paimon::CredentialProvider {
     public:
+       // The factory hands over the path and the file system options of the accesses to
+       // authenticate, which is where you read your own configuration from.
+       MyCredentialProvider(std::string path, std::map<std::string, std::string> options)
+           : path_(std::move(path)), options_(std::move(options)) {}
+
        // Called as each access is signed; reload the credentials before they expire.
        paimon::Result<std::map<std::string, std::string>> GetCredentials() const override {
+           // Request fresh credentials for path_ from the token service in options_.
            return std::map<std::string, std::string>{
                {"fs.oss.accessKeyId", ...}, {"fs.oss.accessKeySecret", ...},
                {"fs.oss.securityToken", ...}};
        }
+
+    private:
+       std::string path_;
+       std::map<std::string, std::string> options_;
    };
 
    class MyCredentialProviderFactory : public paimon::CredentialProviderFactory {

docs/source/user_guide/compaction.rst

@@ -78,8 +78,8 @@ Full Compaction
 Full compaction rewrites all eligible files in the bucket. During full
 compaction:
 
-- Files whose size is already at or above ``compaction.file-size`` (and have no
-  associated deletion vectors) are skipped to avoid unnecessary rewrites.
+- Files whose size is already at or above the compaction file-size threshold (and
+  have no associated deletion vectors) are skipped to avoid unnecessary rewrites.
 - When deletion vectors are enabled, all files are always eligible for
   compaction regardless of size, because deletion vectors must be applied.
 - When ``compaction.force-rewrite-all-files`` is ``true``, all files are
@@ -88,10 +88,17 @@ compaction:
   small files exceeds the number of large files and the total file count is at
   least 3.
 
-After compaction, if the last output file is still smaller than
-``compaction.file-size``, it is placed back into the compaction queue for future
+After compaction, if the last output file is still smaller than the compaction
+file-size threshold, it is placed back into the compaction queue for future
 merging.
 
+The compaction file-size threshold is not a separate option. It is derived from
+``target-file-size`` as ``target-file-size / 10 * 7``, that is 70% of the target
+size, so that a file already close to the target is not compacted twice.
+``target-file-size`` defaults to 256 MiB for append-only tables and 128 MiB for
+primary-key tables, which makes the derived threshold about 179 MiB and 89 MiB
+respectively. Adjust ``target-file-size`` to move the threshold.
+
 Dictionary Passthrough
 ~~~~~~~~~~~~~~~~~~~~~~
 An append-only compaction rewrite copies rows into the new file without
@@ -126,7 +133,7 @@ dictionary-encodes it the same way; this option only selects ``STRING`` columns.
 A file with a serialized Arrow dictionary schema can independently produce a
 ``BINARY`` dictionary, which the value accessors also support. The rewrite also
 overrides the option back to ``false`` when the table writes a format other than Parquet, when
-``parquet.enable-dictionary`` is ``false`` because the writer would only expand
+``parquet.enable.dictionary`` is ``false`` because the writer would only expand
 the values again, or when variant/map shredding is configured because those
 writers reshape each batch against a fixed physical schema. Setting the option
 therefore never makes a rewrite fail; at worst it has no effect.

docs/source/user_guide/global_index.rst

@@ -38,6 +38,22 @@ Global indexes work on top of Data Evolution tables. To use global indexes, your
 - ``'row-tracking.enabled' = 'true'``
 - ``'data-evolution.enabled' = 'true'``
 
+Cleaning Up Build Checkpoints
+-----------------------------
+
+Checkpoint-enabled index builds remove their checkpoints after a successful build. If a process
+crashes, use ``GlobalIndexWriteTask::DeleteCheckpoint`` to remove all checkpoints left by that
+task once it is no longer running or being retried:
+
+.. code-block:: cpp
+
+   PAIMON_RETURN_NOT_OK(paimon::GlobalIndexWriteTask::DeleteCheckpoint(
+       table_path, "build-task-1", options, file_system));
+
+Pass the same table path and task id used in ``WriteIndex``. Task ids must be unique within the
+table. The cleanup recursively removes the whole task directory, other tasks' directories and completed
+index files are preserved.
+
 Bitmap Index
 ------------
 

docs/source/user_guide/manifest_entry_cache.rst

@@ -28,8 +28,8 @@ scans that target the same bucket.
 
 The cache stores decoded and merged live manifest entries by table path, branch,
 and bucket for ``ScanMode::ALL``. Each cache value can retain several snapshot
-results for that bucket. Exact snapshot hits are served from the cache; cache
-misses rebuild the target snapshot bucket from the target snapshot's data
+results for that bucket. Exact snapshot hits are served from the cache without reading or decoding the
+manifest lists again; cache misses rebuild the target snapshot bucket from the target snapshot's data
 manifests and store the rebuilt live entries.
 
 Request-specific filters are not stored in the cache. Partition, level, and
@@ -86,6 +86,14 @@ under
            .AddOption(Options::SCAN_MANIFEST_ENTRY_CACHE_MAX_SNAPSHOTS, "3")
            .Finish());
 
+.. note::
+
+   ``LruCache`` is the built-in weight-bounded implementation, but it is declared in
+   ``src/paimon/common/io/cache/lru_cache.h`` and only ``include/`` is installed, so it is
+   not reachable from an installed Paimon. ``WithCache`` takes the public ``Cache``
+   interface of ``include/paimon/cache/cache.h``; pass your own implementation of it when
+   you build against the installed package.
+
 Cache entries are scoped by table path, branch, and bucket, so they can be
 reused across newly created ``TableScan`` and ``FileStoreScan`` instances as
 long as they share the same cache object and scan the same bucket.
@@ -120,3 +128,6 @@ The scan metrics expose existing counters for the last scan:
 
 - ``lastScannedManifests``: how many manifest files were loaded during this
   scan before manifest entry decoding.
+
+On an exact cache hit, ``lastScannedManifests`` is zero. The cached table file
+count preserves the skipped-file metric even when manifest lists are bypassed.

docs/source/user_guide/primary_key_global_index.rst

@@ -84,10 +84,11 @@ Current scope
   definitions are recognized for validation, but their evaluation conservatively falls
   back to a normal scan until their dedicated payload readers are supported.
 - The read path targets the Java release-2.0.0 layout and scan semantics (source metadata
-  v1, ``GlobalIndexMeta`` with ``_SOURCE_META``, commit message v12). Source-file names
-  currently use the existing C++ length-prefixed UTF-8 streams; ASCII and non-null BMP
-  names are compatible with Java ``writeUTF``, while complete modified UTF-8 support for
-  supplementary code points will be handled by a shared stream-level change.
+  v1, ``GlobalIndexMeta`` with ``_SOURCE_META``). Commit messages are written at version 13
+  and remain readable down to version 3. Source-file names currently use the existing C++
+  length-prefixed UTF-8 streams; ASCII and non-null BMP names are compatible with Java
+  ``writeUTF``, while complete modified UTF-8 support for supplementary code points will be
+  handled by a shared stream-level change.
 - Automatic maintenance is synchronous during prepare-commit. Java's asynchronous build
   scheduling, retries, fairness metrics, and manual rebuild actions are not part of the C++
   API. Realtime and postpone-bucket writers do not build source-backed payloads.