From 9339edf336f05b9d7a7a5c7a803fbfc19feb176f Mon Sep 17 00:00:00 2001 From: zhangstar333 Date: Wed, 19 Aug 2026 14:13:02 +0800 Subject: [PATCH 1/3] [doc](lance)update doc about lance vector_search --- docs/lakehouse/catalogs/lance-catalog.mdx | 125 ++++++++++++++---- .../lakehouse/catalogs/lance-catalog.mdx | 123 ++++++++++++++--- .../lakehouse/catalogs/lance-catalog.mdx | 123 ++++++++++++++--- .../lakehouse/catalogs/lance-catalog.mdx | 125 ++++++++++++++---- 4 files changed, 410 insertions(+), 86 deletions(-) diff --git a/docs/lakehouse/catalogs/lance-catalog.mdx b/docs/lakehouse/catalogs/lance-catalog.mdx index 1e53b1e02cc47..9e058c3cf38ff 100644 --- a/docs/lakehouse/catalogs/lance-catalog.mdx +++ b/docs/lakehouse/catalogs/lance-catalog.mdx @@ -24,31 +24,31 @@ Doris currently provides read-only access to Lance. Creating, writing, updating, | Data queries | Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version | | Predicate pushdown | Supports pushing compatible scalar predicates down to Lance | | File TVFs | Supports querying Lance datasets directly through `s3()` and `local()` | -| Vector search | Supports querying Lance vector indexes or performing Flat Search through `vector_search()` | +| Vector search | Supports querying Lance vector indexes or performing Flat Search per Fragment through `vector_search()`, followed by a Doris global Top-K merge | | Writing to Lance | Not supported | | Time Travel | Not supported | | Full-Text Search / Hybrid Search | Not supported | ## Lance Version and Compatibility -The Doris BE data reader is built with [`lance-c` `v0.1.2`](https://github.com/lance-format/lance-c/tree/v0.1.2). This `lance-c` release declares the Lance Rust crates at version `4.0.1`. These implementation versions are different from the Lance `data_storage_version` recorded in a dataset. +The Doris BE data reader is built with `lance-c v0.1.6`. In Doris, this version is bound to Lance `9.1.0-beta.3` at Lance commit `e934cc2c`. The `lance-c` and Lance Rust crate versions identify the reader implementation integrated with Doris. They are different from the Lance `data_storage_version` recorded in a dataset. The following table describes the file-format compatibility of this reader: | `data_storage_version` | Read support | Notes | |---|---|---| | `0.1` / `legacy` | Supported | Original Lance file format. | -| `2.0` | Supported | The `stable` alias resolves to `2.0` in Lance `4.0.1`; this is the most conservative choice for data that Doris must read. | -| `2.1` | Supported | Uses the newer nested-field encoding in which structural validity is stored in repetition and definition levels. | -| `2.2` | Recognized, but not guaranteed | This format was marked unstable by the Lance version embedded in Doris. Do not rely on it for production interoperability. | -| `2.3` / `next` | Experimental; not guaranteed | The `next` alias is unstable and may change incompatibly. | +| `2.0` (writer-option alias `0.3`) | Supported | An earlier version of the Lance v2 file format. | +| `2.1` / `stable` | Supported; default stable format | In the embedded Lance version, the `stable` writer option and the default format for new datasets both resolve to `2.1`. | +| `2.2` | Supported | The embedded Lance version treats this as a stable format, but it is not the default writer format. | +| `2.3` / `next` | Experimental; not guaranteed | The embedded Lance version marks `2.3` as unstable, and the `next` writer option resolves to `2.3`. | | A later or unknown version | Not supported | Opening or scanning the dataset may fail with an unsupported storage-version error. | Lance SDK release numbers and file-format versions are independent. A dataset written by an older or newer Lance SDK is readable only when its storage format, required table feature flags, index format, and Arrow/Lance data types are all understood by the versions embedded in Doris. Consequently: -- Doris is expected to read datasets written with the stable `0.1`, `2.0`, and `2.1` storage formats, subject to the type limitations documented below. +- Doris is expected to read datasets written with the `0.1`, `2.0`, `2.1`, and `2.2` storage formats, subject to the type limitations documented below. - Forward compatibility is not guaranteed. A dataset written or modified by a later Lance release may be unreadable if it uses a newer storage format, an unknown required manifest feature, a newer index format, or an unsupported extension type. -- For datasets that must remain readable by this Doris release, explicitly set `data_storage_version` to `2.0`. Do not use `next`. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production. +- For datasets that must remain readable by this Doris release, use the current default stable format, `2.1`, and do not use `next`. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production. ## Configure a Catalog @@ -360,21 +360,23 @@ Lance file TVFs have the following additional limitations: ### Syntax and Example ```sql -SELECT row_id, label, _distance +SELECT user_id, label, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3, 0.4]", "top_k" = "10", + "offset" = "3", "metric" = "l2", "nprobes" = "20", "refine_factor" = "10", + "filter" = "category = 'book'", "use_index" = "true" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -The result contains all columns from the Lance source table plus the `_distance` column that the Lance Scanner automatically projects for the nearest-neighbor query. Doris deserializes this Arrow column and exposes it as `FLOAT`. `_distance` is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named `_distance`. A SQL relation does not guarantee output order, so explicitly specify `ORDER BY _distance ASC` when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance. +The relation schema of `vector_search()` contains all columns from the Lance source table plus the `_distance` column generated by the Lance Scanner for the nearest-neighbor query. The final SQL result contains only columns projected by `SELECT`. Doris exposes `_distance` as `FLOAT`. It is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named `_distance`. A SQL relation does not guarantee final display order, so explicitly specify `ORDER BY _distance ASC` when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance. `table` must parse as exactly three `catalog.database.table` name parts. A multilevel Lance Namespace maps to one Doris database name containing `.`, so quote the database part with backticks. For example, use the following value for table `items` in Namespace `doris.analytics`: @@ -402,12 +404,27 @@ Do not use the unquoted form `lance_catalog.doris.analytics.items`; it parses as These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When `metric` is omitted, Doris uses the metric configured when a compatible vector index was created. If there is no compatible index, or if `"use_index" = "false"`, `uint8` vectors use `hamming`, while the other currently supported vector element types use `l2`. +### Supported Vector Index Types + +The embedded `lance-c v0.1.6` explicitly supports the following Lance vector index combinations: + +| Index type | Description | Main query parameters | +|---|---|---| +| `IVF_FLAT` | IVF partitions with original-vector distance computation inside each partition | `nprobes` | +| `IVF_SQ` | IVF with Scalar Quantization | `nprobes`, `refine_factor` | +| `IVF_PQ` | IVF with Product Quantization | `nprobes`, `refine_factor` | +| `IVF_HNSW_FLAT` | IVF with HNSW whose graph nodes retain original vectors | `nprobes`, `ef` | +| `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`, `refine_factor` | +| `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`, `refine_factor` | + +`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, Doris delegates the query to Lance, and Lance selects an existing index compatible with the vector column and metric. If no compatible index is available, Lance performs Flat Search. With `use_index=false`, Flat Search is forced. Flat Search is not an ANN index type; Lance must directly read and compare vectors. + ### Prefilter and Post-Filter -Lance evaluates the TVF `filter` parameter before selecting the Top-K vector candidates: +The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each Fragment, and Lance evaluates it before ANN or Flat Search generates candidates: ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -415,13 +432,15 @@ FROM vector_search( "top_k" = "10", "filter" = "category = 'book'" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -Doris evaluates an outer `WHERE` after Lance has returned its Top-K: +Lance reads and evaluates columns referenced only by `filter` internally. If such a column is not referenced by `SELECT` or another Doris expression, it does not have to be returned to Doris. + +An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each Fragment and before Doris performs its local and global TopN operations. ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -429,18 +448,78 @@ FROM vector_search( "top_k" = "10" ) WHERE category = 'book' -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -Consequently, an outer `WHERE` may reduce the final result to fewer than `top_k` rows. If a filter must participate in nearest-neighbor candidate selection, specify it through the TVF `filter` parameter. +Consequently, an outer `WHERE` only filters candidates that have already been generated and does not cause Lance to replenish them. The final result may contain fewer than `top_k` rows. If the filter must reduce the vector search space and nearest neighbors must be selected from the filtered rows, use the TVF `filter` parameter. ### Current Execution Model -`vector_search()` pins one Lance dataset version and uses one Scanner to search every Fragment in that version, which lets Lance produce a global Top-K result. Doris does not currently split vector search across multiple Scanners or merge candidate sets from multiple Scanners. +`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset: -:::note -Multi-Scanner vector search is planned as a future optimization. It will require index-aware search partitioning and an internal global candidate merge so that `top_k`, `offset`, Prefilter, and `_distance` semantics remain unchanged. -::: +1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. +2. Each Fragment becomes an independent Scan Split that can run in parallel on different BEs and Scanners. +3. For query parameters `top_k=K` and `offset=n`, each Fragment requests at most `K+n` candidates and does not apply the offset locally. Lance evaluates the TVF `filter` before generating candidates. Doris Scan evaluates an outer `WHERE` after candidate generation. +4. Doris performs local TopN, Exchange, and global TopN over candidates from every Fragment, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. + +A Fragment-level TopN therefore supplies candidates for global merging and is not the final result. All Fragments and any later Row-ID fetches use the same pinned snapshot throughout the query. + +The execution order is: + +```text +Pinned dataset snapshot + -> Per Fragment: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates + -> Doris Scan Post-filter + -> Doris local TopN + -> Exchange + -> Doris global TopN (applies offset=n and limit=K) + -> Optional lazy-materialization Fetch +``` + +### Two-Phase TopN Read and Lazy Materialization + +`vector_search()` can use a two-phase read when `experimental_topn_lazy_materialization_threshold` is greater than `0`, `top_k` does not exceed the threshold, and at least one top-level column is eligible for deferred reading. The default threshold is `1024`. Phase 1 carries only the columns required for candidate filtering and TopN, plus an internal Row Location. After global TopN, Phase 2 reads the other output columns only for the retained rows. + +For example, assume the source table has these columns: + +| Column | Purpose | +|---|---| +| `user_id` | Final output column | +| `category` | Post-filter column used by the outer `WHERE` | +| `title`, `payload` | Final output columns | +| `embedding` | Lance vector-search column | + +Run this query with `K=10` and `n=3`: + +```sql +SET experimental_topn_lazy_materialization_threshold = 1024; + +SELECT user_id, title, payload, _distance +FROM vector_search( + "table" = "lance_catalog.default.items", + "column" = "embedding", + "query_vector" = "[0.1, 0.2, 0.3, 0.4]", + "top_k" = "10", + "offset" = "3" +) +WHERE category = 'book'; +``` + +A typical two-phase column flow is: + +| Stage or operator | Columns read or output | Description | +|---|---|---| +| Lance Fragment Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Fragment produces at most `K+n` candidates. | +| Doris Scan Post-filter | `_distance`, `category`, and the internal Row Location | Evaluates `category = 'book'`. A column used by an outer `WHERE` must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into an internal Row Location, which Fetch resolves to the same pinned snapshot. | +| Local and global TopN | Phase-1 required columns and the internal Row Location | Global TopN merges by `_distance` and applies `offset=n` and `limit=K`. | +| Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`, and `payload` | Reads deferred columns for rows retained by global TopN from the same Lance dataset snapshot without rescanning Fragments. | +| Final Materialize | `user_id`, `title`, `payload`, and `_distance` | Combines deferred columns with columns retained from Phase 1 to produce the final SQL output. | + +Phase-1 required columns are not limited to `_distance` and Post-filter columns. Any column referenced by a Doris expression or operator before global TopN is operative and must be read in Phase 1. For example, adding `ORDER BY _distance, user_id` makes `user_id` a Phase-1 column, so it cannot be deferred to the Row-ID Fetch. Nested subcolumn projections are also not currently deferred. A top-level column used only by the final projection can be fetched in Phase 2. + +A Prefilter column referenced in the TVF `filter` differs from an outer `WHERE` column. Lance uses the former internally during search, so appearing in the `filter` string alone does not require the column to be returned to Doris. Doris Scan evaluates the latter, so it must be present in Phase 1. + +Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Fragment and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Whether Lance scans every vector depends on `use_index`: a compatible index performs indexed search, while forced or fallback Flat Search directly compares vectors. ## Current Limitations and Recommendations @@ -450,4 +529,4 @@ Multi-Scanner vector search is planned as a future optimization. It will require - For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify which conditions have been pushed down. - Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set `"use_index" = "false"` to perform Flat Search. - For deterministic vector result ordering, explicitly use `ORDER BY _distance ASC` and add a unique tie-breaker. -- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidate selection. Use an outer `WHERE` only when post-Top-K filtering is intentional. +- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each Fragment and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx index 27c6380221b81..e767ab296872b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx @@ -24,31 +24,31 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan | 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 | | 谓词下推 | 支持将部分标量谓词下推到 Lance 执行 | | 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 | -| 向量检索 | 支持通过 `vector_search()` 查询 Lance 向量索引或执行 Flat Search | +| 向量检索 | 支持通过 `vector_search()` 按 Fragment 并行查询 Lance 向量索引或执行 Flat Search，并由 Doris 合并全局 Top-K | | 写入 Lance | 暂不支持 | | Time Travel | 暂不支持 | | Full-Text Search / Hybrid Search | 暂不支持 | ## Lance 版本与兼容性 -Doris BE 数据读取器使用 [`lance-c` `v0.1.2`](https://github.com/lance-format/lance-c/tree/v0.1.2) 构建；该 `lance-c` 发行版声明的 Lance Rust crates 版本为 `4.0.1`。这些实现版本与数据集中记录的 Lance `data_storage_version` 不是同一个概念。 +Doris BE 数据读取器使用 `lance-c v0.1.6` 构建。该版本在 Doris 中绑定的 Lance 源码版本为 `9.1.0-beta.3`（Lance commit `e934cc2c`）。`lance-c` 和 Lance Rust crates 的版本表示 Doris 集成的读取器实现版本，与数据集中记录的 Lance `data_storage_version` 不是同一个概念。 当前读取器的文件格式兼容情况如下： | `data_storage_version` | 读取支持 | 说明 | |---|---|---| | `0.1` / `legacy` | 支持 | Lance 的初始文件格式。 | -| `2.0` | 支持 | 在 Lance `4.0.1` 中，`stable` 别名对应 `2.0`；对于必须由 Doris 读取的数据，这是最保守的选择。 | -| `2.1` | 支持 | 使用新的嵌套字段编码，将结构字段的有效性信息保存在 repetition/definition level 中。 | -| `2.2` | 可以识别，但不保证兼容 | Doris 内置的 Lance 版本将该格式标记为不稳定，不应依赖它实现生产环境互操作。 | -| `2.3` / `next` | 实验性，不保证兼容 | `next` 别名不稳定，可能发生不兼容变更。 | +| `2.0`（写入选项别名 `0.3`） | 支持 | Lance v2 文件格式的早期版本。 | +| `2.1` / `stable` | 支持，默认稳定格式 | 当前内置 Lance 的 `stable` 写入选项和新数据集默认格式均解析为 `2.1`。 | +| `2.2` | 支持 | 当前内置 Lance 将其视为稳定格式，但它不是默认写入格式。 | +| `2.3` / `next` | 实验性，不保证兼容 | 当前内置 Lance 将 `2.3` 标记为不稳定格式，`next` 写入选项解析为 `2.3`。 | | 后续或未知版本 | 不支持 | 打开或扫描数据集时可能返回不支持存储版本的错误。 | Lance SDK 发行版本号和文件格式版本相互独立。无论数据集由更早还是更新的 Lance SDK 写入，只有当其存储格式、必需的表级 Feature Flag、索引格式以及 Arrow/Lance 数据类型均可被 Doris 内置版本识别时，Doris 才能读取。因此： -- Doris 预期能够读取使用稳定存储格式 `0.1`、`2.0` 和 `2.1` 写入的数据集，同时还需满足下文所述的数据类型限制。 +- Doris 预期能够读取使用 `0.1`、`2.0`、`2.1` 和 `2.2` 存储格式写入的数据集，同时还需满足下文所述的数据类型限制。 - 不保证向前兼容。更新的 Lance 版本写入或修改数据集后，如果使用了更新的存储格式、未知的必需 Manifest Feature、新索引格式或不支持的 Extension 类型，Doris 可能无法读取。 -- 对于必须由当前 Doris 版本持续读取的数据集，建议显式设置 `data_storage_version` 为 `2.0`，且不要使用 `next`。引入更新的写入器或可选 Lance 功能后，应先使用目标 Doris 版本验证生成的数据集，再用于生产环境。 +- 对于必须由当前 Doris 版本持续读取的数据集，建议使用当前默认稳定格式 `2.1`，且不要使用 `next`。引入更新的写入器或可选 Lance 功能后，应先使用目标 Doris 版本验证生成的数据集，再用于生产环境。 ## 配置 Catalog @@ -360,21 +360,23 @@ Lance 文件 TVF 还有以下限制： ### 语法和示例 ```sql -SELECT row_id, label, _distance +SELECT user_id, label, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3, 0.4]", "top_k" = "10", + "offset" = "3", "metric" = "l2", "nprobes" = "20", "refine_factor" = "10", + "filter" = "category = 'book'", "use_index" = "true" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -结果包含 Lance 源表的所有列，以及 Lance Scanner 为最近邻查询自动投影的 `_distance` 列。Doris 会反序列化该 Arrow 列，并以 `FLOAT` 类型提供给用户。`_distance` 表示距离，而不是通用的相似度分数；值越小表示两个向量越接近。源表不能已经包含名为 `_distance` 的列。SQL 关系本身不保证输出顺序，因此需要稳定的最近邻顺序时，应显式使用 `ORDER BY _distance ASC`，并建议增加唯一列作为距离相同情况下的 Tie-breaker。 +`vector_search()` 的关系 Schema 包含 Lance 源表的所有列，以及 Lance Scanner 为最近邻查询生成的 `_distance` 列；最终 SQL 结果只包含 `SELECT` 投影的列。Doris 将 `_distance` 作为 `FLOAT` 提供给用户。它表示距离而不是通用的相似度分数，值越小表示两个向量越接近。源表不能已经包含名为 `_distance` 的列。SQL 关系本身不保证最终展示顺序，因此需要稳定的最近邻顺序时，应显式使用 `ORDER BY _distance ASC`，并建议增加唯一列作为距离相同情况下的 Tie-breaker。 `table` 必须解析为恰好三部分的 `catalog.database.table` 名称。多级 Lance Namespace 在 Doris 中映射为包含 `.` 的单个数据库名，因此必须使用反引号将数据库部分括起来。例如，表 `items` 位于 `doris.analytics` Namespace 时，应写为： @@ -402,12 +404,27 @@ ORDER BY _distance ASC, row_id; 以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的 Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。 +### 支持的向量索引类型 + +当前内置 `lance-c v0.1.6` 明确支持以下 Lance 向量索引组合： + +| 索引类型 | 说明 | 主要查询参数 | +|---|---|---| +| `IVF_FLAT` | IVF 分区，分区内使用原始向量计算距离 | `nprobes` | +| `IVF_SQ` | IVF 与 Scalar Quantization | `nprobes`、`refine_factor` | +| `IVF_PQ` | IVF 与 Product Quantization | `nprobes`、`refine_factor` | +| `IVF_HNSW_FLAT` | IVF 与 HNSW，图节点保存原始向量 | `nprobes`、`ef` | +| `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization | `nprobes`、`ef`、`refine_factor` | +| `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization | `nprobes`、`ef`、`refine_factor` | + +`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，Doris 将查询交给 Lance，由 Lance 选择与向量列和 Metric 兼容的现有索引；没有兼容索引时执行 Flat Search。当 `use_index=false` 时强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 + ### Prefilter 和 Post-filter -TVF 的 `filter` 参数由 Lance 在 Top-K 候选选择前执行： +TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个 Fragment 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -415,13 +432,15 @@ FROM vector_search( "top_k" = "10", "filter" = "category = 'book'" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -外层 `WHERE` 由 Doris 在 Lance 返回 Top-K 后执行： +`filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给 Doris。 + +外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个 Fragment 生成候选之后、Doris 执行局部和全局 TopN 之前。 ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -429,14 +448,78 @@ FROM vector_search( "top_k" = "10" ) WHERE category = 'book' -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -因此，外层 `WHERE` 可能使最终结果少于 `top_k`。如果过滤条件应该参与最近邻候选选择，应使用 TVF 的 `filter` 参数。 +因此，外层 `WHERE` 只过滤已经生成的候选，不会触发 Lance 补充候选，最终结果可能少于 `top_k`。如果过滤条件应该缩小向量候选的搜索空间并保证在过滤后的数据中选择最近邻，应使用 TVF 的 `filter` 参数。 ### 当前执行方式 -`vector_search()` 会固定一个 Lance 数据集版本，并由一个 Scanner 搜索该版本中的全部 Fragment，以保证得到全局 Top-K。当前尚未将向量检索拆分为多个 Scanner，也没有在 Doris 中执行多路候选集的全局 Top-K 合并。 +`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集： + +1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment。 +2. 每个 Fragment 生成一个独立的 Scan Split，可以分发到不同 BE 和 Scanner 并行执行。 +3. 假设查询参数为 `top_k=K`、`offset=n`，每个 Fragment 都请求最多 `K+n` 个候选，并且不在 Fragment 内应用 offset。TVF 的 `filter` 在候选生成前由 Lance 执行；外层 `WHERE` 在候选生成后由 Doris Scan 执行。 +4. Doris 对所有 Fragment 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 + +因此，Fragment 级 TopN 只是为全局合并提供候选，不能直接视为最终结果。查询期间所有 Fragment 以及后续按 Row ID 取列都使用同一个固定快照。 + +执行顺序可以概括为： + +```text +固定数据集快照 + -> 每个 Fragment：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 + -> Doris Scan Post-filter + -> Doris 局部 TopN + -> Exchange + -> Doris 全局 TopN（应用 offset=n 和 limit=K） + -> 可选的延迟物化 Fetch +``` + +### TopN 两阶段读取和延迟物化 + +当 `experimental_topn_lazy_materialization_threshold` 大于 `0`、`top_k` 不超过该阈值，并且存在可以延迟读取的顶层列时，`vector_search()` 可以使用两阶段读取。默认阈值为 `1024`。第一阶段只传递完成候选过滤和 TopN 所必需的列以及内部 Row Location；全局 TopN 完成后，第二阶段只为最终保留的行读取其他输出列。 + +例如，源表包含以下列： + +| 列 | 用途 | +|---|---| +| `user_id` | 最终输出列 | +| `category` | 外层 `WHERE` 的 Post-filter 列 | +| `title`、`payload` | 最终输出列 | +| `embedding` | Lance 向量搜索列 | + +执行以下查询，其中 `K=10`、`n=3`： + +```sql +SET experimental_topn_lazy_materialization_threshold = 1024; + +SELECT user_id, title, payload, _distance +FROM vector_search( + "table" = "lance_catalog.default.items", + "column" = "embedding", + "query_vector" = "[0.1, 0.2, 0.3, 0.4]", + "top_k" = "10", + "offset" = "3" +) +WHERE category = 'book'; +``` + +典型的两阶段列流转如下： + +| 阶段或算子 | 读取或输出的列 | 说明 | +|---|---|---| +| Lance Fragment Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Fragment 最多产生 `K+n` 个候选。 | +| Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行 `category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row Location，Fetch 再通过该映射解析到同一个固定快照。 | +| 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用 `offset=n` 和 `limit=K`。 | +| Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN 保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 | +| 最终 Materialize | `user_id`、`title`、`payload`、`_distance` | 将延迟列与第一阶段保留的列合并，形成 SQL 最终输出。 | + +第一阶段的必需列不只包括 `_distance` 和 Post-filter 列。凡是在全局 TopN 完成前被 Doris 表达式或算子引用的列，都属于第一阶段列。例如，如果查询增加 `ORDER BY _distance, user_id`，`user_id` 也需要提前读取，不能再等到第二阶段 Fetch。嵌套子列投影当前也不会延迟到 Row-ID Fetch。相反，只被最终投影使用的顶层列可以在第二阶段读取。 + +TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在 `filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。 + +将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Fragment 并行生成候选并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。是否在 Lance 内扫描全部向量取决于 `use_index`：使用兼容索引时执行索引搜索，强制或回退到 Flat Search 时才需要直接比较向量。 ## 当前限制和建议 @@ -446,4 +529,4 @@ ORDER BY _distance ASC, row_id; - 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。 - 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat Search。 - 向量查询需要稳定顺序时，显式使用 `ORDER BY _distance ASC` 并增加唯一 Tie-breaker。 -- 需要在向量候选选择前过滤时使用 `vector_search()` 的 `filter`；只有明确需要 Top-K 之后过滤时才使用外层 `WHERE`。 +- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个 Fragment 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index 27c6380221b81..e767ab296872b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -24,31 +24,31 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan | 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 | | 谓词下推 | 支持将部分标量谓词下推到 Lance 执行 | | 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 | -| 向量检索 | 支持通过 `vector_search()` 查询 Lance 向量索引或执行 Flat Search | +| 向量检索 | 支持通过 `vector_search()` 按 Fragment 并行查询 Lance 向量索引或执行 Flat Search，并由 Doris 合并全局 Top-K | | 写入 Lance | 暂不支持 | | Time Travel | 暂不支持 | | Full-Text Search / Hybrid Search | 暂不支持 | ## Lance 版本与兼容性 -Doris BE 数据读取器使用 [`lance-c` `v0.1.2`](https://github.com/lance-format/lance-c/tree/v0.1.2) 构建；该 `lance-c` 发行版声明的 Lance Rust crates 版本为 `4.0.1`。这些实现版本与数据集中记录的 Lance `data_storage_version` 不是同一个概念。 +Doris BE 数据读取器使用 `lance-c v0.1.6` 构建。该版本在 Doris 中绑定的 Lance 源码版本为 `9.1.0-beta.3`（Lance commit `e934cc2c`）。`lance-c` 和 Lance Rust crates 的版本表示 Doris 集成的读取器实现版本，与数据集中记录的 Lance `data_storage_version` 不是同一个概念。 当前读取器的文件格式兼容情况如下： | `data_storage_version` | 读取支持 | 说明 | |---|---|---| | `0.1` / `legacy` | 支持 | Lance 的初始文件格式。 | -| `2.0` | 支持 | 在 Lance `4.0.1` 中，`stable` 别名对应 `2.0`；对于必须由 Doris 读取的数据，这是最保守的选择。 | -| `2.1` | 支持 | 使用新的嵌套字段编码，将结构字段的有效性信息保存在 repetition/definition level 中。 | -| `2.2` | 可以识别，但不保证兼容 | Doris 内置的 Lance 版本将该格式标记为不稳定，不应依赖它实现生产环境互操作。 | -| `2.3` / `next` | 实验性，不保证兼容 | `next` 别名不稳定，可能发生不兼容变更。 | +| `2.0`（写入选项别名 `0.3`） | 支持 | Lance v2 文件格式的早期版本。 | +| `2.1` / `stable` | 支持，默认稳定格式 | 当前内置 Lance 的 `stable` 写入选项和新数据集默认格式均解析为 `2.1`。 | +| `2.2` | 支持 | 当前内置 Lance 将其视为稳定格式，但它不是默认写入格式。 | +| `2.3` / `next` | 实验性，不保证兼容 | 当前内置 Lance 将 `2.3` 标记为不稳定格式，`next` 写入选项解析为 `2.3`。 | | 后续或未知版本 | 不支持 | 打开或扫描数据集时可能返回不支持存储版本的错误。 | Lance SDK 发行版本号和文件格式版本相互独立。无论数据集由更早还是更新的 Lance SDK 写入，只有当其存储格式、必需的表级 Feature Flag、索引格式以及 Arrow/Lance 数据类型均可被 Doris 内置版本识别时，Doris 才能读取。因此： -- Doris 预期能够读取使用稳定存储格式 `0.1`、`2.0` 和 `2.1` 写入的数据集，同时还需满足下文所述的数据类型限制。 +- Doris 预期能够读取使用 `0.1`、`2.0`、`2.1` 和 `2.2` 存储格式写入的数据集，同时还需满足下文所述的数据类型限制。 - 不保证向前兼容。更新的 Lance 版本写入或修改数据集后，如果使用了更新的存储格式、未知的必需 Manifest Feature、新索引格式或不支持的 Extension 类型，Doris 可能无法读取。 -- 对于必须由当前 Doris 版本持续读取的数据集，建议显式设置 `data_storage_version` 为 `2.0`，且不要使用 `next`。引入更新的写入器或可选 Lance 功能后，应先使用目标 Doris 版本验证生成的数据集，再用于生产环境。 +- 对于必须由当前 Doris 版本持续读取的数据集，建议使用当前默认稳定格式 `2.1`，且不要使用 `next`。引入更新的写入器或可选 Lance 功能后，应先使用目标 Doris 版本验证生成的数据集，再用于生产环境。 ## 配置 Catalog @@ -360,21 +360,23 @@ Lance 文件 TVF 还有以下限制： ### 语法和示例 ```sql -SELECT row_id, label, _distance +SELECT user_id, label, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3, 0.4]", "top_k" = "10", + "offset" = "3", "metric" = "l2", "nprobes" = "20", "refine_factor" = "10", + "filter" = "category = 'book'", "use_index" = "true" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -结果包含 Lance 源表的所有列，以及 Lance Scanner 为最近邻查询自动投影的 `_distance` 列。Doris 会反序列化该 Arrow 列，并以 `FLOAT` 类型提供给用户。`_distance` 表示距离，而不是通用的相似度分数；值越小表示两个向量越接近。源表不能已经包含名为 `_distance` 的列。SQL 关系本身不保证输出顺序，因此需要稳定的最近邻顺序时，应显式使用 `ORDER BY _distance ASC`，并建议增加唯一列作为距离相同情况下的 Tie-breaker。 +`vector_search()` 的关系 Schema 包含 Lance 源表的所有列，以及 Lance Scanner 为最近邻查询生成的 `_distance` 列；最终 SQL 结果只包含 `SELECT` 投影的列。Doris 将 `_distance` 作为 `FLOAT` 提供给用户。它表示距离而不是通用的相似度分数，值越小表示两个向量越接近。源表不能已经包含名为 `_distance` 的列。SQL 关系本身不保证最终展示顺序，因此需要稳定的最近邻顺序时，应显式使用 `ORDER BY _distance ASC`，并建议增加唯一列作为距离相同情况下的 Tie-breaker。 `table` 必须解析为恰好三部分的 `catalog.database.table` 名称。多级 Lance Namespace 在 Doris 中映射为包含 `.` 的单个数据库名，因此必须使用反引号将数据库部分括起来。例如，表 `items` 位于 `doris.analytics` Namespace 时，应写为： @@ -402,12 +404,27 @@ ORDER BY _distance ASC, row_id; 以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的 Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。 +### 支持的向量索引类型 + +当前内置 `lance-c v0.1.6` 明确支持以下 Lance 向量索引组合： + +| 索引类型 | 说明 | 主要查询参数 | +|---|---|---| +| `IVF_FLAT` | IVF 分区，分区内使用原始向量计算距离 | `nprobes` | +| `IVF_SQ` | IVF 与 Scalar Quantization | `nprobes`、`refine_factor` | +| `IVF_PQ` | IVF 与 Product Quantization | `nprobes`、`refine_factor` | +| `IVF_HNSW_FLAT` | IVF 与 HNSW，图节点保存原始向量 | `nprobes`、`ef` | +| `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization | `nprobes`、`ef`、`refine_factor` | +| `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization | `nprobes`、`ef`、`refine_factor` | + +`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，Doris 将查询交给 Lance，由 Lance 选择与向量列和 Metric 兼容的现有索引；没有兼容索引时执行 Flat Search。当 `use_index=false` 时强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 + ### Prefilter 和 Post-filter -TVF 的 `filter` 参数由 Lance 在 Top-K 候选选择前执行： +TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个 Fragment 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -415,13 +432,15 @@ FROM vector_search( "top_k" = "10", "filter" = "category = 'book'" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -外层 `WHERE` 由 Doris 在 Lance 返回 Top-K 后执行： +`filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给 Doris。 + +外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个 Fragment 生成候选之后、Doris 执行局部和全局 TopN 之前。 ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -429,14 +448,78 @@ FROM vector_search( "top_k" = "10" ) WHERE category = 'book' -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -因此，外层 `WHERE` 可能使最终结果少于 `top_k`。如果过滤条件应该参与最近邻候选选择，应使用 TVF 的 `filter` 参数。 +因此，外层 `WHERE` 只过滤已经生成的候选，不会触发 Lance 补充候选，最终结果可能少于 `top_k`。如果过滤条件应该缩小向量候选的搜索空间并保证在过滤后的数据中选择最近邻，应使用 TVF 的 `filter` 参数。 ### 当前执行方式 -`vector_search()` 会固定一个 Lance 数据集版本，并由一个 Scanner 搜索该版本中的全部 Fragment，以保证得到全局 Top-K。当前尚未将向量检索拆分为多个 Scanner，也没有在 Doris 中执行多路候选集的全局 Top-K 合并。 +`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集： + +1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment。 +2. 每个 Fragment 生成一个独立的 Scan Split，可以分发到不同 BE 和 Scanner 并行执行。 +3. 假设查询参数为 `top_k=K`、`offset=n`，每个 Fragment 都请求最多 `K+n` 个候选，并且不在 Fragment 内应用 offset。TVF 的 `filter` 在候选生成前由 Lance 执行；外层 `WHERE` 在候选生成后由 Doris Scan 执行。 +4. Doris 对所有 Fragment 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 + +因此，Fragment 级 TopN 只是为全局合并提供候选，不能直接视为最终结果。查询期间所有 Fragment 以及后续按 Row ID 取列都使用同一个固定快照。 + +执行顺序可以概括为： + +```text +固定数据集快照 + -> 每个 Fragment：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 + -> Doris Scan Post-filter + -> Doris 局部 TopN + -> Exchange + -> Doris 全局 TopN（应用 offset=n 和 limit=K） + -> 可选的延迟物化 Fetch +``` + +### TopN 两阶段读取和延迟物化 + +当 `experimental_topn_lazy_materialization_threshold` 大于 `0`、`top_k` 不超过该阈值，并且存在可以延迟读取的顶层列时，`vector_search()` 可以使用两阶段读取。默认阈值为 `1024`。第一阶段只传递完成候选过滤和 TopN 所必需的列以及内部 Row Location；全局 TopN 完成后，第二阶段只为最终保留的行读取其他输出列。 + +例如，源表包含以下列： + +| 列 | 用途 | +|---|---| +| `user_id` | 最终输出列 | +| `category` | 外层 `WHERE` 的 Post-filter 列 | +| `title`、`payload` | 最终输出列 | +| `embedding` | Lance 向量搜索列 | + +执行以下查询，其中 `K=10`、`n=3`： + +```sql +SET experimental_topn_lazy_materialization_threshold = 1024; + +SELECT user_id, title, payload, _distance +FROM vector_search( + "table" = "lance_catalog.default.items", + "column" = "embedding", + "query_vector" = "[0.1, 0.2, 0.3, 0.4]", + "top_k" = "10", + "offset" = "3" +) +WHERE category = 'book'; +``` + +典型的两阶段列流转如下： + +| 阶段或算子 | 读取或输出的列 | 说明 | +|---|---|---| +| Lance Fragment Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Fragment 最多产生 `K+n` 个候选。 | +| Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行 `category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row Location，Fetch 再通过该映射解析到同一个固定快照。 | +| 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用 `offset=n` 和 `limit=K`。 | +| Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN 保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 | +| 最终 Materialize | `user_id`、`title`、`payload`、`_distance` | 将延迟列与第一阶段保留的列合并，形成 SQL 最终输出。 | + +第一阶段的必需列不只包括 `_distance` 和 Post-filter 列。凡是在全局 TopN 完成前被 Doris 表达式或算子引用的列，都属于第一阶段列。例如，如果查询增加 `ORDER BY _distance, user_id`，`user_id` 也需要提前读取，不能再等到第二阶段 Fetch。嵌套子列投影当前也不会延迟到 Row-ID Fetch。相反，只被最终投影使用的顶层列可以在第二阶段读取。 + +TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在 `filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。 + +将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Fragment 并行生成候选并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。是否在 Lance 内扫描全部向量取决于 `use_index`：使用兼容索引时执行索引搜索，强制或回退到 Flat Search 时才需要直接比较向量。 ## 当前限制和建议 @@ -446,4 +529,4 @@ ORDER BY _distance ASC, row_id; - 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。 - 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat Search。 - 向量查询需要稳定顺序时，显式使用 `ORDER BY _distance ASC` 并增加唯一 Tie-breaker。 -- 需要在向量候选选择前过滤时使用 `vector_search()` 的 `filter`；只有明确需要 Top-K 之后过滤时才使用外层 `WHERE`。 +- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个 Fragment 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index 1e53b1e02cc47..9e058c3cf38ff 100644 --- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -24,31 +24,31 @@ Doris currently provides read-only access to Lance. Creating, writing, updating, | Data queries | Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version | | Predicate pushdown | Supports pushing compatible scalar predicates down to Lance | | File TVFs | Supports querying Lance datasets directly through `s3()` and `local()` | -| Vector search | Supports querying Lance vector indexes or performing Flat Search through `vector_search()` | +| Vector search | Supports querying Lance vector indexes or performing Flat Search per Fragment through `vector_search()`, followed by a Doris global Top-K merge | | Writing to Lance | Not supported | | Time Travel | Not supported | | Full-Text Search / Hybrid Search | Not supported | ## Lance Version and Compatibility -The Doris BE data reader is built with [`lance-c` `v0.1.2`](https://github.com/lance-format/lance-c/tree/v0.1.2). This `lance-c` release declares the Lance Rust crates at version `4.0.1`. These implementation versions are different from the Lance `data_storage_version` recorded in a dataset. +The Doris BE data reader is built with `lance-c v0.1.6`. In Doris, this version is bound to Lance `9.1.0-beta.3` at Lance commit `e934cc2c`. The `lance-c` and Lance Rust crate versions identify the reader implementation integrated with Doris. They are different from the Lance `data_storage_version` recorded in a dataset. The following table describes the file-format compatibility of this reader: | `data_storage_version` | Read support | Notes | |---|---|---| | `0.1` / `legacy` | Supported | Original Lance file format. | -| `2.0` | Supported | The `stable` alias resolves to `2.0` in Lance `4.0.1`; this is the most conservative choice for data that Doris must read. | -| `2.1` | Supported | Uses the newer nested-field encoding in which structural validity is stored in repetition and definition levels. | -| `2.2` | Recognized, but not guaranteed | This format was marked unstable by the Lance version embedded in Doris. Do not rely on it for production interoperability. | -| `2.3` / `next` | Experimental; not guaranteed | The `next` alias is unstable and may change incompatibly. | +| `2.0` (writer-option alias `0.3`) | Supported | An earlier version of the Lance v2 file format. | +| `2.1` / `stable` | Supported; default stable format | In the embedded Lance version, the `stable` writer option and the default format for new datasets both resolve to `2.1`. | +| `2.2` | Supported | The embedded Lance version treats this as a stable format, but it is not the default writer format. | +| `2.3` / `next` | Experimental; not guaranteed | The embedded Lance version marks `2.3` as unstable, and the `next` writer option resolves to `2.3`. | | A later or unknown version | Not supported | Opening or scanning the dataset may fail with an unsupported storage-version error. | Lance SDK release numbers and file-format versions are independent. A dataset written by an older or newer Lance SDK is readable only when its storage format, required table feature flags, index format, and Arrow/Lance data types are all understood by the versions embedded in Doris. Consequently: -- Doris is expected to read datasets written with the stable `0.1`, `2.0`, and `2.1` storage formats, subject to the type limitations documented below. +- Doris is expected to read datasets written with the `0.1`, `2.0`, `2.1`, and `2.2` storage formats, subject to the type limitations documented below. - Forward compatibility is not guaranteed. A dataset written or modified by a later Lance release may be unreadable if it uses a newer storage format, an unknown required manifest feature, a newer index format, or an unsupported extension type. -- For datasets that must remain readable by this Doris release, explicitly set `data_storage_version` to `2.0`. Do not use `next`. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production. +- For datasets that must remain readable by this Doris release, use the current default stable format, `2.1`, and do not use `next`. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production. ## Configure a Catalog @@ -360,21 +360,23 @@ Lance file TVFs have the following additional limitations: ### Syntax and Example ```sql -SELECT row_id, label, _distance +SELECT user_id, label, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3, 0.4]", "top_k" = "10", + "offset" = "3", "metric" = "l2", "nprobes" = "20", "refine_factor" = "10", + "filter" = "category = 'book'", "use_index" = "true" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -The result contains all columns from the Lance source table plus the `_distance` column that the Lance Scanner automatically projects for the nearest-neighbor query. Doris deserializes this Arrow column and exposes it as `FLOAT`. `_distance` is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named `_distance`. A SQL relation does not guarantee output order, so explicitly specify `ORDER BY _distance ASC` when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance. +The relation schema of `vector_search()` contains all columns from the Lance source table plus the `_distance` column generated by the Lance Scanner for the nearest-neighbor query. The final SQL result contains only columns projected by `SELECT`. Doris exposes `_distance` as `FLOAT`. It is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named `_distance`. A SQL relation does not guarantee final display order, so explicitly specify `ORDER BY _distance ASC` when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance. `table` must parse as exactly three `catalog.database.table` name parts. A multilevel Lance Namespace maps to one Doris database name containing `.`, so quote the database part with backticks. For example, use the following value for table `items` in Namespace `doris.analytics`: @@ -402,12 +404,27 @@ Do not use the unquoted form `lance_catalog.doris.analytics.items`; it parses as These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When `metric` is omitted, Doris uses the metric configured when a compatible vector index was created. If there is no compatible index, or if `"use_index" = "false"`, `uint8` vectors use `hamming`, while the other currently supported vector element types use `l2`. +### Supported Vector Index Types + +The embedded `lance-c v0.1.6` explicitly supports the following Lance vector index combinations: + +| Index type | Description | Main query parameters | +|---|---|---| +| `IVF_FLAT` | IVF partitions with original-vector distance computation inside each partition | `nprobes` | +| `IVF_SQ` | IVF with Scalar Quantization | `nprobes`, `refine_factor` | +| `IVF_PQ` | IVF with Product Quantization | `nprobes`, `refine_factor` | +| `IVF_HNSW_FLAT` | IVF with HNSW whose graph nodes retain original vectors | `nprobes`, `ef` | +| `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`, `refine_factor` | +| `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`, `refine_factor` | + +`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, Doris delegates the query to Lance, and Lance selects an existing index compatible with the vector column and metric. If no compatible index is available, Lance performs Flat Search. With `use_index=false`, Flat Search is forced. Flat Search is not an ANN index type; Lance must directly read and compare vectors. + ### Prefilter and Post-Filter -Lance evaluates the TVF `filter` parameter before selecting the Top-K vector candidates: +The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each Fragment, and Lance evaluates it before ANN or Flat Search generates candidates: ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -415,13 +432,15 @@ FROM vector_search( "top_k" = "10", "filter" = "category = 'book'" ) -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -Doris evaluates an outer `WHERE` after Lance has returned its Top-K: +Lance reads and evaluates columns referenced only by `filter` internally. If such a column is not referenced by `SELECT` or another Doris expression, it does not have to be returned to Doris. + +An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each Fragment and before Doris performs its local and global TopN operations. ```sql -SELECT row_id, category, _distance +SELECT user_id, category, _distance FROM vector_search( "table" = "lance_catalog.default.items", "column" = "embedding", @@ -429,18 +448,78 @@ FROM vector_search( "top_k" = "10" ) WHERE category = 'book' -ORDER BY _distance ASC, row_id; +ORDER BY _distance ASC, user_id; ``` -Consequently, an outer `WHERE` may reduce the final result to fewer than `top_k` rows. If a filter must participate in nearest-neighbor candidate selection, specify it through the TVF `filter` parameter. +Consequently, an outer `WHERE` only filters candidates that have already been generated and does not cause Lance to replenish them. The final result may contain fewer than `top_k` rows. If the filter must reduce the vector search space and nearest neighbors must be selected from the filtered rows, use the TVF `filter` parameter. ### Current Execution Model -`vector_search()` pins one Lance dataset version and uses one Scanner to search every Fragment in that version, which lets Lance produce a global Top-K result. Doris does not currently split vector search across multiple Scanners or merge candidate sets from multiple Scanners. +`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset: -:::note -Multi-Scanner vector search is planned as a future optimization. It will require index-aware search partitioning and an internal global candidate merge so that `top_k`, `offset`, Prefilter, and `_distance` semantics remain unchanged. -::: +1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. +2. Each Fragment becomes an independent Scan Split that can run in parallel on different BEs and Scanners. +3. For query parameters `top_k=K` and `offset=n`, each Fragment requests at most `K+n` candidates and does not apply the offset locally. Lance evaluates the TVF `filter` before generating candidates. Doris Scan evaluates an outer `WHERE` after candidate generation. +4. Doris performs local TopN, Exchange, and global TopN over candidates from every Fragment, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. + +A Fragment-level TopN therefore supplies candidates for global merging and is not the final result. All Fragments and any later Row-ID fetches use the same pinned snapshot throughout the query. + +The execution order is: + +```text +Pinned dataset snapshot + -> Per Fragment: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates + -> Doris Scan Post-filter + -> Doris local TopN + -> Exchange + -> Doris global TopN (applies offset=n and limit=K) + -> Optional lazy-materialization Fetch +``` + +### Two-Phase TopN Read and Lazy Materialization + +`vector_search()` can use a two-phase read when `experimental_topn_lazy_materialization_threshold` is greater than `0`, `top_k` does not exceed the threshold, and at least one top-level column is eligible for deferred reading. The default threshold is `1024`. Phase 1 carries only the columns required for candidate filtering and TopN, plus an internal Row Location. After global TopN, Phase 2 reads the other output columns only for the retained rows. + +For example, assume the source table has these columns: + +| Column | Purpose | +|---|---| +| `user_id` | Final output column | +| `category` | Post-filter column used by the outer `WHERE` | +| `title`, `payload` | Final output columns | +| `embedding` | Lance vector-search column | + +Run this query with `K=10` and `n=3`: + +```sql +SET experimental_topn_lazy_materialization_threshold = 1024; + +SELECT user_id, title, payload, _distance +FROM vector_search( + "table" = "lance_catalog.default.items", + "column" = "embedding", + "query_vector" = "[0.1, 0.2, 0.3, 0.4]", + "top_k" = "10", + "offset" = "3" +) +WHERE category = 'book'; +``` + +A typical two-phase column flow is: + +| Stage or operator | Columns read or output | Description | +|---|---|---| +| Lance Fragment Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Fragment produces at most `K+n` candidates. | +| Doris Scan Post-filter | `_distance`, `category`, and the internal Row Location | Evaluates `category = 'book'`. A column used by an outer `WHERE` must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into an internal Row Location, which Fetch resolves to the same pinned snapshot. | +| Local and global TopN | Phase-1 required columns and the internal Row Location | Global TopN merges by `_distance` and applies `offset=n` and `limit=K`. | +| Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`, and `payload` | Reads deferred columns for rows retained by global TopN from the same Lance dataset snapshot without rescanning Fragments. | +| Final Materialize | `user_id`, `title`, `payload`, and `_distance` | Combines deferred columns with columns retained from Phase 1 to produce the final SQL output. | + +Phase-1 required columns are not limited to `_distance` and Post-filter columns. Any column referenced by a Doris expression or operator before global TopN is operative and must be read in Phase 1. For example, adding `ORDER BY _distance, user_id` makes `user_id` a Phase-1 column, so it cannot be deferred to the Row-ID Fetch. Nested subcolumn projections are also not currently deferred. A top-level column used only by the final projection can be fetched in Phase 2. + +A Prefilter column referenced in the TVF `filter` differs from an outer `WHERE` column. Lance uses the former internally during search, so appearing in the `filter` string alone does not require the column to be returned to Doris. Doris Scan evaluates the latter, so it must be present in Phase 1. + +Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Fragment and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Whether Lance scans every vector depends on `use_index`: a compatible index performs indexed search, while forced or fallback Flat Search directly compares vectors. ## Current Limitations and Recommendations @@ -450,4 +529,4 @@ Multi-Scanner vector search is planned as a future optimization. It will require - For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify which conditions have been pushed down. - Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set `"use_index" = "false"` to perform Flat Search. - For deterministic vector result ordering, explicitly use `ORDER BY _distance ASC` and add a unique tie-breaker. -- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidate selection. Use an outer `WHERE` only when post-Top-K filtering is intentional. +- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each Fragment and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. From 397645b6c3bdfd8bf905ee594ec0f8c7ea0b9266 Mon Sep 17 00:00:00 2001 From: zhangstar333 Date: Wed, 19 Aug 2026 16:24:25 +0800 Subject: [PATCH 2/3] update --- docs/lakehouse/catalogs/lance-catalog.mdx | 2 ++ .../current/lakehouse/catalogs/lance-catalog.mdx | 2 ++ .../version-4.x/lakehouse/catalogs/lance-catalog.mdx | 2 ++ versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx | 2 ++ 4 files changed, 8 insertions(+) diff --git a/docs/lakehouse/catalogs/lance-catalog.mdx b/docs/lakehouse/catalogs/lance-catalog.mdx index 9e058c3cf38ff..e0a589b9cef2b 100644 --- a/docs/lakehouse/catalogs/lance-catalog.mdx +++ b/docs/lakehouse/catalogs/lance-catalog.mdx @@ -7,6 +7,8 @@ --- :::note +This is an experimental feature. + Lance Catalog is supported starting from Apache Doris 4.2. ::: diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx index e767ab296872b..4a30bbd91724b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx @@ -7,6 +7,8 @@ --- :::note +这是一个实验性功能。 + Lance Catalog 自 Apache Doris 4.2 版本开始支持。 ::: diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index e767ab296872b..4a30bbd91724b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -7,6 +7,8 @@ --- :::note +这是一个实验性功能。 + Lance Catalog 自 Apache Doris 4.2 版本开始支持。 ::: diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index 9e058c3cf38ff..e0a589b9cef2b 100644 --- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -7,6 +7,8 @@ --- :::note +This is an experimental feature. + Lance Catalog is supported starting from Apache Doris 4.2. ::: From f3dd87e204247051871b78d8ea5076b781f6a9e5 Mon Sep 17 00:00:00 2001 From: zhangstar333 Date: Wed, 19 Aug 2026 18:51:48 +0800 Subject: [PATCH 3/3] update --- docs/lakehouse/catalogs/lance-catalog.mdx | 37 +++++++++++-------- .../lakehouse/catalogs/lance-catalog.mdx | 37 +++++++++++-------- .../lakehouse/catalogs/lance-catalog.mdx | 37 +++++++++++-------- .../lakehouse/catalogs/lance-catalog.mdx | 37 +++++++++++-------- 4 files changed, 88 insertions(+), 60 deletions(-) diff --git a/docs/lakehouse/catalogs/lance-catalog.mdx b/docs/lakehouse/catalogs/lance-catalog.mdx index e0a589b9cef2b..cf60bedd4e596 100644 --- a/docs/lakehouse/catalogs/lance-catalog.mdx +++ b/docs/lakehouse/catalogs/lance-catalog.mdx @@ -26,7 +26,7 @@ Doris currently provides read-only access to Lance. Creating, writing, updating, | Data queries | Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version | | Predicate pushdown | Supports pushing compatible scalar predicates down to Lance | | File TVFs | Supports querying Lance datasets directly through `s3()` and `local()` | -| Vector search | Supports querying Lance vector indexes or performing Flat Search per Fragment through `vector_search()`, followed by a Doris global Top-K merge | +| Vector search | Uses physical Lance index segments as parallel splits, keeps uncovered Fragments as Flat Search splits, and performs a Doris global Top-K merge | | Writing to Lance | Not supported | | Time Travel | Not supported | | Full-Text Search / Hybrid Search | Not supported | @@ -402,7 +402,7 @@ Do not use the unquoted form `lance_catalog.doris.analytics.items`; it parses as | `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index partitions to probe. It must be a positive integer. When unset, Lance starts with one partition and can probe additional partitions when a Prefilter leaves too few candidates. Setting it explicitly to `N` fixes both the minimum and maximum number of probes to `N`. | | `refine_factor` | No | Refinement disabled | Candidate refinement multiplier. It must be a positive integer. When unset, Lance does not recompute distances from the original vectors, so `_distance` from a quantized index may be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N` candidates, recomputes their exact distances from the original vectors, and reorders them. Setting it to `1` still enables refinement and therefore differs from leaving it unset. | | `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained during HNSW graph search. It must be a positive integer. If `refine_factor` is also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It has no effect on non-HNSW indexes. | -| `use_index` | No | `true` | When `true`, Doris prefers a compatible Lance vector index and automatically falls back to Flat Search if none is available. When `false`, Doris forces Flat Search. | +| `use_index` | No | `true` | When `true`, Doris plans compatible physical Lance index segments as indexed splits and keeps uncovered Fragments as Flat Search splits. If no usable compatible index metadata is available, Doris falls back to Fragment splits. When `false`, Doris creates one split per visible Fragment and forces Flat Search. | These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When `metric` is omitted, Doris uses the metric configured when a compatible vector index was created. If there is no compatible index, or if `"use_index" = "false"`, `uint8` vectors use `hamming`, while the other currently supported vector element types use `l2`. @@ -419,11 +419,13 @@ The embedded `lance-c v0.1.6` explicitly supports the following Lance vector ind | `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`, `refine_factor` | | `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`, `refine_factor` | -`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, Doris delegates the query to Lance, and Lance selects an existing index compatible with the vector column and metric. If no compatible index is available, Lance performs Flat Search. With `use_index=false`, Flat Search is forced. Flat Search is not an ANN index type; Lance must directly read and compare vectors. +`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, the FE reads vector-index metadata from the pinned dataset snapshot and selects a logical index compatible with the vector column and metric. It then assigns each physical segment of that logical index that still covers visible data to an indexed Scan Split. Each indexed Split carries the segment UUID and the currently visible Fragments covered by that segment, so the BE searches that specific segment instead of asking Lance to choose an index again. + +A logical Lance index can contain multiple physical index segments, and one physical segment can cover multiple Fragments. Fragments not covered by the selected index are not omitted: Doris adds one fallback Split for each such Fragment, which uses Flat Search. If the FE cannot construct a usable index-segment plan, it falls back to Fragment-level splits. With `use_index=false`, Doris skips index metadata planning and forces Flat Search for every visible Fragment. Flat Search is not an ANN index type; Lance must directly read and compare vectors. ### Prefilter and Post-Filter -The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each Fragment, and Lance evaluates it before ANN or Flat Search generates candidates: +The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat Search generates candidates: ```sql SELECT user_id, category, _distance @@ -439,7 +441,7 @@ ORDER BY _distance ASC, user_id; Lance reads and evaluates columns referenced only by `filter` internally. If such a column is not referenced by `SELECT` or another Doris expression, it does not have to be returned to Doris. -An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each Fragment and before Doris performs its local and global TopN operations. +An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each search Split and before Doris performs its local and global TopN operations. ```sql SELECT user_id, category, _distance @@ -457,20 +459,24 @@ Consequently, an outer `WHERE` only filters candidates that have already been ge ### Current Execution Model -`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset: +`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset. Its Split boundary depends on index coverage: -1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. -2. Each Fragment becomes an independent Scan Split that can run in parallel on different BEs and Scanners. -3. For query parameters `top_k=K` and `offset=n`, each Fragment requests at most `K+n` candidates and does not apply the offset locally. Lance evaluates the TVF `filter` before generating candidates. Doris Scan evaluates an outer `WHERE` after candidate generation. -4. Doris performs local TopN, Exchange, and global TopN over candidates from every Fragment, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. +1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. When `use_index=true`, it also reads vector-index metadata. +2. If a compatible logical vector index has usable segment coverage, each physical index segment that still covers visible Fragments becomes one indexed Scan Split. The Split contains that segment's UUID and the intersection of its Fragment bitmap with the visible Fragments in the pinned snapshot. A Split can therefore contain multiple Fragment IDs. +3. Every visible Fragment not covered by those indexed Splits becomes an independent fallback Fragment Split. This keeps data appended after index creation searchable without requiring the index to be optimized first. If no usable index-segment plan exists, all visible Fragments use Fragment splits. With `use_index=false`, all visible Fragments use Flat Search splits directly. +4. For query parameters `top_k=K` and `offset=n`, every indexed or fallback Split requests at most `K+n` candidates and does not apply the offset locally. An indexed Split searches only its assigned physical index segment; a fallback Fragment Split performs Flat Search for its Fragment. Lance evaluates the TVF `filter` before generating candidates, while Doris Scan evaluates an outer `WHERE` afterward. +5. Doris performs local TopN, Exchange, and global TopN over candidates from all Splits, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. -A Fragment-level TopN therefore supplies candidates for global merging and is not the final result. All Fragments and any later Row-ID fetches use the same pinned snapshot throughout the query. +A Split-level candidate set therefore only supplies candidates for global merging and is not the final result. Index-segment splits, fallback Fragment splits, and any later Row-ID fetches all use the same pinned snapshot throughout the query. Refreshing index coverage changes how newly appended Fragments are searched, but uncovered Fragments remain part of the result space through Flat Search. The execution order is: ```text Pinned dataset snapshot - -> Per Fragment: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates + -> FE Split planning + -> Indexed coverage: one Split per physical Index Segment -> ANN Search + -> Uncovered or unindexed data: one Split per Fragment -> Flat Search + -> Per Split: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates -> Doris Scan Post-filter -> Doris local TopN -> Exchange @@ -511,7 +517,7 @@ A typical two-phase column flow is: | Stage or operator | Columns read or output | Description | |---|---|---| -| Lance Fragment Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Fragment produces at most `K+n` candidates. | +| Lance Split Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Index Segment or fallback Fragment Split produces at most `K+n` candidates. | | Doris Scan Post-filter | `_distance`, `category`, and the internal Row Location | Evaluates `category = 'book'`. A column used by an outer `WHERE` must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into an internal Row Location, which Fetch resolves to the same pinned snapshot. | | Local and global TopN | Phase-1 required columns and the internal Row Location | Global TopN merges by `_distance` and applies `offset=n` and `limit=K`. | | Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`, and `payload` | Reads deferred columns for rows retained by global TopN from the same Lance dataset snapshot without rescanning Fragments. | @@ -521,7 +527,7 @@ Phase-1 required columns are not limited to `_distance` and Post-filter columns. A Prefilter column referenced in the TVF `filter` differs from an outer `WHERE` column. Lance uses the former internally during search, so appearing in the `filter` string alone does not require the column to be returned to Doris. Doris Scan evaluates the latter, so it must be present in Phase 1. -Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Fragment and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Whether Lance scans every vector depends on `use_index`: a compatible index performs indexed search, while forced or fallback Flat Search directly compares vectors. +Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Index Segment or fallback Fragment Split and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Indexed splits use their assigned physical index segments, while forced or fallback Flat Search directly compares vectors. ## Current Limitations and Recommendations @@ -531,4 +537,5 @@ Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the - For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify which conditions have been pushed down. - Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set `"use_index" = "false"` to perform Flat Search. - For deterministic vector result ordering, explicitly use `ORDER BY _distance ASC` and add a unique tie-breaker. -- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each Fragment and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. +- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each search Split and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. +- Use `EXPLAIN` to inspect `lanceSearchFragments` and `lanceSearchIndexSegments`. The former is the number of visible Fragments in the pinned snapshot; the latter is the number of physical Index Segment splits selected by the FE. Additional fallback Fragment splits may also be present. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx index 4a30bbd91724b..972212034ceb2 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx @@ -26,7 +26,7 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan | 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 | | 谓词下推 | 支持将部分标量谓词下推到 Lance 执行 | | 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 | -| 向量检索 | 支持通过 `vector_search()` 按 Fragment 并行查询 Lance 向量索引或执行 Flat Search，并由 Doris 合并全局 Top-K | +| 向量检索 | 使用物理 Lance Index Segment 作为并行 Split，对未覆盖的 Fragment 保留 Flat Search Split，并由 Doris 合并全局 Top-K | | 写入 Lance | 暂不支持 | | Time Travel | 暂不支持 | | Full-Text Search / Hybrid Search | 暂不支持 | @@ -402,7 +402,7 @@ ORDER BY _distance ASC, user_id; | `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用 Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 | | `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的 `_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N` 个候选，再用原始向量计算真实距离并重新排序。即使设置为 `1` 也会执行精排，因此与不设置不同。 | | `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW 图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset) × refine_factor)`。对非 HNSW 索引无效。 | -| `use_index` | 否 | `true` | `true` 表示存在兼容索引时优先使用索引，否则自动执行 Flat Search；`false` 强制执行 Flat Search。 | +| `use_index` | 否 | `true` | `true` 表示将兼容的物理 Lance Index Segment 规划为索引 Split，并将未覆盖的 Fragment 保留为 Flat Search Split；如果没有可用的兼容索引元数据，则退回按 Fragment 拆分。`false` 表示每个可见 Fragment 生成一个 Split，并强制执行 Flat Search。 | 以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的 Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。 @@ -419,11 +419,13 @@ ORDER BY _distance ASC, user_id; | `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization | `nprobes`、`ef`、`refine_factor` | | `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization | `nprobes`、`ef`、`refine_factor` | -`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，Doris 将查询交给 Lance，由 Lance 选择与向量列和 Metric 兼容的现有索引；没有兼容索引时执行 Flat Search。当 `use_index=false` 时强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 +`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，FE 从固定的数据集快照读取向量索引元数据，并选择与向量列和 Metric 兼容的一个逻辑索引；随后将该逻辑索引中仍覆盖可见数据的每个物理 Segment 分配给一个索引 Scan Split。每个索引 Split 都携带 Segment UUID 以及该 Segment 覆盖且在当前快照中可见的 Fragment，因此 BE 会检索指定的物理 Segment，而不是再次让 Lance 自行选择索引。 + +一个 Lance 逻辑索引可以包含多个物理 Index Segment，一个物理 Segment 也可以覆盖多个 Fragment。未被所选索引覆盖的 Fragment 不会被遗漏：Doris 会为每个这样的 Fragment 增加一个执行 Flat Search 的回退 Split。如果 FE 无法生成可用的 Index Segment 计划，则退回按 Fragment 拆分。当 `use_index=false` 时，Doris 跳过索引元数据规划，并对每个可见 Fragment 强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 ### Prefilter 和 Post-filter -TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个 Fragment 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： +TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个搜索 Split 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： ```sql SELECT user_id, category, _distance @@ -439,7 +441,7 @@ ORDER BY _distance ASC, user_id; `filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给 Doris。 -外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个 Fragment 生成候选之后、Doris 执行局部和全局 TopN 之前。 +外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个搜索 Split 生成候选之后、Doris 执行局部和全局 TopN 之前。 ```sql SELECT user_id, category, _distance @@ -457,20 +459,24 @@ ORDER BY _distance ASC, user_id; ### 当前执行方式 -`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集： +`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集。Split 的边界取决于索引覆盖范围： -1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment。 -2. 每个 Fragment 生成一个独立的 Scan Split，可以分发到不同 BE 和 Scanner 并行执行。 -3. 假设查询参数为 `top_k=K`、`offset=n`，每个 Fragment 都请求最多 `K+n` 个候选，并且不在 Fragment 内应用 offset。TVF 的 `filter` 在候选生成前由 Lance 执行；外层 `WHERE` 在候选生成后由 Doris Scan 执行。 -4. Doris 对所有 Fragment 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 +1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment；当 `use_index=true` 时，还会读取向量索引元数据。 +2. 如果存在具有可用 Segment 覆盖信息的兼容逻辑向量索引，每个仍覆盖可见 Fragment 的物理 Index Segment 都会生成一个索引 Scan Split。该 Split 包含 Segment UUID，以及其 Fragment Bitmap 与固定快照中可见 Fragment 的交集，因此一个 Split 可以包含多个 Fragment ID。 +3. 没有被这些索引 Split 覆盖的每个可见 Fragment，都会生成一个独立的回退 Fragment Split。这样，即使数据是在索引创建后追加的、尚未执行索引优化，也仍然可以被检索。如果不存在可用的 Index Segment 计划，所有可见 Fragment 都按 Fragment 拆分；当 `use_index=false` 时，所有可见 Fragment 直接使用 Flat Search Split。 +4. 假设查询参数为 `top_k=K`、`offset=n`，每个索引或回退 Split 都请求最多 `K+n` 个候选，并且不在 Split 内应用 offset。索引 Split 只检索为其分配的物理 Index Segment，回退 Fragment Split 对自身 Fragment 执行 Flat Search。TVF 的 `filter` 在候选生成前由 Lance 执行，外层 `WHERE` 则在候选生成后由 Doris Scan 执行。 +5. Doris 对所有 Split 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 -因此，Fragment 级 TopN 只是为全局合并提供候选，不能直接视为最终结果。查询期间所有 Fragment 以及后续按 Row ID 取列都使用同一个固定快照。 +因此，Split 级候选集只用于向全局合并提供候选，不能直接视为最终结果。索引 Segment Split、回退 Fragment Split 以及后续按 Row ID 取列都使用同一个固定快照。刷新索引覆盖会改变新追加 Fragment 的检索方式，但未被索引覆盖的 Fragment 仍会通过 Flat Search 进入检索范围。 执行顺序可以概括为： ```text 固定数据集快照 - -> 每个 Fragment：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 + -> FE Split 规划 + -> 索引覆盖：每个物理 Index Segment 一个 Split -> ANN Search + -> 未覆盖或无索引数据：每个 Fragment 一个 Split -> Flat Search + -> 每个 Split：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 -> Doris Scan Post-filter -> Doris 局部 TopN -> Exchange @@ -511,7 +517,7 @@ WHERE category = 'book'; | 阶段或算子 | 读取或输出的列 | 说明 | |---|---|---| -| Lance Fragment Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Fragment 最多产生 `K+n` 个候选。 | +| Lance Split Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Index Segment 或回退 Fragment Split 最多产生 `K+n` 个候选。 | | Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行 `category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row Location，Fetch 再通过该映射解析到同一个固定快照。 | | 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用 `offset=n` 和 `limit=K`。 | | Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN 保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 | @@ -521,7 +527,7 @@ WHERE category = 'book'; TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在 `filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。 -将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Fragment 并行生成候选并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。是否在 Lance 内扫描全部向量取决于 `use_index`：使用兼容索引时执行索引搜索，强制或回退到 Flat Search 时才需要直接比较向量。 +将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Index Segment 或回退 Fragment Split 并行生成候选，并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。索引 Split 使用为其分配的物理 Index Segment，强制或回退到 Flat Search 时才直接比较向量。 ## 当前限制和建议 @@ -531,4 +537,5 @@ TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 L - 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。 - 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat Search。 - 向量查询需要稳定顺序时，显式使用 `ORDER BY _distance ASC` 并增加唯一 Tie-breaker。 -- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个 Fragment 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 +- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个搜索 Split 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 +- 使用 `EXPLAIN` 检查 `lanceSearchFragments` 和 `lanceSearchIndexSegments`。前者表示固定快照中的可见 Fragment 数量，后者表示 FE 选择的物理 Index Segment Split 数量；此外还可能存在回退 Fragment Split。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index 4a30bbd91724b..972212034ceb2 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -26,7 +26,7 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan | 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 | | 谓词下推 | 支持将部分标量谓词下推到 Lance 执行 | | 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 | -| 向量检索 | 支持通过 `vector_search()` 按 Fragment 并行查询 Lance 向量索引或执行 Flat Search，并由 Doris 合并全局 Top-K | +| 向量检索 | 使用物理 Lance Index Segment 作为并行 Split，对未覆盖的 Fragment 保留 Flat Search Split，并由 Doris 合并全局 Top-K | | 写入 Lance | 暂不支持 | | Time Travel | 暂不支持 | | Full-Text Search / Hybrid Search | 暂不支持 | @@ -402,7 +402,7 @@ ORDER BY _distance ASC, user_id; | `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用 Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 | | `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的 `_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N` 个候选，再用原始向量计算真实距离并重新排序。即使设置为 `1` 也会执行精排，因此与不设置不同。 | | `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW 图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset) × refine_factor)`。对非 HNSW 索引无效。 | -| `use_index` | 否 | `true` | `true` 表示存在兼容索引时优先使用索引，否则自动执行 Flat Search；`false` 强制执行 Flat Search。 | +| `use_index` | 否 | `true` | `true` 表示将兼容的物理 Lance Index Segment 规划为索引 Split，并将未覆盖的 Fragment 保留为 Flat Search Split；如果没有可用的兼容索引元数据，则退回按 Fragment 拆分。`false` 表示每个可见 Fragment 生成一个 Split，并强制执行 Flat Search。 | 以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的 Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。 @@ -419,11 +419,13 @@ ORDER BY _distance ASC, user_id; | `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization | `nprobes`、`ef`、`refine_factor` | | `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization | `nprobes`、`ef`、`refine_factor` | -`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，Doris 将查询交给 Lance，由 Lance 选择与向量列和 Metric 兼容的现有索引；没有兼容索引时执行 Flat Search。当 `use_index=false` 时强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 +`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true` 时，FE 从固定的数据集快照读取向量索引元数据，并选择与向量列和 Metric 兼容的一个逻辑索引；随后将该逻辑索引中仍覆盖可见数据的每个物理 Segment 分配给一个索引 Scan Split。每个索引 Split 都携带 Segment UUID 以及该 Segment 覆盖且在当前快照中可见的 Fragment，因此 BE 会检索指定的物理 Segment，而不是再次让 Lance 自行选择索引。 + +一个 Lance 逻辑索引可以包含多个物理 Index Segment，一个物理 Segment 也可以覆盖多个 Fragment。未被所选索引覆盖的 Fragment 不会被遗漏：Doris 会为每个这样的 Fragment 增加一个执行 Flat Search 的回退 Split。如果 FE 无法生成可用的 Index Segment 计划，则退回按 Fragment 拆分。当 `use_index=false` 时，Doris 跳过索引元数据规划，并对每个可见 Fragment 强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance 内直接读取并比较向量。 ### Prefilter 和 Post-filter -TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个 Fragment 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： +TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个搜索 Split 的 Lance Scanner，Lance 在 ANN 或 Flat Search 生成候选之前执行过滤： ```sql SELECT user_id, category, _distance @@ -439,7 +441,7 @@ ORDER BY _distance ASC, user_id; `filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给 Doris。 -外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个 Fragment 生成候选之后、Doris 执行局部和全局 TopN 之前。 +外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的 Prefilter。它的执行位置是：Lance 为每个搜索 Split 生成候选之后、Doris 执行局部和全局 TopN 之前。 ```sql SELECT user_id, category, _distance @@ -457,20 +459,24 @@ ORDER BY _distance ASC, user_id; ### 当前执行方式 -`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集： +`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集。Split 的边界取决于索引覆盖范围： -1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment。 -2. 每个 Fragment 生成一个独立的 Scan Split，可以分发到不同 BE 和 Scanner 并行执行。 -3. 假设查询参数为 `top_k=K`、`offset=n`，每个 Fragment 都请求最多 `K+n` 个候选，并且不在 Fragment 内应用 offset。TVF 的 `filter` 在候选生成前由 Lance 执行；外层 `WHERE` 在候选生成后由 Doris Scan 执行。 -4. Doris 对所有 Fragment 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 +1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment；当 `use_index=true` 时，还会读取向量索引元数据。 +2. 如果存在具有可用 Segment 覆盖信息的兼容逻辑向量索引，每个仍覆盖可见 Fragment 的物理 Index Segment 都会生成一个索引 Scan Split。该 Split 包含 Segment UUID，以及其 Fragment Bitmap 与固定快照中可见 Fragment 的交集，因此一个 Split 可以包含多个 Fragment ID。 +3. 没有被这些索引 Split 覆盖的每个可见 Fragment，都会生成一个独立的回退 Fragment Split。这样，即使数据是在索引创建后追加的、尚未执行索引优化，也仍然可以被检索。如果不存在可用的 Index Segment 计划，所有可见 Fragment 都按 Fragment 拆分；当 `use_index=false` 时，所有可见 Fragment 直接使用 Flat Search Split。 +4. 假设查询参数为 `top_k=K`、`offset=n`，每个索引或回退 Split 都请求最多 `K+n` 个候选，并且不在 Split 内应用 offset。索引 Split 只检索为其分配的物理 Index Segment，回退 Fragment Split 对自身 Fragment 执行 Flat Search。TVF 的 `filter` 在候选生成前由 Lance 执行，外层 `WHERE` 则在候选生成后由 Doris Scan 执行。 +5. Doris 对所有 Split 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局 TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。 -因此，Fragment 级 TopN 只是为全局合并提供候选，不能直接视为最终结果。查询期间所有 Fragment 以及后续按 Row ID 取列都使用同一个固定快照。 +因此，Split 级候选集只用于向全局合并提供候选，不能直接视为最终结果。索引 Segment Split、回退 Fragment Split 以及后续按 Row ID 取列都使用同一个固定快照。刷新索引覆盖会改变新追加 Fragment 的检索方式，但未被索引覆盖的 Fragment 仍会通过 Flat Search 进入检索范围。 执行顺序可以概括为： ```text 固定数据集快照 - -> 每个 Fragment：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 + -> FE Split 规划 + -> 索引覆盖：每个物理 Index Segment 一个 Split -> ANN Search + -> 未覆盖或无索引数据：每个 Fragment 一个 Split -> Flat Search + -> 每个 Split：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选 -> Doris Scan Post-filter -> Doris 局部 TopN -> Exchange @@ -511,7 +517,7 @@ WHERE category = 'book'; | 阶段或算子 | 读取或输出的列 | 说明 | |---|---|---| -| Lance Fragment Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Fragment 最多产生 `K+n` 个候选。 | +| Lance Split Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部 Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Index Segment 或回退 Fragment Split 最多产生 `K+n` 个候选。 | | Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行 `category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row Location，Fetch 再通过该映射解析到同一个固定快照。 | | 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用 `offset=n` 和 `limit=K`。 | | Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN 保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 | @@ -521,7 +527,7 @@ WHERE category = 'book'; TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在 `filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。 -将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Fragment 并行生成候选并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。是否在 Lance 内扫描全部向量取决于 `use_index`：使用兼容索引时执行索引搜索，强制或回退到 Flat Search 时才需要直接比较向量。 +将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果 `top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Index Segment 或回退 Fragment Split 并行生成候选，并由 Doris 合并全局 TopN；它不会因此退化为 Doris 对整张表做普通全列扫描。索引 Split 使用为其分配的物理 Index Segment，强制或回退到 Flat Search 时才直接比较向量。 ## 当前限制和建议 @@ -531,4 +537,5 @@ TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 L - 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。 - 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat Search。 - 向量查询需要稳定顺序时，显式使用 `ORDER BY _distance ASC` 并增加唯一 Tie-breaker。 -- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个 Fragment 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 +- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个搜索 Split 已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。 +- 使用 `EXPLAIN` 检查 `lanceSearchFragments` 和 `lanceSearchIndexSegments`。前者表示固定快照中的可见 Fragment 数量，后者表示 FE 选择的物理 Index Segment Split 数量；此外还可能存在回退 Fragment Split。 diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index e0a589b9cef2b..cf60bedd4e596 100644 --- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -26,7 +26,7 @@ Doris currently provides read-only access to Lance. Creating, writing, updating, | Data queries | Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version | | Predicate pushdown | Supports pushing compatible scalar predicates down to Lance | | File TVFs | Supports querying Lance datasets directly through `s3()` and `local()` | -| Vector search | Supports querying Lance vector indexes or performing Flat Search per Fragment through `vector_search()`, followed by a Doris global Top-K merge | +| Vector search | Uses physical Lance index segments as parallel splits, keeps uncovered Fragments as Flat Search splits, and performs a Doris global Top-K merge | | Writing to Lance | Not supported | | Time Travel | Not supported | | Full-Text Search / Hybrid Search | Not supported | @@ -402,7 +402,7 @@ Do not use the unquoted form `lance_catalog.doris.analytics.items`; it parses as | `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index partitions to probe. It must be a positive integer. When unset, Lance starts with one partition and can probe additional partitions when a Prefilter leaves too few candidates. Setting it explicitly to `N` fixes both the minimum and maximum number of probes to `N`. | | `refine_factor` | No | Refinement disabled | Candidate refinement multiplier. It must be a positive integer. When unset, Lance does not recompute distances from the original vectors, so `_distance` from a quantized index may be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N` candidates, recomputes their exact distances from the original vectors, and reorders them. Setting it to `1` still enables refinement and therefore differs from leaving it unset. | | `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained during HNSW graph search. It must be a positive integer. If `refine_factor` is also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It has no effect on non-HNSW indexes. | -| `use_index` | No | `true` | When `true`, Doris prefers a compatible Lance vector index and automatically falls back to Flat Search if none is available. When `false`, Doris forces Flat Search. | +| `use_index` | No | `true` | When `true`, Doris plans compatible physical Lance index segments as indexed splits and keeps uncovered Fragments as Flat Search splits. If no usable compatible index metadata is available, Doris falls back to Fragment splits. When `false`, Doris creates one split per visible Fragment and forces Flat Search. | These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When `metric` is omitted, Doris uses the metric configured when a compatible vector index was created. If there is no compatible index, or if `"use_index" = "false"`, `uint8` vectors use `hamming`, while the other currently supported vector element types use `l2`. @@ -419,11 +419,13 @@ The embedded `lance-c v0.1.6` explicitly supports the following Lance vector ind | `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`, `refine_factor` | | `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`, `refine_factor` | -`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, Doris delegates the query to Lance, and Lance selects an existing index compatible with the vector column and metric. If no compatible index is available, Lance performs Flat Search. With `use_index=false`, Flat Search is forced. Flat Search is not an ANN index type; Lance must directly read and compare vectors. +`vector_search()` only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With `use_index=true`, the FE reads vector-index metadata from the pinned dataset snapshot and selects a logical index compatible with the vector column and metric. It then assigns each physical segment of that logical index that still covers visible data to an indexed Scan Split. Each indexed Split carries the segment UUID and the currently visible Fragments covered by that segment, so the BE searches that specific segment instead of asking Lance to choose an index again. + +A logical Lance index can contain multiple physical index segments, and one physical segment can cover multiple Fragments. Fragments not covered by the selected index are not omitted: Doris adds one fallback Split for each such Fragment, which uses Flat Search. If the FE cannot construct a usable index-segment plan, it falls back to Fragment-level splits. With `use_index=false`, Doris skips index metadata planning and forces Flat Search for every visible Fragment. Flat Search is not an ANN index type; Lance must directly read and compare vectors. ### Prefilter and Post-Filter -The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each Fragment, and Lance evaluates it before ANN or Flat Search generates candidates: +The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat Search generates candidates: ```sql SELECT user_id, category, _distance @@ -439,7 +441,7 @@ ORDER BY _distance ASC, user_id; Lance reads and evaluates columns referenced only by `filter` internally. If such a column is not referenced by `SELECT` or another Doris expression, it does not have to be returned to Doris. -An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each Fragment and before Doris performs its local and global TopN operations. +An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each search Split and before Doris performs its local and global TopN operations. ```sql SELECT user_id, category, _distance @@ -457,20 +459,24 @@ Consequently, an outer `WHERE` only filters candidates that have already been ge ### Current Execution Model -`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset: +`vector_search()` uses distributed candidate search instead of one Scanner for the entire dataset. Its Split boundary depends on index coverage: -1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. -2. Each Fragment becomes an independent Scan Split that can run in parallel on different BEs and Scanners. -3. For query parameters `top_k=K` and `offset=n`, each Fragment requests at most `K+n` candidates and does not apply the offset locally. Lance evaluates the TVF `filter` before generating candidates. Doris Scan evaluates an outer `WHERE` after candidate generation. -4. Doris performs local TopN, Exchange, and global TopN over candidates from every Fragment, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. +1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. When `use_index=true`, it also reads vector-index metadata. +2. If a compatible logical vector index has usable segment coverage, each physical index segment that still covers visible Fragments becomes one indexed Scan Split. The Split contains that segment's UUID and the intersection of its Fragment bitmap with the visible Fragments in the pinned snapshot. A Split can therefore contain multiple Fragment IDs. +3. Every visible Fragment not covered by those indexed Splits becomes an independent fallback Fragment Split. This keeps data appended after index creation searchable without requiring the index to be optimized first. If no usable index-segment plan exists, all visible Fragments use Fragment splits. With `use_index=false`, all visible Fragments use Flat Search splits directly. +4. For query parameters `top_k=K` and `offset=n`, every indexed or fallback Split requests at most `K+n` candidates and does not apply the offset locally. An indexed Split searches only its assigned physical index segment; a fallback Fragment Split performs Flat Search for its Fragment. Lance evaluates the TVF `filter` before generating candidates, while Doris Scan evaluates an outer `WHERE` afterward. +5. Doris performs local TopN, Exchange, and global TopN over candidates from all Splits, merging by `_distance ASC`. Only the global TopN applies `offset=n`: it skips the first `n` rows and returns `K` rows. -A Fragment-level TopN therefore supplies candidates for global merging and is not the final result. All Fragments and any later Row-ID fetches use the same pinned snapshot throughout the query. +A Split-level candidate set therefore only supplies candidates for global merging and is not the final result. Index-segment splits, fallback Fragment splits, and any later Row-ID fetches all use the same pinned snapshot throughout the query. Refreshing index coverage changes how newly appended Fragments are searched, but uncovered Fragments remain part of the result space through Flat Search. The execution order is: ```text Pinned dataset snapshot - -> Per Fragment: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates + -> FE Split planning + -> Indexed coverage: one Split per physical Index Segment -> ANN Search + -> Uncovered or unindexed data: one Split per Fragment -> Flat Search + -> Per Split: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates -> Doris Scan Post-filter -> Doris local TopN -> Exchange @@ -511,7 +517,7 @@ A typical two-phase column flow is: | Stage or operator | Columns read or output | Description | |---|---|---| -| Lance Fragment Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Fragment produces at most `K+n` candidates. | +| Lance Split Search | Uses `embedding` internally; returns `_distance`, `category`, and the internal Lance Row ID to Doris | `embedding` participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Index Segment or fallback Fragment Split produces at most `K+n` candidates. | | Doris Scan Post-filter | `_distance`, `category`, and the internal Row Location | Evaluates `category = 'book'`. A column used by an outer `WHERE` must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into an internal Row Location, which Fetch resolves to the same pinned snapshot. | | Local and global TopN | Phase-1 required columns and the internal Row Location | Global TopN merges by `_distance` and applies `offset=n` and `limit=K`. | | Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`, and `payload` | Reads deferred columns for rows retained by global TopN from the same Lance dataset snapshot without rescanning Fragments. | @@ -521,7 +527,7 @@ Phase-1 required columns are not limited to `_distance` and Post-filter columns. A Prefilter column referenced in the TVF `filter` differs from an outer `WHERE` column. Lance uses the former internally during search, so appearing in the `filter` string alone does not require the column to be returned to Doris. Doris Scan evaluates the latter, so it must be present in Phase 1. -Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Fragment and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Whether Lance scans every vector depends on `use_index`: a compatible index performs indexed search, while forced or fallback Flat Search directly compares vectors. +Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the two-phase read. A single-phase read is also used when `top_k` exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Index Segment or fallback Fragment Split and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Indexed splits use their assigned physical index segments, while forced or fallback Flat Search directly compares vectors. ## Current Limitations and Recommendations @@ -531,4 +537,5 @@ Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables the - For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify which conditions have been pushed down. - Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set `"use_index" = "false"` to perform Flat Search. - For deterministic vector result ordering, explicitly use `ORDER BY _distance ASC` and add a unique tie-breaker. -- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each Fragment and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. +- Use the `vector_search()` `filter` parameter when filtering must occur before vector candidates are generated. An outer `WHERE` filters only the candidates already generated by each search Split and runs before Doris global TopN, so allow for a final result with fewer than `top_k` rows. +- Use `EXPLAIN` to inspect `lanceSearchFragments` and `lanceSearchIndexSegments`. The former is the number of visible Fragments in the pinned snapshot; the latter is the number of physical Index Segment splits selected by the FE. Additional fallback Fragment splits may also be present.