Skip to content

refactor!: fold lance-csr-catalog into lance-csr and pin namespace reads - #124

Merged
prrao87 merged 3 commits into
mainfrom
refactor/fold-catalog-into-lance-csr
Nov 4, 2026
Merged

prrao87 merged 3 commits into
mainfrom
refactor/fold-catalog-into-lance-csr

Conversation

@prrao87

@prrao87 prrao87 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Note

The lance-csr-catalog crate is now retired. We only need to depend on lance-csr instead.

Why

Upcoming work on frontend and ingestion UIs will let applications that store graphs as plain Lance tables share one graph contract from lance-csr, instead of each keeping its own copy of the mapping types, its own mapping checks and its own code for opening tables at a pinned version. This PR prepares lance-csr for that, ahead of a 0.8.0 release.

Two things stand in the way:

  • lance-csr-catalog doesn't really require a crate of its own. It is about 240 lines, lance-csr is its only dependent, and publishing two crates in order adds release steps and a recovery procedure for when only the first one publishes.
  • Querying through a namespace always opened the latest version of each table and dropped the storage options the namespace returned, so a namespace could not pin a snapshot.

What changed

  • The catalog lives in lance-csr. GraphSourceCatalog, InMemoryCatalog, SimpleTableSource and DirNamespace move into the lance_csr::source_catalog and lance_csr::namespace modules. The root re-exports are unchanged, so Python users and code that imports lance_csr::DirNamespace see no difference. lance-csr-catalog is no longer built, version-bumped or published, and the release guide drops its two-crate recovery steps.
  • Namespace reads honour pinned versions. CypherQuery::execute_with_namespace accepts any LanceNamespace, such as Lance's own DirectoryNamespace, and opens each table at the version and with the storage options describe_table returns. With no version it opens the latest, as before, which is always the case for DirNamespace.
  • A shared graph mapping. A new mapping module adds:
    • GraphMapping, the version 1 JSON graph descriptor (label to table, key, display column, and relationship endpoints), with validate() for the rules that need no table data: at least one node label, at most 64 labels, labels that are ASCII identifiers unique ignoring case, one graph role per table, and endpoints that name a mapped node label.
    • GraphMapping::graph_config, which builds a GraphConfig from the tables' Arrow schemas and checks that key, display and endpoint columns exist, keys are scalar, and endpoint types match the node keys.
    • GraphSnapshot, a list of tables pinned to a path and Lance version plus the mapping. It ignores unknown fields, so it reads a publication manifest that also carries row counts or schemas. open() opens every mapped table at its pinned version and returns what CypherEngine::new takes.
  • open_lance_dataset is public, as the one helper for opening a table at a pinned version.

Breaking (Rust only)

  • The lance-csr-catalog crate is gone; depend on lance-csr instead.
  • execute_with_namespace takes impl LanceNamespace and execute_with_namespace_arc takes Arc<dyn LanceNamespace>. Existing calls with a DirNamespace or Arc<DirNamespace> still compile.

The version bump to 0.8.0 follows in its own PR, as RELEASE.md describes. The crates.io page for lance-csr-catalog will keep its 0.7.1 README until a final version pointing to lance-csr is published by hand.

Validation

  • cargo test --workspace: all pass, including new tests for the mapping checks, reading a descriptor and a publication manifest, opening a snapshot at a pinned version, and a namespace that pins version 1 after the table was overwritten.
  • cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all --check: clean.
  • Python: make install, make test (1706 passed) and make lint: clean.
  • uv run --project docs --locked zensical build --strict and uvx typos@1.26.0 .: clean.
  • CI pending.

🤖 Generated with Claude Code

Move GraphSourceCatalog, InMemoryCatalog, SimpleTableSource and
DirNamespace into lance-csr and stop building and publishing the
lance-csr-catalog crate. The root re-exports are unchanged.

execute_with_namespace now accepts any LanceNamespace and opens each
table at the version and with the storage options describe_table
returns.

Add a mapping module with GraphMapping, its structural checks,
GraphSnapshot for pinned tables, and conversion to GraphConfig, and
make open_lance_dataset public.

Breaking for Rust: the lance-csr-catalog crate is gone, and the
namespace methods take a LanceNamespace trait object.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Deploying lance-csr with  Cloudflare Pages  Cloudflare Pages

Latest commit: 06a0b30
Status:⚡️  Build in progress...

View logs

@prrao87
prrao87 marked this pull request as ready for review November 4, 2026 15:31
@prrao87
prrao87 merged commit caa8c2d into main Nov 4, 2026
2 of 3 checks passed
@prrao87
prrao87 deleted the refactor/fold-catalog-into-lance-csr branch November 4, 2026 16:28
prrao87 added a commit that referenced this pull request Nov 4, 2026
Minor bump: #124 is a breaking change (refactor!). It removes the lance-csr-catalog crate and changes the Rust namespace method signatures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant