Skip to content

General (but specially in b2view) improvements for accessing remote datasets (Caterva2, but also fsspec based) - #733

Merged
FrancescAlted merged 35 commits into
mainfrom
cat2-improvements
Oct 6, 2026
Merged

FrancescAlted merged 35 commits into
mainfrom
cat2-improvements

Conversation

@FrancescAlted

Copy link
Copy Markdown
Member
  • Add automatic Caterva2 discovery and lazy browsing of repositories, mounts, and datasets.
  • Extend b2view to local and fsspec files/directories, with bounded tree search and improved navigation.
  • Preview text, images, compressed documents, and saved Jupyter notebook outputs without executing code; support safe downloads and external opening.
  • Improve viewer startup, loading feedback, remote lifecycle handling, and error recovery.
  • Fix Caterva2 structured dtypes, lazy Parquet slicing, and schema metadata round-trips.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved security and remote-file correctness issues need fixes before approval.

Review effort: Balanced
Findings: 2 High severity · 4 Medium severity

Open (6)
What changed in this PR

Extends Python-Blosc2’s remote-access layer and b2view with lazy service browsing and broader file support.

Changes:

  • Adds Caterva2 discovery, repository browsing, and original-byte file downloads.
  • Adds local/fsspec browsing, bounded tree search, and passive document/notebook previews.
  • Improves dtype and Parquet handling, regression coverage, and documentation.
File Description
tests/​test_remote_file.py Tests file transport, caching, and downloads.
tests/​test_remote_caterva2.py Extends deterministic service fixtures.
tests/​test_caterva2_gateway.py Adds gateway acceptance tests.
tests/​test_caterva2_dtype.py Tests structured and subarray dtypes.
tests/​ctable/​test_schema_compiler.py Tests tagged non-finite metadata.
tests/​ctable/​test_remote_parquet.py Tests bounded Parquet selections.
tests/​b2view/​test_tree_search.py Tests search limits and navigation.
tests/​b2view/​test_startup_errors.py Tests clean startup failure handling.
tests/​b2view/​test_remote_compressed_files.py Tests range-backed document carriers.
tests/​b2view/​test_ordinary_files.py Tests direct file previews and copying.
tests/​b2view/​test_notebooks.py Tests passive notebook previews.
tests/​b2view/​test_files.py Tests previews and explicit file actions.
tests/​b2view/​test_directories.py Tests directories and dataset mounts.
tests/​b2view/​test_compressed_files.py Tests local compressed documents.
tests/​b2view/​test_cli.py Updates backend-option expectations.
tests/​b2view/​test_caterva2.py Tests service browsing and background loading.
src/​blosc2/​schunk.py Adds service discovery and opening routes.
src/​blosc2/​schema_compiler.py Decodes tagged non-finite values.
src/​blosc2/​remote_repository.py Adds lazy multi-root browsing.
src/​blosc2/​remote_file.py Adds bounded byte reads and downloads.
src/​blosc2/​proxy_source.py Adds range-backed SChunk document reading.
src/​blosc2/​ctable.py Avoids unnecessary Parquet scans.
src/​blosc2/​caterva2_url.py Adds URL recognition and discovery.
src/​blosc2/​c2array.py Parses structured dtype literals.
src/​blosc2/​b2view/​tree_search.py Implements bounded listing searches.
src/​blosc2/​b2view/​search_screen.py Adds the search dialog.
src/​blosc2/​b2view/​render.py Renders file and notebook previews.
src/​blosc2/​b2view/​ordinary_file.py Adapts local and fsspec files.
src/​blosc2/​b2view/​notebook_preview.py Parses bounded passive notebook content.
src/​blosc2/​b2view/​file_preview.py Adds previews and restricted external opening.
src/​blosc2/​b2view/​directory.py Adds lazy directories and mounted datasets.
src/​blosc2/​b2view/​compressed_file.py Decodes compressed document carriers.
src/​blosc2/​b2view/​cli.py Exposes backend selection and exit status.
src/​blosc2/​__init__.py Exports new remote types.
RELEASE_NOTES.md Summarizes access and viewer improvements.
pyproject.toml Adds the image-preview extra.
plans/​caterva2-demo-dataset-audit.md Records demo audit results.
plans/​b2view-tree-search.md Records search behavior and limits.
doc/​reference/​remotestore.rst Documents lazy Caterva2 groups.
doc/​reference/​remoterepository.rst Documents repository browsing.
doc/​reference/​remotefile.rst Documents byte reads and downloads.
doc/​reference/​classes.rst Links new remote references.
doc/​guides/​remote_objects.md Explains discovery and cache scope.
doc/​guides/​b2view.rst Documents browsing, previews, and search.
doc/​getting_started/​installation.rst Documents image-preview dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/blosc2/b2view/directory.py
Comment thread src/blosc2/caterva2_url.py Outdated
Comment thread src/blosc2/b2view/compressed_file.py Outdated
Comment thread src/blosc2/remote_file.py Outdated
Comment thread src/blosc2/remote_file.py
Comment thread tests/test_caterva2_gateway.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-cutting discovery, transport, decoding, and viewer changes retain unresolved risks and require human integration review.

Review effort: Balanced
Findings: 2 High severity

Open (2)
Resolved since last review (6)
Previously missed (2)

In code that hasn't changed since last review

Medium severity Normalize lazy=None for automatic service selection

src/​blosc2/​schunk.py:3162

lazy=None means automatic selection, but setdefault leaves it unchanged here and in _open_service_base. Consequently, both open("https://host/@public/group", lazy=None) and opening a single-root service with lazy=None take the nonlazy path and reject the group, whereas omitting the argument succeeds. Array URLs also return C2Array rather than the default RemoteArray. Normalize None to an omitted option before these string-service branches, preserving explicit False and existing URLPath behavior.

Low severity Align extras table rows and continuation lines

doc/​getting_started/​installation.rst:40

The hires and images row markers are one space deeper than the surrounding list-table rows, and their cells and continuation lines are inconsistently indented. This breaks the two-level list structure required for the extras table. Align both complete rows with tui and parquet.

Comment thread src/blosc2/b2view/ordinary_file.py Outdated
Comment thread src/blosc2/caterva2_url.py Outdated
@FrancescAlted
FrancescAlted merged commit c90a3eb into main Oct 6, 2026
42 checks passed
@FrancescAlted
FrancescAlted deleted the cat2-improvements branch October 6, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants