Skip to content

feat(manifest): record the release the catalog resolved to as catalog_release - #65

Merged
timtreis merged 2 commits into
mainfrom
feat-manifest-resolved-release
Nov 2, 2026
Merged

timtreis merged 2 commits into
mainfrom
feat-manifest-resolved-release

Conversation

@timtreis

@timtreis timtreis commented Nov 2, 2026

Copy link
Copy Markdown
Member

Summary: A manifest.json and a frozen cohort now record the dated catalog release that the data came from, including when the catalog was opened with Catalog() and no version. catalog_version keeps its current meaning as the opt-in pin that later downloads check, so rerunning a download of the latest catalog into the same folder still works after a new release. sddb-lead chose this over reusing catalog_version (option A).

Changes:

  • Catalog.release: version when set. For the default catalog-latest URL, it's the date in the parquet's sddb_generated_at, which matches the release tag for 2026-10-16, 2026-10-28 and 2026-11-01, and the 2026-11-01 file is byte-identical to catalog-latest. For any other URL it's None.
  • Source.release carries it to cohorts and, through a new optional source argument, to Dataset. So Catalog().dataset(uid).to_sdata(elements=..., dest=...) records it too.
  • Manifest.catalog_release is written by download(), by partial copies and by refetch(). Old manifests read with None.
  • The frozen cohort's source gets release. For a cohort frozen from catalog-latest with a recorded release, verify() checks the sha256 against that dated release, so a changed catalog file raises CatalogChangedError as it would for a pinned release. Before, it only warned because latest moves.
  • docs/start.md: replaces the sentence from docs: catalog download size; manifests record the release only when version= is set #64, which was wrong. version= alone never recorded the release, and only pin_versions=True did.

Checks:

  • pytest: 456 passed, 4 skipped. The new test test_unpinned_download_records_the_release_without_pinning checks that an unpinned download writes catalog_release, keeps catalog_version null, and doesn't raise when it's rerun after a new release.
  • Live, against the current catalog:
    • Catalog().release is "2026-11-01".
    • The partial copy and download() both record it.
    • The frozen source has release.
    • verify() passes.
    • Catalog(version="2026-10-28").release is "2026-10-28".
  • ruff check and ruff format are clean on the touched files.

@read-the-docs-community

read-the-docs-community Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.67%. Comparing base (4ac1f00) to head (59a96dd).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #65      +/-   ##
==========================================
+ Coverage   94.64%   94.67%   +0.03%     
==========================================
  Files          22       22              
  Lines        2278     2291      +13     
==========================================
+ Hits         2156     2169      +13     
  Misses        122      122              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timtreis
timtreis merged commit 227778c into main Nov 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant