[CHORE] Deprecate overture_releases.yaml#529
Open
John McCall (lowlydba) wants to merge 3 commits into
Open
Conversation
- Replace fetch-releases-from-s3.py with fetch_releases_from_stac.py driven by https://stac.overturemaps.org/catalog.json instead of S3 listing - Drop obstore dependency from requirements.txt - Add deprecation header to overture_releases.yaml (kept for existing consumers) - Add index.html landing page for labs.overturemaps.org/data (was 404) - Add unit tests (19) for fetch_releases_from_stac.py - Add test.yml CI workflow (runs on push + PR) - Update Python 3.11 -> 3.12 in both workflows Published bundle before/after: releases.json S3 listing (~30 entries) -> STAC catalog (~2 active releases) latest.ddb unchanged latest.dbb unchanged (legacy alias, now marked deprecated) registry-manifest unchanged overture_releases.yaml unchanged data, gains deprecation comment index.html new: directory landing page (was 404) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
John McCall (lowlydba)
requested review from
Jennings Anderson (jenningsanderson) and
Francisco Jiménez (jjcfrancisco)
There was a problem hiding this comment.
Pull request overview
This PR shifts release discovery for published artifacts away from an S3 listing + overture_releases.yaml to the Overture STAC catalog, while keeping legacy outputs (e.g., releases.json, latest.ddb/.dbb, overture_releases.yaml) available but increasingly marked as deprecated. It also adds a simple landing page and introduces CI test coverage for the new STAC-based fetch script.
Changes:
- Replace the S3-based release fetch script with
fetch_releases_from_stac.pythat reads fromhttps://stac.overturemaps.org/catalog.json, and remove theobstoredependency. - Add unit tests for the new fetch script and a new GitHub Actions workflow to run them.
- Add a root
index.htmldirectory landing page and add a deprecation header tooverture_releases.yaml.
Reviewed changes
Copilot reviewed 9 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
utils/tests/test_fetch_releases_from_stac.py |
Adds unit tests for STAC fetch + release parsing + DuckDB view SQL generation. |
utils/tests/__init__.py |
Initializes the utils.tests package. |
utils/requirements.txt |
Removes obstore; retains DuckDB + PyArrow requirements. |
utils/requirements-test.txt |
Adds pytest test requirements for the utils tests. |
utils/fetch-releases-from-s3.py |
Removes legacy S3-listing-based script. |
utils/fetch_releases_from_stac.py |
New STAC-based implementation for generating releases.json and DuckDB views. |
overture_releases.yaml |
Adds a deprecation header pointing consumers to STAC. |
index.html |
Adds a new static landing page listing published artifacts and deprecation notes. |
.gitignore |
Adds Python bytecode/cache ignores. |
.github/workflows/test.yml |
Adds CI job to run utils unit tests on PRs and main branch pushes. |
.github/workflows/build-manifests.yml |
Updates publishing workflow to Python 3.12 and runs the new STAC fetch script; publishes index.html. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add 30s timeout + User-Agent header to fetch_catalog - Robust href parsing via urlparse (handles absolute + relative hrefs) - Close DuckDB connection in finally block - Remove unused test imports (tempfile, BytesIO, duckdb, pytest) - Update test_uses_provided_url to assert on Request.full_url + timeout - Add tests for absolute href and no-./-prefix relative href Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: John McCall <john@overturemaps.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #526
Published bundle before/after
releases.jsonlatest.ddblatestfieldlatest.dbblatest.ddbregistry-manifest.jsonsimple-registry-manifest.pyoverture_releases.yamlindex.htmlChanges
fetch-releases-from-s3.pywithfetch_releases_from_stac.py, sourcing releases from https://stac.overturemaps.org/catalog.jsonobstoredependency (no longer needed)overture_releases.yamlindex.htmllanding page for labs.overturemaps.org/datatest.ymlCI workflow running on push and PRsNew Index
For ease of consumption, a new index.html acts as a directory page that will land at
https://labs.overturemaps.org/dataQuestions
overture_releases.yaml?overture_releases.yamlon June's release and not update after that?