bring streaming search traits and get ready for pgstac 0.10#1077
Open
bitner wants to merge 5 commits into
Open
bring streaming search traits and get ready for pgstac 0.10#1077bitner wants to merge 5 commits into
bitner wants to merge 5 commits into
Conversation
bitner
added a commit
to stac-utils/pgstac
that referenced
this pull request
Jul 10, 2026
…tac#1077 on the rustac side
Addresses review feedback on #1088: the `BoxError` alias was public API that had nothing to do with streaming, and the writer returned it rather than the crate's own error. The stream types and `write_item_collection` now use `stac_io::Error`, and backends convert their own errors through a new `Error::Backend` boxed variant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follows the #1088 review change: `write_search` now returns `stac_io::Error`, so the CLI no longer needs `anyhow::Error::from_boxed`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bitner
added a commit
to stac-utils/pgstac
that referenced
this pull request
Jul 21, 2026
* feat: v0.10 export/streaming engine + configurable parquet row-groups (squash of v010-export-search onto main) main already carries the v0.10 foundation (#438 rust crate, #448 fragments, #456 keyset). This adds the v010-export-search work on top: the pgstac-rs export/dump/restore engine (Rust loader, search_plan search, dump/export) with Python bindings + the pgstac CLI; flat-memory CLI search streaming via the server portal with fields projection; configurable parquet row-group size (builder, --row-group-size, PGSTAC_PARQUET_ROW_GROUP_SIZE env, Python search_to_geoparquet); SQL maintenance/comment cleanup; pypgstac updates. Foundation overlaps resolved to the developed v010io versions. * refactor(pgstac-rs): drop profiling artifacts; fix 3D bbox footprint Move memprofile from a shipped [[bin]] to examples/ (it is a dev heap profiler, not a user tool). Remove the env-gated PGSTAC_LOAD_PROFILE / PGSTAC_PIPE_PROFILE / PGSTAC_DECODE_ONLY instrumentation from the ingest and parquet-pipeline hot paths. Fix Footprint::observe to read east/north at len/2 and len/2+1 so 6-element 3D bboxes are handled correctly (the old last-two logic placed north and max-elevation into east and north). Fix the stale stream_ndjson test call to pass the new token argument. * feat(pgstac-rs): implement stac::api client traits on PgstacPool PgstacPool now implements stac::api::{ItemsClient, CollectionsClient, StreamItemsClient, TransactionClient}, returning stac::api::ItemCollection via a new TryFrom<Page>, so a pgstac database is a first-class rustac backend. Each method routes through the existing engine (the keyset search in crate::search and the Rust loader for writes) - the rustac-native surface over one engine, not a second implementation. Enables stac's async feature for StreamItemsClient, whose search_stream maps the crate's flat-memory portal stream into stac::api Items (constant memory, not page-by-page). * ci: enable the rust-crate test job Turn the rust-crate CI job back on (it was gated off with if: false pending v0.10). Build the three template databases the tests need (clean + ingest templates and the rich fragment template) from the committed pgstac.sql and tracked fixtures via the setup scripts, then compile-check all features and run the suite with --features cli. Testing with cli rather than --all-features avoids linking libpython at runtime - the python feature is only a compile target here and has no runtime tests. * feat(pgstac): error on items with a nonexistent collection in SQL ingest items_staging_triggerfunc now raises if any staged item references a collection that does not exist, instead of silently dropping it in the collections JOIN below - matching the Rust loader, which already errors on this. Validated with pgtap + basicsql + pg_dump/restore (no regression). pgstac.sql reassembled via stageversion (unreleased, no version bump); the regenerated migrations are intentionally not committed (v0.10 migrations regenerate per-run and are excluded from the gate). * refactor(pgstac-rs): reorganize modules; drop dev artifacts and dead role Groups the flat crate root into db/ read/ load/ json/ api/ python/ and finishes the rustac-native + cleanup pass. Precheck stays a library method, not yet wired into load. pgstac-rs: - reorg: flat root modules into db/ read/ load/ json/ api/ python/ (public API unchanged) - search CLI dedup: route the itemcollection command through stac::api ItemsClient::search; delete SearchSource and page_to_feature_collection; map bad search tokens to Error::InvalidToken - remove the profiling feature, memprofile example, and dhat dep (preserved on the archive/memprofile branch); drop the lib.rs dhat shim - drop the EXPERIMENTAL hedge from the precheck docs; comments describe current behavior only pgstac (SQL): - remove the unused pgstac_load up-privilege role (added this cycle, never released); the SECURITY DEFINER write functions remain the only write path through the privilege wall - add a pgtap proof-test that the wall holds: pgstac_ingest cannot directly INSERT/UPDATE/DELETE items, partition_stats, or item_fragments - regenerate pgstac.sql * feat(pgstac-rs): add --skip-unchanged precheck to load; id-only for ignore Opt-in --skip-unchanged toggle on `load` routes each batch through the per-partition precheck before loading: on re-ingest / sync it skips items already present-and-current and loads only new + changed. Benchmarks (NAIP 89.8k, release): re-ingest identical ~3.8-5.1x faster, 10%-changed ~3.3x; ~18% slower on an all-new load, so it stays opt-in. - policy-aware precheck: ignore/error skip an existing id by id alone (no content hash, and skip changed-existing too, matching ignore semantics); upsert/delsert keep the content-hash compare - error + --skip-unchanged is rejected (error must fail on an existing id, not skip it) - the survivor-count preflight bump makes partition_stats.n tighter on a precheck load (ignore exact, upsert new+changed) with no extra query and no stats-on-flush - concurrency test covering the id-only ignore path * fix(pgstac-rs): gate precheck behind `pool`; add feature-matrix CI The precheck path takes `&PgstacPool`, so it only compiles with the `pool` feature, but CI only ran `cargo check --all-features` — so the no-`pool` build was broken (E0425: cannot find `PgstacPool`). Gate the precheck items and their helpers behind `#[cfg(feature = "pool")]`, gate the two export-only integration tests, and replace the single all-features check with a feature-matrix check (no-default / pool / export / cli / all-features) so the break cannot recur. Per the #458 review. * docs(pgstac-rs): fix broken rustdoc links + features list; check docs in CI cargo doc emitted 5 broken-intra-doc-link warnings — links to private / non-Rust items (precheck_one_partition, hydrate_fragment_core, PgConn, stac_daterange) become plain code, and GenericClient links to its real tokio_postgres path. Also refresh the crate features list (pool/export/cli/python; the old `tls` entry was stale) and the export module wording. Add a RUSTDOCFLAGS=-D warnings `cargo doc` CI step so doc warnings cannot regress. Per the #458 review. * docs(pgstac): correct the jsonb_canonical "RFC 8785" comments The SQL comments called jsonb_canonical/jsonb_hash "RFC 8785-aligned" and pointed external implementers at the `rfc8785` crate / a generic RFC 8785 canonicalizer — but the actual form is a custom canonicalization (byte-order keys via COLLATE "C" + `float8::text` numbers) that is NOT RFC 8785 and would hash differently. Correct the comments so nobody reimplements the hash against the wrong spec. The Rust side already documented this correctly, and parity is CI-gated by tests/canonical_parity.rs. Comment-only; pgstac.sql regenerated. * refactor(pgstac-rs): dedup the CLI policy enum; erroring int cast; named constants - CLI --policy uses `ConflictPolicy` directly (cli-gated `ValueEnum` derive), dropping the duplicate `LoadPolicy` enum + its `From` impl; `delsert` is now also a CLI policy. - The Int queryable dehydrate uses `i32::try_from` (erroring) instead of `as i32`, so an out-of-range integer property fails loudly instead of silently truncating. - `DEFAULT_BATCH_SIZE` / `MAX_INGEST_PARALLELISM` constants replace the inline 5000 / 8. Per the #458 review. * feat(pgstac): finish indexes_pending — defer queryable indexes to an async sweep New partitions are created index-light (indexes_pending=true) for fast ingest, but nothing consumed the flag. Wire it end-to-end: - the queryables trigger now flags every partition (SECURITY DEFINER, so it works for pgstac_ingest past the partition_stats write wall) instead of rebuilding all partitions synchronously in the trigger; - a new build_pending_indexes(_limit) maintenance sweep builds the missing per-partition queryable indexes and clears the flag, mirroring tighten_dirty_partition_stats (schedule via pg_cron / the maintenance CLI). Per the #458 review (indexes_pending was set but never consumed). * fix(pgstac-rs): set search_path on cloned test databases CREATE DATABASE ... TEMPLATE copies the schema but not the per-database settings (pg_db_role_setting) of the template, so each cloned per-test DB lost the search_path = pgstac, public the template sets — and every Pgstac-trait test that uses an unqualified pgstac reference failed in CI (pgstac_version etc. passed because they are qualified). Re-apply the search_path on the clone. Full cargo test --features cli is green, matrix + docs clean. * refactor(pgstac-rs): delete the Pgstac trait; one Client surface (#458) Having both the `Pgstac` trait (a bare-connection trait returning `Page`/`JsonValue`) and `Client`/`PgstacPool` (the `stac::api` traits returning stac types) was two public ways to do the same thing. The trait has no external consumers except rustac, so remove it entirely: - low-level SQL-call helpers move to `db::call` (pub(crate) free fns); the 3 unused ones (pgstac_vec/opt/value) are dropped - version/settings/collection-maintenance/delete_item become inherent `Client` methods; search/item/collections/collection/add_collection stay on the `stac::api` impls - `ItemsClient::search` folds in the malformed-token to InvalidToken mapping and uses the existing `TryFrom<Page>` instead of a hand-rolled ItemCollection rebuild (also carrying number_matched/number_returned/links the old code dropped) - the pool search/collections impls delegate to `Client` (one impl, shared cache) - `Pgstac` and `Page` leave the public API; the crate docs/examples lead with `Client`; the ~80 test sites move onto the Client surface - net -140 lines Engines (ingest/search/stream/hydrate) are untouched; a pre/post A/B shows no perf regression and streaming is unchanged. Follow-up (separate rustac PR): migrate rustac crates/server/src/backend/pgstac.rs from the `Pgstac` trait to the `stac::api` traits on `PgstacPool`/`Client`. * fix(pgstac-rs): an explicit --dsn overrides ambient PG* env `ConnectConfig::from_env` reads `PGDATABASE` (and `PGHOST`/`PGPORT`/...) into the config, and `to_pg_config` applied those over a later-set `--dsn` — so `pgstac load --dsn <db>` connected to `PGDATABASE` instead of the database the DSN named. In CI (PostgreSQL 18, `PGDATABASE=postgis`) the CLI load tests failed with `function upsert_collection(jsonb, text) does not exist`, because the CLI connected to `postgis` (no pgstac) rather than the cloned test database. Locally it was masked by an empty `PGDATABASE` and a pgstac-first cluster `search_path`. Add `ConnectConfig::with_dsn`, which makes an explicit DSN the authoritative connection target (clearing host/port/dbname/user/password from the environment while keeping TLS/options/search_path), and route the CLI `--dsn` flag through it. Verified on a local PG18 container with `PGDATABASE=postgis`. * address issues from reviews * fix issue in item hash parity between postgres/rust * fix tests * address issues from review * fix rust test runner * make sure rust and rust/py have feature parity and testing parity with pypgstac and remove pypgstac * further reduce gap between rustac and pgstac, requires stac-utils/rustac#1077 on the rustac side * pin to rustac branch so ci can pass * rename python module/package name to just pgstac, include the rust cli binary in the python wheel * refactor: adapt to stac-io's streaming writer returning stac_io::Error rustac#1088 review feedback dropped the public `BoxError` alias and moved the stream types and `write_item_collection` to `stac_io::Error`. Backend errors now convert through `Error::Backend` via a local `backend` helper. * chore: update stac-io patch to the pgstac-search-parity tip Picks up the streaming writer's move to `stac_io::Error`. --------- Anthropic Claude was used to assist in the development of this work.
gadomski
added a commit
that referenced
this pull request
Jul 21, 2026
Adds `stac_io::stream`: a streaming JSON writer for a search-response `ItemCollection`. The `features` array is written one item at a time, and the rest of the collection (links, context, counts) is supplied by a `finalize` callback after the items drain — the `next` link needs the last item, and a `numberMatched` count may run concurrently. Public surface: `StreamSearch`, `StreamedSearch`, `ItemStream`, `Finalize`, `write_item_collection`. ## Why this is split out This is the shared trait surface that pgstac 0.10 implements for its keyset-portal search, so a pgstac database can stream a search response with flat memory through the same writer rustac uses. Right now there's a dependency cycle: #1077 needs the unreleased pgstac 0.10 crate, and pgstac's `v010io` branch needs this `stac-io` code (it currently carries a `[patch.crates-io]` git dependency on the rustac branch). Landing this on its own breaks it: 1. **this PR** → release `stac-io` 0.3.1 2. pgstac `v010io` drops its `[patch.crates-io]` lines and builds against published `stac-io` → release pgstac 0.10 3. #1077 rebases with `pgstac = "0.10"` from crates.io Because the change is purely additive, `^0.3` resolves 0.3.1 — pgstac needs no version bump, just the patch removal. ## Scope Two files, no manifest changes: - `crates/io/src/stream.rs` (new) - `crates/io/src/lib.rs` (+2: module declaration and re-export) `futures` was already a `stac-io` dependency. No existing behavior is touched. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Pete Gadomski <pete.gadomski@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Streaming Search Traits and prep things for pgstac 0.10 work.