Repository navigation
%gpkg: stream Dremio tables and views to a GeoPackage, pushable with %dds (ref #309764) - #100
Merged
Merged
Conversation
GpkgController exports the tables and views under a catalog path (default: the %catalog use(...) context) to one .gpkg file, written with sqlite3, shapely and pyproj (no GDAL): - Spatial or tabular is decided per table from the data: a column holding WKB/EWKB, WKT/EWKT, hex WKB or GeoJSON, else latitude/longitude columns. Spatial tables become feature layers, the rest attribute tables. - Each row's CRS comes from the geometry (EWKB/EWKT SRID), else a CRS column (srid, sr_id, epsg, crs, ...), else EPSG:4326 for latitude/longitude. - Options: tables, limit per table, srid to reproject to, filename, output_dir (default <dataset>.gpkg next to the notebook), overwrite, workers. - CRS definitions for reprojection and gpkg_spatial_ref_sys come from geopackage/epsg_definitions.json, falling back to pyproj's EPSG database. Streaming: workers tables are read at once, each over its own Arrow Flight gRPC channel, a batch at a time; workers decode, reproject and encode, and one thread owns the SQLite file, fed through a bounded queue. The file is built under a temporary name with no journal, in one transaction, and renamed into place only when every table succeeded. FlightSqlExecutor gains stream_batches (no overall deadline on reading, parallel endpoints, cancel on early stop), fork() and thread-safe login; Catalog gains stream_rows and open_flight_channel. Also ignores *:Zone.Identifier download metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Redmine-Hook: v1
- export prints a line as each table/view is written (layer, source, kind, geometry type, SRS, rows, seconds, notes) or fails (naming the table and the error; the export then stops and no file is written), plus a start line and a total. GpkgController takes a progress callback (default: log only); the %gpkg session passes print. Each layer records its seconds. - GpkgReport is os.PathLike (the written file), so `report = %gpkg export()` then `%dds push(DELIVERABLES, report)` uploads the .gpkg. .gpkg is registered as application/geopackage+sqlite3. - %gpkg / %dds help and the README show both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Redmine-Hook: v1
Brings development up to date with staging (ingest PY-7 to PY-13, %ingest diagnose / options / cleanup_on_failure, foldertree copy and move, non-UTF-8 CSV check, the main ingest fix). catalog/client.py conflicted only because both sides added to Catalog: kept both — stream_rows / open_flight_channel (development) and getentriesfrom / entryexists / getviewsql / createviewas / getfoldersfrom and the foldertree imports (staging). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge staging into development
test_a_gpkg_export_can_be_pushed_with_dds imported FakeSource from tests.geopackage.conftest. That works under `python -m pytest` (the repo root is on sys.path) but not under plain `pytest` as CI runs it, where `tests` is not importable. The test now defines its own one-table source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Redmine-Hook: v1
debugger/test_catalog_magic.ipynb gains cells for %gpkg help, exporting a dataset (limit=100) and pushing the result with %dds push(DELIVERABLES, ...). The output.gpkg those cells write is git-ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Redmine-Hook: v1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the GeoPackage export from development into staging. Ref #309764.
What's in it
%gpkgmagic andgeopackagepackage. It exports the tables and views under a Dremio catalog path (by default the%catalog usecontext) to a GeoPackage 1.4 file. It is written withsqlite3, shapely and pyproj only, with no GDAL.tables,limit,srid(reprojection),filename,output_dir,overwrite,workers.epsg_definitions.json; codes missing from it fall back to pyproj's EPSG database.FlightSqlExecutor.stream_batchesreads the endpoints in parallel, keeps a bounded queue, and is cancelled when the export stops early. A pool of Flight channels shares one authentication. A single SQLite writer thread does the inserts in bulk.report = %gpkg export()followed by%dds push(DELIVERABLES, report)works.tests, so plainpytestpasses.%gpkg helpexamples, a.gitignoreentry for*:Zone.Identifier, and a notebook example.Testing
tests/geopackage/(writer, spatial, crs, controller, session),tests/catalog/test_flight_stream.py, plus%gpkgtests intests/notebook/test_magics.py.🤖 Generated with Claude Code