Repository navigation
Merge staging into development - #99
Merged
Merged
Conversation
Checks, in plain language and without starting anything, whether a transfer would work: DDS reachable (GET /health); the credential accepted (GET /ingest/limits, or the transfer list on a server predating it); DDS's own S3 write test (POST /storage-selftest); and a dry run of this exact folder, which covers the privileges on the target, the target folder, collisions and the table definition. A check that cannot run is reported as skipped with the reason; an older server's dry-run session is cancelled. Returns a Diagnosis (one DiagnosisStep per check) and prints each when show_progress is on. IngestClient gains health(), limits() and storage_selftest(). Additive; 97 tests pass (+5); mypy has no new errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ropped 101b09c, merging eea_bliki into main, dropped part of 774691a: BeginResult.from_json raised NameError on every begin, and FolderIngest(upload_mode="proxy") reported a proxy upload while it uploaded to S3. Restore 774691a as written: BeginResult.upload_mode is None when the server sends none and is read from the top level or from inside s3, and _upload_one(target, session_id, proxy) sends a proxied file through DDS. PY-1's network-error fallback stays, as an addition: only when both FolderIngest and the server say "auto" (_falls_back_to_proxy). PY-1's own test now expects None for an absent hint, as 774691a does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
folderscan maps every sub-folder, table and view under a source to where it would land; foldercopy reproduces the subtree, compares COUNT(*) on both sides for each table and recreates each view from its own SQL with references into the subtree remapped (remap_view_sql); foldermove deletes the source only once the copy verified. New primitives under them: getentriesfrom, entryexists, getviewsql, createviewas, getfoldersfrom and CatalogRestClient.children / list_folders. Merged with main's catalog session work: deletetable is main's, so the copy of it written here is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark PY-1 … PY-7 done with as-built notes, and correct PY-1's premise: 774691a already honoured the proxy hint. A new §5 records main's broken ingest and its repair, where each task stands, and the follow-up tasks PY-8 … PY-13, with PY-11 (the new options in %ingest) left to Oskar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge Jan´s changes and FIX CI errors
PY-2 cancels the session when an upload fails, and retry() then raised "no session to retry". IngestSession.commit(retry=True) re-raises that, so %ingest showed it instead of the upload error. run() now records the upload failure, and retry() acts on it before anything else: it uploads again under a new session when the error can pass (a network error, a 5xx, or DDS saying retryable) and the cancel went through; otherwise it raises IngestStateError quoting the upload error. IngestSession is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Y-13)
IngestApiError.failed_stage: where a failed commit broke ("upload" or
"load"), taken from the session as the failure note already was. PY-5
named it as an attribute; it was only in the note text.
IngestOutcome.files_staged: how many uploaded files went through DDS, so
a network-error fallback is visible. IngestOutcome.upload_mode is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IngestClient.retry(session_id, definition=...) sends DDS's optional retry body (DI-12.6), and FolderIngest.retry(table_definition=...) passes it on: a commit that failed on its columns (schema_required, schema_mismatch) is fixed without uploading again, and a retry that does upload again sends it on begin. Without a definition, both calls are exactly what they were: no request body, and retry(session_id) on the client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /ingest/stage has its own cap (stage_max_bytes) and DDS checks it only when each file arrives, after the files ahead of it have uploaded. FolderIngest now reads GET /ingest/limits (DI-12.11) at most once per run, and only when a file may go through DDS: - upload_mode proxy: a file over the cap is refused before anything is uploaded (IngestApiError, retryable False; the session is cancelled). - the network-error fallback: a file over the cap is not sent through DDS; the network error gets a note saying why. A server without the endpoint (404), or a failed request, keeps today's behaviour. A direct upload makes no extra request. diagnose() gains a "File sizes" step: the files against max_bytes and stage_max_bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New recipes: CSV/JSON into an editable table, with and without a table definition; checking a transfer first (diagnose, dry_run); fixing the columns and resuming (retry with a definition). The "S3 is unreachable" recipe uses upload_mode="proxy" instead of driving /stage by hand. Part 1 gains three sections — table definitions and inferred columns; upload_mode, including the fallback and the /stage cap; dry_run and diagnose — and its reference tables catch up: FolderIngest's parameters and methods, retry's decisions after a failed upload, IngestOutcome, IngestClient's endpoints, the models, and what an IngestApiError carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each follow-up task gets its as-built note and commit, the earlier tasks' "open" notes point at what closed them, and §5.2 / §5.5 say what is left: the §5.1 repair reaching main, PY-11 (Oskar's decision), and the end-to-end runs after deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IngestSession.ingest — and so `%ingest ingest(...)` — forwards four more FolderIngest options, each of which fits on the one line that queues the ingest: upload_mode="proxy" (S3 unreachable), csv_options, json_options, and table_definition as a dict. Left out on purpose: infer_schema (turning inference off only makes DDS ask for the columns, which table_definition gives anyway) and cleanup_on_failure (its cancel after a failed commit would leave commit(retry=True) nothing to retry). The queue-then-commit design is unchanged; %ingest help shows the new parameters from the signature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IngestSession.diagnose() runs FolderIngest.diagnose() for every queued ingest and prints its lines — DDS reachable, the credential, the file sizes, DDS's storage, and a dry run — under a header naming the ingest. Nothing is uploaded and the queue is kept, so it fits between ingest(...) and commit: `%ingest diagnose`, one line, like `%ingest commit`. A missing folder or credential is printed as the finding instead of stopping the check. No separate dry_run verb: diagnose ends with one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scope agreed after talking to Oskar (keep IngestSession's concept; only what fits on one %ingest line), what was added and what was left out, and why. The main repair is shown as PR #97, left to the user and Oskar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IngestSession.ingest — and so `%ingest ingest(...)` — forwards
cleanup_on_failure too: for most one-off ingests a retry will not help,
and a failed commit should leave nothing behind.
Its cancel used to leave retry() with no session ("no session to
retry"), which is what commit(retry=True) would have shown. A commit
that cleanup_on_failure cancelled is now recorded like a failed upload
(PY-8), so retry() uploads again under a new session — the failure was
retryable, or the cleanup would not have run. If the cancel itself
failed, the session is still there and retry() resumes it on the server
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DDS accepts csv_options["encoding"] but passes no encoding on to Dremio, which reads CSV as UTF-8, so a Latin-1 or Windows-1252 file loads garbled. Worse, PY-6's inference read its text columns as binary (pyarrow's reading of text that is not UTF-8) and sent VARBINARY columns. The client now refuses, before begin, with a message naming the cause: - a csv_options encoding that is not UTF-8 (utf8, UTF-8, utf_8 are); - while inferring columns, a file that is not valid UTF-8 — a binary column from a CSV is that — named in the error. Only that failure path reads the files again, to name the file. The request body is unchanged: an encoding given is still sent as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
staging carries Oskar's %dds push work and his own repair of the 101b09c breakage, ea142ca, which eea_bliki had repaired separately (60cad54). The two behave the same; Oskar's reads "no hint" as "direct" and routes uploads through _effective_upload_mode. His version is taken: - models.py: BeginResult is exactly staging's. - folder.py: _effective_upload_mode and _upload_one(target, begin) as in ea142ca; our _falls_back_to_proxy and the extra arguments are gone. The later tasks are re-applied on top: the /stage cap check in the fallback branch (PY-10), files_staged now counting every file _stage sent (PY-13), the cancelled-run retry (PY-8) and the CSV encoding check are untouched. - PY-1's test expects "direct" again; the guide and the plan say so. 780 tests pass (staging alone: 683). Every line Oskar wrote is still here; the only ones changed are those PY-11 extended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings development up to date with staging (ingest PY-7 to PY-13, %ingest diagnose / options / cleanup_on_failure, foldertree copy and move, non-UTF-8 CSV check, the main ingest fix). catalog/client.py conflicted only because both sides added to Catalog: kept both — stream_rows / open_flight_channel (development) and getentriesfrom / entryexists / getviewsql / createviewas / getfoldersfrom and the foldertree imports (staging). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
developmentup to date withstaging(22 commits that are onstagingbut notdevelopment).What comes in
FolderIngest.diagnose(), retry after a cancelled upload or with a corrected table definition,failed_stage/files_staged, the DDS/stagesize cap, refusing non-UTF-8 CSV before upload.%ingest: the one-line ingest options,diagnose,cleanup_on_failureand retry after a cleanup.foldertree), withgetentriesfrom,entryexists,getviewsql,createviewasandgetfoldersfromonCatalog.Why a merge branch
A PR straight from
stagingconflicts insrc/eea_datalakehouse/catalog/client.py, and resolving it on GitHub would changestaging. This branch isdevelopmentwithstagingmerged in and the conflict resolved, sostagingstays as it is.The conflict was only because both sides added to
Catalog. Both are kept:stream_rows/open_flight_channelfromdevelopment(used by%gpkg);staging.The full test suite passes on the merged result: 922 passed, 1 skipped.
🤖 Generated with Claude Code