Summary
The per-score-set export loop has no error handling, and the archive is opened on its final filename. A failure mid-loop leaves a truncated zip indistinguishable by name from a complete dump.
Problem
src/mavedb/scripts/export_public_data.py:169 opens ZipFile(zip_file_name, "w") where zip_file_name is the final archive name. writestr flushes incrementally, so an exception leaves a valid-looking partial archive in the working directory.
- The loop at
:180 has no try/except, so any raising record aborts the run.
- Nothing in the filesystem distinguishes a completed dump from one that died partway through the score set list.
Proposed behavior
- Write to a
.partial filename and rename to the final name only after the loop completes and the archive closes cleanly.
- On a per-score-set failure: log the score set urn and exception, delete the partial archive, exit non-zero.
- Add
--continue-on-error, off by default: skip the failing score set, collect (urn, artifact, exception_class, message), write it as errors.json into the archive, and still exit non-zero.
Acceptance criteria
- A run that fails partway leaves no file matching the final archive name pattern.
- A failing run exits non-zero.
- With
--continue-on-error the archive contains errors.json listing every skipped score set, and the run still exits non-zero.
- A successful run produces exactly the artifacts it produces today.
Implementation notes
- Per-score-set artifacts are built at
:186–:276: scores CSV, annotations CSV, counts CSV, mapped-variants JSON, VA NDJSON. Failure granularity is the score set, not the artifact.
Summary
The per-score-set export loop has no error handling, and the archive is opened on its final filename. A failure mid-loop leaves a truncated zip indistinguishable by name from a complete dump.
Problem
src/mavedb/scripts/export_public_data.py:169opensZipFile(zip_file_name, "w")wherezip_file_nameis the final archive name.writestrflushes incrementally, so an exception leaves a valid-looking partial archive in the working directory.:180has notry/except, so any raising record aborts the run.Proposed behavior
.partialfilename and rename to the final name only after the loop completes and the archive closes cleanly.--continue-on-error, off by default: skip the failing score set, collect(urn, artifact, exception_class, message), write it aserrors.jsoninto the archive, and still exit non-zero.Acceptance criteria
--continue-on-errorthe archive containserrors.jsonlisting every skipped score set, and the run still exits non-zero.Implementation notes
:186–:276: scores CSV, annotations CSV, counts CSV, mapped-variants JSON, VA NDJSON. Failure granularity is the score set, not the artifact.