Skip to content

fix(export): public data dump fails fast and never leaves a partial archive #837

Description

@bencap

Summary

The per-score-set export loop has no error handling, and the archive is opened on its final filename. A failure mid-loop leaves a truncated zip indistinguishable by name from a complete dump.

Problem

  • src/mavedb/scripts/export_public_data.py:169 opens ZipFile(zip_file_name, "w") where zip_file_name is the final archive name. writestr flushes incrementally, so an exception leaves a valid-looking partial archive in the working directory.
  • The loop at :180 has no try/except, so any raising record aborts the run.
  • Nothing in the filesystem distinguishes a completed dump from one that died partway through the score set list.

Proposed behavior

  • Write to a .partial filename and rename to the final name only after the loop completes and the archive closes cleanly.
  • On a per-score-set failure: log the score set urn and exception, delete the partial archive, exit non-zero.
  • Add --continue-on-error, off by default: skip the failing score set, collect (urn, artifact, exception_class, message), write it as errors.json into the archive, and still exit non-zero.

Acceptance criteria

  • A run that fails partway leaves no file matching the final archive name pattern.
  • A failing run exits non-zero.
  • With --continue-on-error the archive contains errors.json listing every skipped score set, and the run still exits non-zero.
  • A successful run produces exactly the artifacts it produces today.

Implementation notes

  • Per-score-set artifacts are built at :186:276: scores CSV, annotations CSV, counts CSV, mapped-variants JSON, VA NDJSON. Failure granularity is the score set, not the artifact.

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions