feat: Variant Level CSV Download - #804
Merged
bencap merged 9 commits intoAug 12, 2026
Merged
Conversation
bencap
force-pushed
the
feature/bencap/variant-level-csv-download
branch
from
August 6, 2026 23:09
e39af46 to
c0f0bb9
Compare
bencap
changed the base branch from
release-2026.2.8
to
bugfix/bencap/calibration-permissions
August 6, 2026 23:10
bencap
marked this pull request as ready for review
August 6, 2026 23:12
Coverage Report for CI Build 31619922039Warning No base build found for commit Coverage: 89.285%Details
Uncovered Changes
Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
Move all CSV-generation code for score set variant downloads out of score_sets.py into a new dedicated mavedb/lib/score_set_csv.py module, splitting it into pure functions (column planning, header assembly, row formatting) and DB-bound fetching to keep the logic testable in isolation. - Add parse_clinvar_namespace() to clinvar/utils.py, replacing the inline regex parsing that lived in score_sets.py - Add is_csv_output_null() to mave/utils.py to centralize null-value detection used when formatting CSV output - Move variant_to_csv_row, variants_to_csv_rows, get_score_set_variants_as_csv, and drop_na_columns_from_csv_file_rows into score_set_csv.py, and update routers/score_sets.py and scripts/export_public_data.py to import from the new location - Move corresponding tests into tests/lib/test_score_set_csv.py and add new unit tests for parse_clinvar_namespace and is_csv_output_null
Split the monolithic score_set_csv.py into mavedb/lib/csv/, a package organized by responsibility (namespaces, columns, entries, fetch, annotations, specs, deprecated_params) instead of one file mixing column planning, row formatting, and DB fetching for a single record type. score_set.py and variant.py sit on top as the two record-specific entry points, sharing the namespace/column machinery beneath them. This is the foundation for variant-level CSV export and namespace discovery (following commits): the old module had no way to answer "what does this record have data for" or to serve a single variant, so extending it in place would have meant bolting both concepts onto code that already conflated planning and fetching. - Introduce namespace discovery as first-class: CsvNamespace enum, CsvNamespaceGroup, and available_score_set_csv_namespaces / available_variant_csv_namespaces, replacing the hand-maintained CLINVAR_NS_PATTERN validity check in the router - Add lib/annotation/flatten.py to project a variant's VA-Spec interpretation onto scalar CSV columns, sharing acmg.py's new acmg_evidence_outcome_code() with the VA-Spec evidence-line builder so the two representations can't drift - Thread an optional containing_classification_ids set through functional_classification_of_variant and pathogenicity_classification_of_variant, resolving the long-standing TODOs about O(1) membership checks when classifying many variants - Move parse_clinvar_namespace (clinvar/utils.py) and CSV-null-output detection (mave/utils.py) into the csv package, since both were export-side concerns living outside it - Add drop_unused_hgvs_columns, scores_custom, and score_set namespaces; keep drop_na_columns, include_post_mapped_hgvs, and include_custom_columns working as deprecated aliases via resolve_deprecated_csv_params - Add title to ShorterScoreSet so namespace-discovery responses can label a calibration's owning score set without a second lookup BREAKING CHANGE: score_set_csv.py is removed. Callers importing get_score_set_variants_as_csv or variants_to_csv_rows from it must import from mavedb.lib.csv.score_set / mavedb.lib.csv.columns instead.
…points
Add GET /variants/{urn}/csv and GET /variants/{urn}/csv-namespaces,
mirroring the score-set CSV endpoints but scoped to a single variant.
The variant endpoint widens over the variant's equivalent measurements
(as the annotation endpoints already do), so a calibration belonging
to another score set that also measured this allele is included too,
and emits one row per current measurement with the requested variant
first.
Add GET /score-sets/{urn}/csv-namespaces alongside it, and switch
GET /score-sets/{urn}/variants/data to build its namespace list and
validation from discovery instead of a hand-maintained
_VALID_STATIC_NAMESPACES set and the ClinVar regex check.
- Accept drop_unused_hgvs_columns on the score-set CSV endpoints,
keeping drop_na_columns, include_post_mapped_hgvs, and
include_custom_columns working as deprecated query params via
resolve_deprecated_csv_params
- Ask calibration READ permission separately from score-set READ on
every CSV endpoint, since a private calibration's interpretation is
not implied by being able to read the measurement it applies to
- /score-sets/{urn}/scores now includes both score namespaces
("scores" plus "scores_custom"), matching its historical behavior of
returning every score column the investigator uploaded
- Make _stream_generated_annotations resilient to a single variant's
annotation failing mid-stream: log and emit it as unannotated rather
than raising, since the response has already started and an
unhandled exception would silently truncate the file
Replace the hand-maintained list of ClinVar release namespaces in export_public_data.py with annotation_export_namespaces(), which asks available_score_set_csv_namespaces() what the score set actually has and subtracts the score/count/identity groups already covered by their own files in the dump. The previous list named releases one by one, so it silently emitted all-NA columns for a release never ingested and needed a code change for every new one. Discovery-derived namespaces also add score calibration interpretations to the annotations CSV for the first time, since those are now part of what discovery reports. Update scripts/resources/README.md to document the calibration column group and clarify that the ClinVar and calibration groups vary by score set and should be read from the header rather than assumed fixed.
…ll tests The namespace refactor moved risk into the compatibility layer while the test mass stayed where it had always been, on the shared NA coercion. - Add test_deprecated_params.py. This layer is all that keeps a pre-namespace client working: FastAPI ignores unknown query parameters, so an unmapped old spelling would silently return different columns rather than erroring, and Galaxy calls these endpoints. It is pure and total, so it is specified here rather than reached incidentally through the routers — unit coverage goes from 47% to 100%. Pins the precedence rules (the current name wins, the boolean flags append rather than replace), namespace-append idempotency, the published token spellings, and the RFC 8594 response headers. - Collapse TestVariantToCsvRowNullHandling from 15 tests to 5. Eight of them asserted the value -> NA rule through different namespaces, but every namespace reaches it through the single _value_or_na call in variant_to_csv_row, and test_is_output_null already specifies that rule over 18 values — so those tests could not fail unless it failed too. What remains covers the layer above: building the per-row source a resolver is handed, where absent data has several distinct shapes. Score and count columns are one mechanism selected by RowSource, so they are parametrized together instead of hand-duplicated. Deliberately not added: per-namespace null tests, and registry invariants. test_specs.py already asserts every namespace has a spec, every declared column resolves, every resolver tolerates a None source, and that a namespace reading through a relationship declares the fetch it needs.
bencap
force-pushed
the
feature/bencap/variant-level-csv-download
branch
from
August 10, 2026 23:52
d122b22 to
a895695
Compare
get_digest_from_post_mapped is renamed to get_id_from_post_mapped and
now returns the VRS object's `id` field verbatim (e.g.
`ga4gh:VA.{digest}`) instead of the bare `digest`. Only `id` is
indexed by ix_mapped_variants_post_mapped_id and matched by
GET /mapped-variants/vrs/{identifier}, so exporting the digest produced identifiers
that could not be resolved back against MaveDB.
- rename the CSV column mavedb.post_mapped_vrs_digest to
mavedb.post_mapped_vrs_id and update README.md's column reference
- never synthesize an id from a digest-only object, and leave nested
VRS 1.x variation.id unread, matching the lookup endpoint's reach
- add coverage for the digest/id divergence, the digest-only case, and
the VRS 1.x nesting case
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
GET /variants/{urn}/csvand namespace-discovery endpoints for both score sets and variants, on top of a restructured CSV export package. Serves the UI-side column picker and variant download in VariantEffect/mavedb-ui#706.New endpoints
GET /variants/{urn}/csvmirrors the score-set CSV endpoints scoped to a single variant. It widens over the variant's equivalent measurements, as the annotation endpoints already do, so a calibration belonging to another score set that also measured this allele is included; it emits one row per current measurement with the requested variant first.GET /variants/{urn}/csv-namespacesandGET /score-sets/{urn}/csv-namespacesreport what a record actually has data for, grouped and labeled for direct display.GET /score-sets/{urn}/variants/datanow builds its namespace list and validation from discovery instead of the hand-maintained_VALID_STATIC_NAMESPACESset and the ClinVar regex check.GET /score-sets/{urn}/scoresincludes both score namespaces (scoresandscores_custom), matching its historical behavior of returning every score column the investigator uploaded.CSV package restructure
score_set_csv.pyis replaced bymavedb/lib/csv/, organized by responsibility (namespaces,columns,entries,fetch,annotations,specs,deprecated_params), withscore_set.pyandvariant.pyas the two record-specific entry points sharing the machinery beneath them. The old module mixed column planning, row formatting, and DB fetching for a single record type, and had no way to answer "what does this record have data for" or to serve one variant.CsvNamespaceenum,CsvNamespaceGroup, andavailable_score_set_csv_namespaces/available_variant_csv_namespaces.lib/annotation/flatten.pyprojects a variant's VA-Spec interpretation onto scalar CSV columns. It sharesacmg.py's newacmg_evidence_outcome_code()with the VA-Spec evidence-line builder so the flat and nested representations cannot drift.functional_classification_of_variantandpathogenicity_classification_of_varianttake an optionalcontaining_classification_idsset, resolving the standing TODOs about O(1) membership checks when classifying many variants.parse_clinvar_namespaceand CSV-null detection move into the CSV package fromclinvar/utils.pyandmave/utils.py; both were export-side concerns living outside it.titleis added toShorterScoreSetso discovery responses can label a calibration's owning score set without a second lookup.Breaking change:
score_set_csv.pyis removed. Callers importingget_score_set_variants_as_csvorvariants_to_csv_rowsmust import frommavedb.lib.csv.score_set/mavedb.lib.csv.columns.Query parameters
drop_unused_hgvs_columns,scores_custom, andscore_setnamespaces.drop_na_columns,include_post_mapped_hgvs, andinclude_custom_columnskeep working as deprecated aliases throughresolve_deprecated_csv_params.Permissions
ScoreCalibrationViewer.Public data export
export_public_data.pyderives its namespaces fromannotation_export_namespaces(), which asks discovery what the score set has and subtracts the score, count, and identity groups already covered by their own files in the dump. The previous hand-maintained list named ClinVar releases one by one, so it emitted all-NA columns for any release never ingested and needed a code change for every new one.scripts/resources/README.mddocuments the calibration column group and notes that the ClinVar and calibration groups vary by score set, so consumers should read the header rather than assume a fixed set.Streaming robustness
_stream_generated_annotationslogs a single variant's annotation failure and emits it as unannotated rather than raising. The response has already started by that point, so an unhandled exception silently truncated the file.Tests
New unit tests under
tests/lib/csv/(columns, entries, namespaces, specs, variant, deprecated params) andtests/lib/annotation/test_flatten.py, plus router coverage intests/routers/test_variant.pyandtests/routers/test_score_set.py.