Populate per-asset dataStandard for BIDS, HED, NWB, and extensions by yarikoptic · Pull Request #1811 · dandi/dandi-cli

yarikoptic · 2026-02-24T22:12:34Z

Summary

Populate per-asset dataStandard field (new in dandischema) for all standards:
- BIDS: from dataset_description.json with BIDSVersion
- HED: from HEDVersion in dataset_description.json, including library schemas (e.g. "sc:1.0.0") as extensions
- NWB: from NWB file metadata, with ndx-* extensions extracted from h5file["specifications"] group
- OME/NGFF: for .ome.zarr assets in BIDS datasets
Centralize backward-compat guard as _SCHEMA_BAREASSET_HAS_DATASTANDARD in bases.py — single bool gates all new dandischema features (dataStandard, version, extensions), with RuntimeError if dandischema >= 0.12.2 unexpectedly lacks the field
Add get_nwb_extensions() to pynwb_utils.py — reads HDF5 specifications group, filters core namespaces, returns {name: latest_version} for each extension
All new code is no-op against released dandischema (< 0.12.2)

Depends on: dandi/dandischema#XXXX

Test plan

TestBIDSDatasetDescriptionDataStandard — 5 tests covering BIDS always set, HED detected/absent, HED list form, HED library schemas as extensions
test_get_nwb_extensions — verifies ndx-* extraction with version sorting, core namespace filtering
test_get_nwb_extensions_no_specs — empty dict when no specifications group
460 tests pass with dev dandischema, 426 pass with released dandischema 0.12.1 (new feature tests correctly skip)
TODO1: add matrix run to test against dandi-schema#371, and then remove that or do it properly as a dedicated run based on description here annotation

PR in dandi-schema:

Add per-asset dataStandard, HED standard, and extensions to StandardsType dandi-schema#371

🤖 Generated with Claude Code

dandi/files/bids.py

codecov · 2026-02-24T23:00:05Z

Codecov Report

❌ Patch coverage is 47.53086% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.76%. Comparing base (0fe497d) to head (279c297).

Files with missing lines	Patch %	Lines
dandi/files/bids.py	20.75%	42 Missing ⚠️
dandi/files/bases.py	4.54%	21 Missing ⚠️
dandi/tests/test_files.py	62.22%	17 Missing ⚠️
dandi/pynwb_utils.py	75.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1811      +/-   ##
==========================================
- Coverage   75.11%   74.76%   -0.36%     
==========================================
  Files          84       84              
  Lines       11925    12083     +158     
==========================================
+ Hits         8958     9034      +76     
- Misses       2967     3049      +82

Flag	Coverage Δ
unittests	`74.76% <47.53%> (-0.36%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- BIDSDatasetDescriptionAsset.get_metadata(): always sets BIDS standard (with BIDSVersion), adds HED when HEDVersion present in JSON, warns on read failure - NWBAsset.get_metadata(): sets NWB standard - ZarrBIDSAsset.get_metadata(): sets OME/NGFF for .ome.zarr assets - Guard for older dandischema without dataStandard on BareAsset; RuntimeError if dandischema >= 0.12.2 lacks it - Register ai_generated pytest marker in tox.ini Requires dandischema with per-asset dataStandard support (0.12.2+). Works silently with older dandischema. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add get_nwb_extensions() to pynwb_utils that reads h5file["specifications"] to discover ndx-* namespaces and their versions (filtering out core/hdmf) - NWBAsset.get_metadata() populates StandardsType.extensions with ndx-* extensions found in the NWB file - BIDSDatasetDescriptionAsset.get_metadata() extracts HED library schemas from list-valued HEDVersion (e.g. ["8.2.0", "sc:1.0.0"]) into extensions - Add tests for both NWB extension extraction and HED library schema parsing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…S_DATASTANDARD Single bool in bases.py guards all dataStandard/version/extensions features that ship together in dandischema >= 0.12.2, replacing scattered "field in Model.model_fields" checks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move json, h5py, get_nwb_extensions, and _SCHEMA_BAREASSET_HAS_DATASTANDARD imports to module top level in tests. Keep pynwb_utils import deferred in bases.py (heavy transitive deps: h5py/pynwb/hdmf/numpy) per existing convention. Add import guidance to CLAUDE.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

hed_standard does not exist in dandischema < 0.12.2, so gate the import on _SCHEMA_BAREASSET_HAS_DATASTANDARD (all new symbols ship together). HED detection is skipped when unavailable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bendichter · 2026-03-03T19:58:58Z

dandi/files/bids.py

 BIDS_DATASET_ERRORS = ("BIDS.MANDATORY_FILE_MISSING_PLACEHOLDER",)


+def _add_standard(


line 542 of bases appears to have similar logic

Move _SCHEMA_BAREASSET_HAS_DATASTANDARD guard block after all imports to fix E402. Use getattr/setattr for dataStandard access and type: ignore[call-arg] for StandardsType(version=...) to satisfy mypy against released dandischema that lacks these fields. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

dandi/files/bases.py

dandi/files/bids.py

+from .bases import (
+    _SCHEMA_BAREASSET_HAS_DATASTANDARD,
+    GenericAsset,
+    LocalFileAsset,
+    NWBAsset,
+)


In general, the clean way to break an import cycle is to avoid having two modules that directly depend on each other’s runtime definitions. For class hierarchies, this usually means either (1) moving shared abstractions into a third, lower-level module, or (2) converting some imports to type-only imports using typing.TYPE_CHECKING, so the runtime import no longer occurs.

Within dandi/files/bids.py, we can break the cycle without changing behavior by changing how we import from .bases. The key is to avoid importing GenericAsset (a base abstraction that likely needs to stay independent), while still ensuring that any references to it remain valid for static typing. Since Python 3.7+ with from __future__ import annotations supports postponed evaluation of annotations, we can safely import GenericAsset only under TYPE_CHECKING. At runtime, code in bids.py does not need the GenericAsset name unless it is used directly (e.g., for isinstance checks); if it is only used in type hints, moving it under TYPE_CHECKING removes the cycle.

Concretely, in dandi/files/bids.py we will:

Keep importing LocalFileAsset and NWBAsset from .bases at runtime.

Stop importing GenericAsset at runtime.

Add a TYPE_CHECKING guarded import for GenericAsset so type checkers still see it.

This involves:

Adding from typing import TYPE_CHECKING to the imports.

Modifying the .bases import block so that GenericAsset is no longer in the runtime import.

Adding a small if TYPE_CHECKING: block that imports GenericAsset from .bases for typing only.

No functionality changes for executed code, but the runtime import graph changes so that bids.py no longer pulls in GenericAsset during module initialization, thus breaking or at least relaxing the cycle.

bendichter · 2026-03-03T20:09:54Z

@yarikoptic lgtm! Let me just check on some of my own data

Use the existing get_nwb_version() to populate the version field on the NWB StandardsType entry so per-asset metadata reports the NWB spec version (e.g. 2.9.0) alongside extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bendichter · 2026-03-03T20:23:11Z

Added NWB spec version extraction in 279c297.

get_nwb_version() already existed in pynwb_utils.py but wasn't being used when constructing the StandardsType. Now the version is populated from the HDF5 nwb_version attribute.

Tested against dandiset 001636 (346 NWB files, macaque electrophysiology):

Before:

Neurodata Without Borders (NWB) (version=None)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

After:

Neurodata Without Borders (NWB) (version=2.9.0)
  ext: ndx-events v0.4.0
  ext: ndx-hed v0.2.0

bendichter

ok, I made a minor correction. LGTM now!

yarikoptic mentioned this pull request Feb 24, 2026

Add per-asset dataStandard, HED standard, and extensions to StandardsType dandi/dandi-schema#371

Open

4 tasks

github-advanced-security bot found potential problems Feb 24, 2026

View reviewed changes

dandi/files/bids.py Fixed Show fixed Hide fixed

yarikoptic added the minor Increment the minor version when merged label Mar 3, 2026

yarikoptic and others added 6 commits March 3, 2026 14:52

chore: git ignore uv.lock

1cd1d6a

bendichter reviewed Mar 3, 2026

View reviewed changes

yarikoptic force-pushed the hed branch from d000be8 to 4f7d154 Compare March 3, 2026 20:07

bendichter reviewed Mar 3, 2026

View reviewed changes

dandi/files/bases.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Mar 3, 2026

View reviewed changes

Extract NWB spec version into dataStandard

279c297

Use the existing get_nwb_version() to populate the version field on the NWB StandardsType entry so per-asset metadata reports the NWB spec version (e.g. 2.9.0) alongside extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bendichter self-requested a review March 3, 2026 20:23

bendichter approved these changes Mar 3, 2026

View reviewed changes

@@ -7,6 +7,7 @@
             from pathlib import Path
             from threading import Lock
             import weakref
+            from typing import TYPE_CHECKING
             from dandischema.models import (
                 BareAsset,
@@ -20,11 +21,13 @@
             from .bases import (
                 _SCHEMA_BAREASSET_HAS_DATASTANDARD,
-                GenericAsset,
                 LocalFileAsset,
                 NWBAsset,
             )
+            if TYPE_CHECKING:
+                from .bases import GenericAsset
             if _SCHEMA_BAREASSET_HAS_DATASTANDARD:
                 from dandischema.models import hed_standard  # type: ignore[attr-defined]
             else:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions#1811

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions#1811
yarikoptic wants to merge 8 commits intomasterfrom
hed

yarikoptic commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

codecov bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

bendichter Mar 3, 2026

Uh oh!

Uh oh!

Check notice

Copilot Autofix

bendichter commented Mar 3, 2026

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

bendichter left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		BIDS_DATASET_ERRORS = ("BIDS.MANDATORY_FILE_MISSING_PLACEHOLDER",)


		def _add_standard(

Conversation

yarikoptic commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bendichter Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Check notice

Uh oh!

Copilot Autofix

bendichter commented Mar 3, 2026

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

bendichter left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yarikoptic commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading