Skip to content

Testing: improve generic test to also test concrete expected values #1945

Open

Description

@h-mayorquin

Right now the tests that every reader gets are the compliance checks in BaseTestRawIO.test_read_all. These are consistency checks (shapes, dtypes, different ways of reading the same data agreeing with each other) and as they are the same for every format they can't express what values a given file should contain.

However, these recent fixes were all for readers that returned the right shape with the wrong values:

What these PRs share is that each one adds a separate test against specific values which means we load the data again. I think we should include this mechanism within our testing framework. Each test class would list the values it expects from the files it already tests, and test_read_all would check them on the reader it has already parsed. This way we reuse the files we already download, we don't read them twice and we have one place to test values. A test class that lists nothing doesn't change:

class BaseTestRawIO:
    ...
    expected_values = {}

    def test_read_all(self):
        for entity_name in self.entities_to_test:
            ...  # build the reader, parse the header and run the compliance checks as today
            expected = self.expected_values.get(entity_name, {})

            event_names = list(reader.header["event_channels"]["name"])
            for event_name, timestamps in expected.get("event_timestamps", {}).items():
                actual, _, _ = reader.get_event_timestamps(event_channel_index=event_names.index(event_name))
                np.testing.assert_array_equal(actual, timestamps)

            if "scaled_samples" in expected:
                ...  # read the first samples, rescale and compare


class TestBrainVisionRawIO(BaseTestRawIO, unittest.TestCase):
    ...
    expected_values = {
        "brainvision/File_brainvision_3_int16.vhdr": {
            # SegmentDataPoints=294 in the .vhdr, a "New Segment" marker on the first sample of each segment
            "event_timestamps": {"New Segment": np.arange(34) * 294},
        },
    }

Every expected value has to come from outside neo (the specification, the vendor's reader or the file's own header) with a comment naming the source. Values copied from neo's current output would only freeze its current bugs.

I am not married to the specifics but I think we need something like this. I do something similar in neuroconv, where the EDF test class states the session start time and the subject sex in the header of edf+C.edf and the shared metadata test checks them:

https://github.com/catalystneuro/neuroconv/blob/0b1d6d4bd38f6fba982d89748bb5535731a3f921/tests/test_on_data/ecephys/test_recording_interfaces.py#L354-L373

Activity

  1. changed the title [-]Check values against an outside source in the rawio tests[/-] [+]Testing: improve generic test to also test concrete expected values[/+] on Nov 4, 2026
  2. h-mayorquin commented on Nov 6, 2026

    @h-mayorquin
    ContributorAuthor

    Another case, from the other side. SpikeGLXRawIO.event_count has returned the count of channel 0 for every digital channel since 2024. The shared read_events check catches it, but spikeglx/DigitalChannelTest_g0, the only test file with digital transitions, was read in its own test and never went through test_read_all. With the expected values attached to the files in entities_to_test, that file would have run the shared checks as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions