Repository navigation
Re-reference blocks of samples at a time in common_reference to reduce peak memory - #4908
Open
h-mayorquin wants to merge 1 commit into
Open
h-mayorquin wants to merge 1 commit into
h-mayorquin wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR continues #4840 (Filter subsets of channels at a time to reduce peak memory), #4903 (Filter subsets of channels at a time in
gaussian_filterto reduce peak memory) and #4904 (Shift subsets of channels at a time inphase_shiftto reduce peak memory), now forcommon_reference. Once #4904 is in, it is the step that sets the peak memory of the usual Neuropixels chain on int16 traces.Subtracting the float64 median from an int16 chunk makes a float64 copy of the whole chunk. Doing it in blocks of samples takes the peak from 5 to 1.2 times the chunk (6 to 2.6 times for the chain) and it is a bit faster. The output is identical to main except for the local average, where a few values can move by 1 depending on where the chunk is cut, as already happens on main. The local median is about 25% slower.
This touches the same lines as #4854 (Faster exact median for
common_reference) by @galenlynch and I am fine with fixing the conflicts if that one goes first.Neuropixels 2.0 recording, 384 channels at 30 kHz, int16. Peak memory from
tracemallocas a percentage of the chunk (including the output) and time as the best of 3 runs. The chain isphase_shiftfrom #4904,bandpass_filterandcommon_referencewith the global median.