Repository navigation
Set the chunk size of each preprocessing step with per_step_job_kwargs - #4930
Open
h-mayorquin wants to merge 2 commits into
Open
h-mayorquin wants to merge 2 commits into
h-mayorquin wants to merge 2 commits into
Conversation
Each step accepts either chunk_size or chunk_duration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@alejoe91 mentioned in #4833 that distributing the work across processing steps would be very hard to implement. This PR is a prototype of the simplest part of #4905 to show that this part is simple: each step of a chain gets its own chunk size inside the worker that owns the job. It adds a
per_step_job_kwargsjob keyword that maps the name of a preprocessing function to its ownchunk_sizeorchunk_duration:The top level job kwargs keep their meaning and set what each worker gets.
per_step_job_kwargsonly splits that job inside the worker: a step asked for more than its chunk size computes the request in equal pieces. It also works fordetect_peaksandSortingAnalyzer.computewithout changing them as it is a job keyword. It can't be set globally as it describes the steps of one chain.The main limitation is that a step can't use a larger chunk size than the steps after it in the chain. Each step only computes what the next one asks for, so a larger setting has no effect, and the chunk size of the job is the largest of all.
I think this covers most cases and is the smallest change that:
recording.save, would then fix the problem in Addnum_chunk_per_batchto job kwargs and use it for zarr v3 shard writes #4833.I have an idea to make the sizes of the steps independent of each other (inside the samples a worker gets), but I think this is enough progress for now and it would make the proposal harder to discuss. If we decide we want to go there, there is a path forward.
Also, this keeps
n_jobsshared across the same pipeline.