Conversation
…into lazy-extensions
…into lazy-extensions
|
|
||
| def finalize_buffers(self, squeeze_output=False): | ||
| import zarr | ||
|
|
There was a problem hiding this comment.
How do we ensure that the last buffer is written if the last chunk is not complete ?
There was a problem hiding this comment.
what do you mean? the pipipeline takes care of all chunks
| return self.arrays[0] | ||
|
|
||
|
|
||
| def _split_zarr_store_path(dataset_path): |
There was a problem hiding this comment.
I am not sure that th zarr suffix is a good aproach for this.
Looking for "extensions" would be safer for the analyzer no ?
There was a problem hiding this comment.
I don't understand what you mean here
There was a problem hiding this comment.
ok, @samuelgarcia mean to just get rid of this function and created the dataset directly with the full zarr path
|
OK for me. |
Co-authored-by: Garcia Samuel <sam.garcia.die@gmail.com>
for more information, see https://pre-commit.ci
…nto gather-to-zarr
| root_cache = {} | ||
| for i, dataset in enumerate(folder): | ||
| if isinstance(dataset, (str, Path)): | ||
| store_path, internal_path = _split_zarr_store_path(dataset) |
There was a problem hiding this comment.
we could directly open the final folder and get rid of the _split_zarr_store_path
3 participants
This PR continues in the direction of support for very long chronic data. This feature prevents loading all spike vector based extension data to memory prior to saving!
GatherToZarrclass to save node pipeline data as it's being computed to zarr datasets.GatherToNpyto be able to specify multiple folders for different filesThese two changes allow to extend the
SortingAnalyzercompute functionality to directly save extension data from node pipelines computations (usually spike-vector based extensions) directly to their final location (numpy files or zarr datasets).This is only enabled when the
SortingAnalyzerhas a folder backend (binary folder or zarr).