Repository navigation
Allow a separate store root for each modality - #26
Open
Raminyazdani wants to merge 2 commits into
Open
Raminyazdani wants to merge 2 commits into
Raminyazdani wants to merge 2 commits into
Conversation
Read optional sources.<modality>.store_root through the existing path resolver. Forward it to dataset, statistics and baseline metadata readers. Keep the shared-root fallback and directory/ZIP selection unchanged. Include explicit modality roots in preload cache keys so changing roots cannot return data cached from another location. Keep existing default cache keys and scientific settings unchanged. Add focused path, config, cache and reader-forwarding regressions.
Document the optional source setting, its shared-root fallback and Hydra syntax. Explain when explicit modality roots separate preload caches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #24.
This adds an optional
sources.<modality>.store_root, so a DeepLST store can be kept in a different directory from the shared stores. For example:Here SEVIRI uses the separate directory, while the other modalities keep the shared root. Leaving
store_rootunset or null keeps the previous behavior. Relative paths and directory/ZIP selection work as before.The selected root is used by the dataset preload, statistics reader and baseline metadata reader. An explicit root is also included in the preload cache key, so data from a different root is not reused by mistake. Runs without this setting keep their existing cache keys.
Local checks passed: 242 tests passed and 2 were skipped, and pre-commit checks passed. The added checks cover path selection, config forwarding and switching roots without reusing the old cache. I also checked that a small cache made before the change can still be loaded. The tests use small temporary files and replace the real readers; I haven't tested this with the real cluster stores yet.