Skip to content

Allow a separate store root for each modality - #26

Open
Raminyazdani wants to merge 2 commits into
developfrom
rz/modality-store-roots
Open

Raminyazdani wants to merge 2 commits into
developfrom
rz/modality-store-roots

Conversation

@Raminyazdani

@Raminyazdani Raminyazdani commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Related to #24.

This adds an optional sources.<modality>.store_root, so a DeepLST store can be kept in a different directory from the shared stores. For example:

uv run python deep_lst/train.py +sources.seviri.store_root=/data/deep_lst

Here SEVIRI uses the separate directory, while the other modalities keep the shared root. Leaving store_root unset or null keeps the previous behavior. Relative paths and directory/ZIP selection work as before.

The selected root is used by the dataset preload, statistics reader and baseline metadata reader. An explicit root is also included in the preload cache key, so data from a different root is not reused by mistake. Runs without this setting keep their existing cache keys.

Local checks passed: 242 tests passed and 2 were skipped, and pre-commit checks passed. The added checks cover path selection, config forwarding and switching roots without reusing the old cache. I also checked that a small cache made before the change can still be loaded. The tests use small temporary files and replace the real readers; I haven't tested this with the real cluster stores yet.

Read optional sources.<modality>.store_root through the existing path
resolver. Forward it to dataset, statistics and baseline metadata readers.
Keep the shared-root fallback and directory/ZIP selection unchanged.

Include explicit modality roots in preload cache keys so changing roots
cannot return data cached from another location. Keep existing default
cache keys and scientific settings unchanged.

Add focused path, config, cache and reader-forwarding regressions.
Document the optional source setting, its shared-root fallback and Hydra
syntax. Explain when explicit modality roots separate preload caches.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 23:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants