Skip to content

Allow training data and cache paths to be set in config - #18

Merged
MWesselkamp merged 4 commits into
developfrom
rz/config-paths
Nov 12, 2026
Merged

MWesselkamp merged 4 commits into
developfrom
rz/config-paths

Conversation

@Raminyazdani

@Raminyazdani Raminyazdani commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Related to #17.

This makes the training data and cache paths configurable without editing Python files. The defaults are defined in ExperimentPaths in deep_lst/utils/io.py, which is used by train.py and the PATHS instance in deep_lst/dataset/dataset.py.

The two settings are paths.eozarr_root and paths.preload_cache_root. For example:

uv run python deep_lst/train.py paths.eozarr_root=/data/eozarr paths.preload_cache_root=/scratch/deep_lst-cache

An explicit cache setting takes priority. Without it, the cache uses DEEP_LST_CACHE_ROOT when it is set and nonempty, otherwise /Net/Groups/BGI/tscratch/$USER/tmp. If USER is missing or empty, it uses the account name. The data-store default is unchanged. Relative paths still use the working directory, and the settings are saved in the run's config.yml.

The Hydra overrides cover training, including validation and test splits. Other commands use the shared ExperimentPaths defaults, including the updated cache default. Cache keys are unchanged, so different data contents should use a separate cache directory. The paths=bgc|local profiles are left for a separate PR, as discussed in #17.

The local test suite passed: 222 tests passed and 2 were skipped. The added tests cover the environment default, the current-user fallback, and explicit override priority. The existing config and rollout-loss golden fixtures are unchanged. Pre-commit checks also passed. I haven't run training with the real cluster data yet.

Expose optional data/cache root overrides in the existing Hydra training
config and pass one ExperimentPaths instance through the DataModule and
multimodal dataset. Keep null defaults, working-directory-relative paths,
cache fingerprints and scientific settings unchanged.

Add regression checks for independent overrides, saved run configuration,
reader/statistics path wiring, and cold/warm cache sample equality. Reuse
existing config and rollout-loss golden fixtures without changing them.

Validation on the final working tree: 217 passed, 2 skipped; all six
pre-commit hooks passed. All 137 passing pre-change baseline tests still
pass. Real institutional-store training equivalence remains untested.
Document CLI/config examples, preserved defaults and relative-path
semantics, plus the saved run configuration. Replace source-edit advice
for training with Hydra overrides.

State the training-only scope and unchanged cache-key behavior: different
data contents should use separate cache roots. Other entry points keep
their existing defaults.

Validation: documentation was included in the passing final pre-commit
run. No production or test content changed after the 217-pass suite.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 16:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@MWesselkamp MWesselkamp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me! just a small comment on the description, and on the default paths.

Comment thread docs/CONFIGURATION.md Outdated
### Training data and cache paths

`paths.eozarr_root` and `paths.preload_cache_root` override the data-store and preload-cache
roots used by `deep_lst/train.py`, for all three training/validation/test splits. Both default

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says the paths are in train.py but they are the ExperimentPaths defaults in deep_lst/utils/io.py (used by train.py and by the PATHS singleton in dataset.py). Please correct docs/CONFIGURATION.md and the PR text so the next reader finds them.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I corrected the docs and the PR description. They now point to ExperimentPaths in deep_lst/utils/io.py and also mention the PATHS instance in dataset.py.

Comment thread docs/RUNBOOK.md Outdated

1. `ExperimentPaths.preload_cache_root` in `deep_lst/utils/io.py` points at the maintainer's
tscratch directory, which you cannot write to. Point it at your own tscratch directory.
1. `ExperimentPaths.preload_cache_root` in `deep_lst/utils/io.py` defaults to the maintainer's

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

preload_cache_root must not default to a path containing a username (it is unwritable for anyone else, see RUNBOOK §3). Derive the default from $DEEP_LST_CACHE_ROOT or $USER; paths.preload_cache_root keeps overriding it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I changed this to use DEEP_LST_CACHE_ROOT, or the current user's scratch directory when it is not set. The config override still takes priority. I added tests for this too, including when USER is missing.

Use DEEP_LST_CACHE_ROOT when set, otherwise choose the current user's
cluster scratch path. Keep explicit configuration overrides first.

Test environment defaults and override priority, including missing USER.
Point readers to ExperimentPaths in deep_lst/utils/io.py and the PATHS
instance in dataset.py. Explain the cache fallback and override priority.
Comment thread docs/CONFIGURATION.md

This first slice applies to training; standalone inference, statistics and fine-tuning entry
points still use their existing paths. Model, loss and data-processing settings are unchanged.
These Hydra overrides apply to training. Standalone inference, statistics and fine-tuning use

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for this self-contained addition. But this override should eventually be applied to all modules, I'll add it to the issue.
React

@MWesselkamp MWesselkamp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MWesselkamp
MWesselkamp merged commit c081846 into develop Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants