Repository navigation
Allow training data and cache paths to be set in config - #18
Conversation
Expose optional data/cache root overrides in the existing Hydra training config and pass one ExperimentPaths instance through the DataModule and multimodal dataset. Keep null defaults, working-directory-relative paths, cache fingerprints and scientific settings unchanged. Add regression checks for independent overrides, saved run configuration, reader/statistics path wiring, and cold/warm cache sample equality. Reuse existing config and rollout-loss golden fixtures without changing them. Validation on the final working tree: 217 passed, 2 skipped; all six pre-commit hooks passed. All 137 passing pre-change baseline tests still pass. Real institutional-store training equivalence remains untested.
Document CLI/config examples, preserved defaults and relative-path semantics, plus the saved run configuration. Replace source-edit advice for training with Hydra overrides. State the training-only scope and unchanged cache-key behavior: different data contents should use separate cache roots. Other entry points keep their existing defaults. Validation: documentation was included in the passing final pre-commit run. No production or test content changed after the 217-pass suite.
MWesselkamp
left a comment
There was a problem hiding this comment.
looks good to me! just a small comment on the description, and on the default paths.
| ### Training data and cache paths | ||
|
|
||
| `paths.eozarr_root` and `paths.preload_cache_root` override the data-store and preload-cache | ||
| roots used by `deep_lst/train.py`, for all three training/validation/test splits. Both default |
There was a problem hiding this comment.
The description says the paths are in train.py but they are the ExperimentPaths defaults in deep_lst/utils/io.py (used by train.py and by the PATHS singleton in dataset.py). Please correct docs/CONFIGURATION.md and the PR text so the next reader finds them.
There was a problem hiding this comment.
Thanks, I corrected the docs and the PR description. They now point to ExperimentPaths in deep_lst/utils/io.py and also mention the PATHS instance in dataset.py.
|
|
||
| 1. `ExperimentPaths.preload_cache_root` in `deep_lst/utils/io.py` points at the maintainer's | ||
| tscratch directory, which you cannot write to. Point it at your own tscratch directory. | ||
| 1. `ExperimentPaths.preload_cache_root` in `deep_lst/utils/io.py` defaults to the maintainer's |
There was a problem hiding this comment.
preload_cache_root must not default to a path containing a username (it is unwritable for anyone else, see RUNBOOK §3). Derive the default from $DEEP_LST_CACHE_ROOT or $USER; paths.preload_cache_root keeps overriding it.
There was a problem hiding this comment.
Thanks, I changed this to use DEEP_LST_CACHE_ROOT, or the current user's scratch directory when it is not set. The config override still takes priority. I added tests for this too, including when USER is missing.
Use DEEP_LST_CACHE_ROOT when set, otherwise choose the current user's cluster scratch path. Keep explicit configuration overrides first. Test environment defaults and override priority, including missing USER.
Point readers to ExperimentPaths in deep_lst/utils/io.py and the PATHS instance in dataset.py. Explain the cache fallback and override priority.
|
|
||
| This first slice applies to training; standalone inference, statistics and fine-tuning entry | ||
| points still use their existing paths. Model, loss and data-processing settings are unchanged. | ||
| These Hydra overrides apply to training. Standalone inference, statistics and fine-tuning use |
There was a problem hiding this comment.
Good for this self-contained addition. But this override should eventually be applied to all modules, I'll add it to the issue.
React
Related to #17.
This makes the training data and cache paths configurable without editing Python files. The defaults are defined in
ExperimentPathsindeep_lst/utils/io.py, which is used bytrain.pyand thePATHSinstance indeep_lst/dataset/dataset.py.The two settings are
paths.eozarr_rootandpaths.preload_cache_root. For example:An explicit cache setting takes priority. Without it, the cache uses
DEEP_LST_CACHE_ROOTwhen it is set and nonempty, otherwise/Net/Groups/BGI/tscratch/$USER/tmp. IfUSERis missing or empty, it uses the account name. The data-store default is unchanged. Relative paths still use the working directory, and the settings are saved in the run'sconfig.yml.The Hydra overrides cover training, including validation and test splits. Other commands use the shared
ExperimentPathsdefaults, including the updated cache default. Cache keys are unchanged, so different data contents should use a separate cache directory. Thepaths=bgc|localprofiles are left for a separate PR, as discussed in #17.The local test suite passed: 222 tests passed and 2 were skipped. The added tests cover the environment default, the current-user fallback, and explicit override priority. The existing config and rollout-loss golden fixtures are unchanged. Pre-commit checks also passed. I haven't run training with the real cluster data yet.