From fe39e358500ceb141831a1c771d42b5dbcc8f815 Mon Sep 17 00:00:00 2001 From: Jared Willard Date: Wed, 14 Oct 2026 00:16:17 +0000 Subject: [PATCH] Fix SciTS tutorial issues found in an end-to-end Aurora run - evaluation.md: replace the DDP/HSDP discussion with an overview of the evaluation pipeline (training signals, checkpoints, held-out scoring in the notebook) and a metrics guide; keep limitations short at the end. - README/environment: the trunk is OLMo-1B, not Qwen3-0.6B; link the notebook; add a recursive clone + submodule step (the env build fails at step 5/7 without the walrus submodule). - training.md: pass --project (defaults to ModCon). - data.md: point hackathon users at the pre-staged data and note the ~3 h full-conversion runtime. - notebook: show the effective max_steps override, hide weight-loading progress bars, install ipywidgets alongside jupyterlab. Co-Authored-By: Claude Opus 5.5 --- docs/tutorial/README.md | 23 +++-- docs/tutorial/data.md | 12 ++- docs/tutorial/environment.md | 21 ++++- docs/tutorial/evaluation.md | 119 ++++++++++++++++++------ docs/tutorial/olmo_scits_forecast.ipynb | 10 +- docs/tutorial/training.md | 7 +- 6 files changed, 145 insertions(+), 47 deletions(-) diff --git a/docs/tutorial/README.md b/docs/tutorial/README.md index 2a5c81f..523db48 100644 --- a/docs/tutorial/README.md +++ b/docs/tutorial/README.md @@ -1,10 +1,10 @@ # SciTS Forecasting with Intern-S2 Preview 397B -A short path from environment setup through training with the numeric -`SciTS-forecast` WebDataset split, including the current evaluation boundary. +A short path from environment setup through training and held-out evaluation +on the numeric `SciTS-forecast` WebDataset split. -This configuration combines a **Qwen3-0.6B** language-model trunk, the -pretrained **[Intern-S2 Preview 397B](https://arxiv.org/html/2608.13505) time-series encoder**, and PRISM's own +This configuration combines a frozen **OLMo-1B** language-model trunk, the +frozen pretrained **[Intern-S2 Preview 397B](https://arxiv.org/html/2608.13505) time-series encoder**, and PRISM's own trainable linear quantile forecast head. It extracts only the time-series encoder weights from the 397B checkpoint; it does not train or load the full 397B language model, and it does not use Intern-S2's native forecaster. @@ -16,8 +16,15 @@ encoder weights from the 397B checkpoint; it does not train or load the full | [Environment](environment.md) | Aurora environment, encoder checkpoint extraction, model cache | | [Data](data.md) | Convert SciTS into train/validation shards with numeric targets and masks | | [Training](training.md) | Smoke and full `SCITS-FORECAST` runs | -| [Evaluation](evaluation.md) | Training-loss signals and the missing numeric held-out evaluator | +| [Evaluation](evaluation.md) | Training signals, checkpoints, and held-out forecast scoring | +| [Notebook](olmo_scits_forecast.ipynb) | The smoke run and checkpoint evaluation, step by step on a debug node | -You need an Aurora allocation for training and a local SciTS source tree for -conversion. Run the commands from the repository root. Replace path -placeholders with shared paths available on both the login and compute nodes. +You need an Aurora allocation for training. Run the commands from the +repository root. Replace path placeholders with shared paths available on +both the login and compute nodes. + +> **Genesis hackathon:** the model cache, extracted encoder, converted +> `SciTS-forecast` data, and a trained `step_2000` checkpoint are pre-staged +> under `/flare/GenesisHackathonOct26/scits`. You can skip the encoder +> extraction and data conversion steps and point `HF_HOME` and +> `PRISM_DATA_ROOT` there (see the [notebook](olmo_scits_forecast.ipynb)). diff --git a/docs/tutorial/data.md b/docs/tutorial/data.md index 50e0d39..ac58917 100644 --- a/docs/tutorial/data.md +++ b/docs/tutorial/data.md @@ -5,6 +5,11 @@ PRISM's converter writes `SciTS-forecast/` with `shards/`, `val_shards/`, and `manifest.json`. Forecast examples contain a history array, a numeric target, an observation mask, prompt text, and metadata. +> **Genesis hackathon:** a converted copy is pre-staged at +> `/flare/GenesisHackathonOct26/scits/data/SciTS-forecast`. Set +> `PRISM_DATA_ROOT=/flare/GenesisHackathonOct26/scits/data`, run the checks +> below, and skip the conversion. + ## Convert SciTS The source tree must contain `meta_data.jsonl` and the referenced series files. @@ -26,7 +31,10 @@ python applications/timeseries/convert_scits_to_webdataset.py \ The split is deterministic by sample ID. The converter keeps forecast targets in `.gt_ts.npy` and their validity in `.gt_ts_mask.npy`; missing observations are not treated as target values. The model preset supports up to ten target -channels. Do not use `--max-samples` for the production conversion. +channels. A full conversion processes about 52,000 samples at roughly +4–5 samples per second, so expect about 3 hours on a login node. To try the +converter quickly, add `--max-samples 300` and use a separate output +directory; do not use `--max-samples` for the production conversion. ## Check the converted dataset @@ -40,4 +48,4 @@ Pass the `SciTS-forecast` root to the Aurora WebDataset launcher. It validates the manifest and stages training shards from Lustre to the compute nodes; the held-out shard directory is made available to the trainer as well. -Next: [train the forecast model](training.md). \ No newline at end of file +Next: [train the forecast model](training.md). diff --git a/docs/tutorial/environment.md b/docs/tutorial/environment.md index a76304c..325189c 100644 --- a/docs/tutorial/environment.md +++ b/docs/tutorial/environment.md @@ -1,9 +1,21 @@ # 1. Environment -Training uses Qwen3-0.6B, PRISM's Intern-S2 Preview 397B time-series encoder, +Training uses OLMo-1B, PRISM's Intern-S2 Preview 397B time-series encoder, and a newly initialized linear quantile head. The 397B label identifies the source model; PRISM extracts only its time-series encoder weights. +## Clone the repository + +PRISM vendors the Walrus encoder as a git submodule, and the environment build +fails if it is missing, so clone recursively: + +```bash +git clone --recursive https://github.com/AI-ModCon/PRISM.git +cd PRISM +git checkout scits-demo +git submodule update --init --recursive +``` + ## Build the Aurora environment The Intern-S2 implementation requires the `--intern-s2` environment variant, @@ -13,7 +25,7 @@ which installs Transformers 5.2.0. On Aurora: module purge module use /opt/aurora/26.26.0/modulefiles module load oneapi/release/2025.3.1 -module load mpich/prd/5.0.0.aurora_test.51a9474 +module load mpich/prd/5.0.0.aurora_test.51a9474 module load frameworks/2025.3.1 export VENV_PATH=/flare/GenesisHackathonOct26/$USER/prism-envs/py3.12-intern-s2 bash tools/build_aurora_env.sh --intern-s2 @@ -30,13 +42,14 @@ with vLLM's declared version range; vLLM is not needed for this tutorial. See Choose shared cache locations visible from compute nodes. Set `HF_TOKEN` in your local ignored `.env` or through your site's secret manager; the extraction -script reads it from the environment or `.env`. For the Genesis hackathon, the project field can be `GenesisHackathonOct26` +script reads it from the environment or `.env`. For the Genesis hackathon, use +`GenesisHackathonOct26` as the project in paths and job submissions. ```bash export HF_HOME=/flare/GenesisHackathonOct26/$USER/huggingface ``` -Make sure the public Qwen3-0.6B checkpoint is also present in the shared +Make sure the public `allenai/OLMo-1B-0724-hf` checkpoint is also present in the shared Hugging Face cache before launching a job that cannot access the network. ## Extract the time-series encoder diff --git a/docs/tutorial/evaluation.md b/docs/tutorial/evaluation.md index 89df1a6..d338af0 100644 --- a/docs/tutorial/evaluation.md +++ b/docs/tutorial/evaluation.md @@ -1,30 +1,93 @@ # 4. Evaluation -The full `SCITS-FORECAST` experiment enables evaluation logging every 250 -steps, but the current path does **not** produce a numeric held-out forecast -score. Under the HSDP strategy used by the launch recipe, the native trainer -skips its rank-0-only held-out forward to avoid unmatched collectives and -records the running training loss instead. That training loss includes the -masked quantile objective, but is not a validation result. - -Switching to DDP does not fix the numeric validation gap yet: the native -time-series validation loader currently reads only `ts.npy` and `text` from -each held-out sample. It does not pass `.gt_ts.npy` or `.gt_ts_mask.npy` to the -model, so that forward cannot score the forecast head against numeric truth. - -## Text evaluator limitation - -`applications/timeseries/launch_capacity_intern_s2_397b.sh` appends -`tools/universal_evaluator.py --mode verify_timeseries_scits` after training. -That evaluator also reads only the time-series history and `.text` prompt / -answer; it does not load `.gt_ts.npy` or `.gt_ts_mask.npy`. SciTS-forecast -examples carry forecast truth in those numeric members, so the text evaluator -is not a numeric forecast score. This branch does not yet provide a separate -post-training numeric metric CLI. - -For now, the logged training loss can confirm that optimization is progressing, -but it cannot establish held-out forecast quality. A numeric MAE or quantile -score requires an evaluator that runs the checkpoint on `val_shards/` and -consumes predictions together with the target mask. - -Back to the [tutorial index](README.md). \ No newline at end of file +Evaluation has three stages: watch the training loss while the job runs, +collect the checkpoints it saves, then score a checkpoint's forecasts against +the held-out `val_shards/`. Only the last stage measures forecast quality. + +| Stage | Where | What it tells you | +|---|---|---| +| 1. Training signals | Job log and `checkpoints/perf.jsonl` | Whether optimization is progressing | +| 2. Checkpoints | `checkpoints/step_/model.safetensors` | Which model states you can score | +| 3. Held-out forecast scoring | [Tutorial notebook](olmo_scits_forecast.ipynb), section 4 | Forecast accuracy and calibration on unseen series | + +All paths below are relative to the run directory, +`outputs///