Fix SciTS tutorial issues found in an end-to-end Aurora run

docs/tutorial/README.md

@@ -1,10 +1,10 @@
 # SciTS Forecasting with Intern-S2 Preview 397B
 
-A short path from environment setup through training with the numeric
-`SciTS-forecast` WebDataset split, including the current evaluation boundary.
+A short path from environment setup through training and held-out evaluation
+on the numeric `SciTS-forecast` WebDataset split.
 
-This configuration combines a **Qwen3-0.6B** language-model trunk, the
-pretrained **[Intern-S2 Preview 397B](https://arxiv.org/html/2608.13505) time-series encoder**, and PRISM's own
+This configuration combines a frozen **OLMo-1B** language-model trunk, the
+frozen pretrained **[Intern-S2 Preview 397B](https://arxiv.org/html/2608.13505) time-series encoder**, and PRISM's own
 trainable linear quantile forecast head. It extracts only the time-series
 encoder weights from the 397B checkpoint; it does not train or load the full
 397B language model, and it does not use Intern-S2's native forecaster.
@@ -16,8 +16,15 @@ encoder weights from the 397B checkpoint; it does not train or load the full
 | [Environment](environment.md) | Aurora environment, encoder checkpoint extraction, model cache |
 | [Data](data.md) | Convert SciTS into train/validation shards with numeric targets and masks |
 | [Training](training.md) | Smoke and full `SCITS-FORECAST` runs |
-| [Evaluation](evaluation.md) | Training-loss signals and the missing numeric held-out evaluator |
+| [Evaluation](evaluation.md) | Training signals, checkpoints, and held-out forecast scoring |
+| [Notebook](olmo_scits_forecast.ipynb) | The smoke run and checkpoint evaluation, step by step on a debug node |
 
-You need an Aurora allocation for training and a local SciTS source tree for
-conversion. Run the commands from the repository root. Replace path
-placeholders with shared paths available on both the login and compute nodes.
+You need an Aurora allocation for training. Run the commands from the
+repository root. Replace path placeholders with shared paths available on
+both the login and compute nodes.
+
+> **Genesis hackathon:** the model cache, extracted encoder, converted
+> `SciTS-forecast` data, and a trained `step_2000` checkpoint are pre-staged
+> under `/flare/GenesisHackathonOct26/scits`. You can skip the encoder
+> extraction and data conversion steps and point `HF_HOME` and
+> `PRISM_DATA_ROOT` there (see the [notebook](olmo_scits_forecast.ipynb)).

docs/tutorial/data.md

@@ -5,6 +5,11 @@ PRISM's converter writes `SciTS-forecast/` with `shards/`, `val_shards/`, and
 `manifest.json`. Forecast examples contain a history array, a numeric target,
 an observation mask, prompt text, and metadata.
 
+> **Genesis hackathon:** a converted copy is pre-staged at
+> `/flare/GenesisHackathonOct26/scits/data/SciTS-forecast`. Set
+> `PRISM_DATA_ROOT=/flare/GenesisHackathonOct26/scits/data`, run the checks
+> below, and skip the conversion.
+
 ## Convert SciTS
 
 The source tree must contain `meta_data.jsonl` and the referenced series files.
@@ -26,7 +31,10 @@ python applications/timeseries/convert_scits_to_webdataset.py \
 The split is deterministic by sample ID. The converter keeps forecast targets
 in `.gt_ts.npy` and their validity in `.gt_ts_mask.npy`; missing observations
 are not treated as target values. The model preset supports up to ten target
-channels. Do not use `--max-samples` for the production conversion.
+channels. A full conversion processes about 52,000 samples at roughly
+4–5 samples per second, so expect about 3 hours on a login node. To try the
+converter quickly, add `--max-samples 300` and use a separate output
+directory; do not use `--max-samples` for the production conversion.
 
 ## Check the converted dataset
 
@@ -40,4 +48,4 @@ Pass the `SciTS-forecast` root to the Aurora WebDataset launcher. It validates
 the manifest and stages training shards from Lustre to the compute nodes; the
 held-out shard directory is made available to the trainer as well.
 
-Next: [train the forecast model](training.md).
+Next: [train the forecast model](training.md).

docs/tutorial/environment.md

@@ -1,9 +1,21 @@
 # 1. Environment
 
-Training uses Qwen3-0.6B, PRISM's Intern-S2 Preview 397B time-series encoder,
+Training uses OLMo-1B, PRISM's Intern-S2 Preview 397B time-series encoder,
 and a newly initialized linear quantile head. The 397B label identifies the
 source model; PRISM extracts only its time-series encoder weights.
 
+## Clone the repository
+
+PRISM vendors the Walrus encoder as a git submodule, and the environment build
+fails if it is missing, so clone recursively:
+
+```bash
+git clone --recursive https://github.com/AI-ModCon/PRISM.git
+cd PRISM
+git checkout scits-demo
+git submodule update --init --recursive
+```
+
 ## Build the Aurora environment
 
 The Intern-S2 implementation requires the `--intern-s2` environment variant,
@@ -13,7 +25,7 @@ which installs Transformers 5.2.0. On Aurora:
 module purge
 module use /opt/aurora/26.26.0/modulefiles
 module load oneapi/release/2025.3.1
-module load mpich/prd/5.0.0.aurora_test.51a9474 
+module load mpich/prd/5.0.0.aurora_test.51a9474
 module load frameworks/2025.3.1
 export VENV_PATH=/flare/GenesisHackathonOct26/$USER/prism-envs/py3.12-intern-s2
 bash tools/build_aurora_env.sh --intern-s2
@@ -30,13 +42,14 @@ with vLLM's declared version range; vLLM is not needed for this tutorial. See
 
 Choose shared cache locations visible from compute nodes. Set `HF_TOKEN` in
 your local ignored `.env` or through your site's secret manager; the extraction
-script reads it from the environment or `.env`. For the Genesis hackathon, the project field can be `GenesisHackathonOct26`
+script reads it from the environment or `.env`. For the Genesis hackathon, use
+`GenesisHackathonOct26` as the project in paths and job submissions.
 
 ```bash
 export HF_HOME=/flare/GenesisHackathonOct26/$USER/huggingface
 ```
 
-Make sure the public Qwen3-0.6B checkpoint is also present in the shared
+Make sure the public `allenai/OLMo-1B-0724-hf` checkpoint is also present in the shared
 Hugging Face cache before launching a job that cannot access the network.
 
 ## Extract the time-series encoder

docs/tutorial/evaluation.md

@@ -1,30 +1,93 @@
 # 4. Evaluation
 
-The full `SCITS-FORECAST` experiment enables evaluation logging every 250
-steps, but the current path does **not** produce a numeric held-out forecast
-score. Under the HSDP strategy used by the launch recipe, the native trainer
-skips its rank-0-only held-out forward to avoid unmatched collectives and
-records the running training loss instead. That training loss includes the
-masked quantile objective, but is not a validation result.
-
-Switching to DDP does not fix the numeric validation gap yet: the native
-time-series validation loader currently reads only `ts.npy` and `text` from
-each held-out sample. It does not pass `.gt_ts.npy` or `.gt_ts_mask.npy` to the
-model, so that forward cannot score the forecast head against numeric truth.
-
-## Text evaluator limitation
-
-`applications/timeseries/launch_capacity_intern_s2_397b.sh` appends
-`tools/universal_evaluator.py --mode verify_timeseries_scits` after training.
-That evaluator also reads only the time-series history and `.text` prompt /
-answer; it does not load `.gt_ts.npy` or `.gt_ts_mask.npy`. SciTS-forecast
-examples carry forecast truth in those numeric members, so the text evaluator
-is not a numeric forecast score. This branch does not yet provide a separate
-post-training numeric metric CLI.
-
-For now, the logged training loss can confirm that optimization is progressing,
-but it cannot establish held-out forecast quality. A numeric MAE or quantile
-score requires an evaluator that runs the checkpoint on `val_shards/` and
-consumes predictions together with the target mask.
-
-Back to the [tutorial index](README.md).
+Evaluation has three stages: watch the training loss while the job runs,
+collect the checkpoints it saves, then score a checkpoint's forecasts against
+the held-out `val_shards/`. Only the last stage measures forecast quality.
+
+| Stage | Where | What it tells you |
+|---|---|---|
+| 1. Training signals | Job log and `checkpoints/perf.jsonl` | Whether optimization is progressing |
+| 2. Checkpoints | `checkpoints/step_<N>/model.safetensors` | Which model states you can score |
+| 3. Held-out forecast scoring | [Tutorial notebook](olmo_scits_forecast.ipynb), section 4 | Forecast accuracy and calibration on unseen series |
+
+All paths below are relative to the run directory,
+`outputs/<run-id>/<date>/<time>/`.
+
+## 1. Training signals
+
+The trainer appends one JSON record per log interval to
+`checkpoints/perf.jsonl`. Records with `"site": "trainer_native_per_log"` hold
+the step, the global training `loss`, and throughput. Every
+`eval_every_n_steps` (250 for the full run), it also writes an `"event": "eval"`
+record. That record repeats the current running training loss
+(`"loss_source": "train_running_mean"`). It is **not** a held-out score.
+
+The job log breaks the loss down by objective on each logged step, for
+example from the smoke run:
+
+```text
+Step 20: Loss 1.1451 (rank_mean=1.1852, rank0=2.0761, tokens=44985)
+  [LOSS] text=1.3194 (n=3153) | time_series=0.3805 (n=95)
+```
+
+- `time_series` is the masked pinball (quantile) loss of the forecast head
+  over `n` valid target cells. This is the number to watch for forecasting.
+- `text` is next-token loss over `n` answer tokens. SciTS also contains
+  non-forecast tasks (classification, multiple choice) whose answers are text;
+  forecast samples mask their own text labels, so they do not contribute here.
+
+The headline `Loss` combines both terms, so it can fall mostly because the
+text term falls. Judge forecasting progress from `time_series`. A healthy
+run shows finite values for both terms from the first logged step.
+
+## 2. Checkpoints
+
+The full design saves a checkpoint every 500 steps (`step_500`, `step_1000`,
+…); the smoke run saves `step_25` and `step_50`. Each directory holds
+`model.safetensors` plus the optimizer and scheduler state needed to resume.
+Only `model.safetensors` is needed for evaluation.
+
+## 3. Held-out forecast scoring
+
+Section 4 of the [notebook](olmo_scits_forecast.ipynb) scores one
+checkpoint end to end:
+
+1. **Load** – build the model from the
+   `prism_olmo1b_intern_s2_397b_ts_forecast` preset and load the
+   checkpoint's `model.safetensors`.
+2. **Read held-out data** – stream `val_shards/` with the same loader the
+   trainer uses. The loader caps the history length, decodes the numeric
+   target (`.gt_ts.npy`) and its validity mask (`.gt_ts_mask.npy`), and
+   computes each channel's history mean and scale.
+3. **Forecast** – run the training forward pass and capture the forecast
+   head's de-normalized quantiles (horizon × channels × {0.1, 0.5, 0.9}).
+4. **Score** – compare the predictions with the target on valid (unmasked)
+   cells only, and report the metrics per sample and per SciTS task.
+
+| Metric | Meaning |
+|---|---|
+| `mae`, `rmse` | Error of the median forecast, in the series' original units |
+| `smae` | MAE divided by the channel's history scale, comparable across tasks |
+| `smae_mean`, `smae_last` | The same score for two naive baselines: history mean and last observed value |
+| `skill_vs_last` | `1 - smae / smae_last`; positive means the model beats persistence |
+| `pinball` | Scaled pinball loss over all quantiles (the training objective) |
+| `coverage80` | Fraction of targets inside the 0.1–0.9 band; ≈ 0.8 is well calibrated |
+
+Start with `skill_vs_last` and `coverage80`. A model that does not beat
+last-value persistence has not yet learned a useful forecast. A coverage far
+below 0.8 means the quantile band is too narrow.
+
+To track progress during training, point the notebook's `EVAL_CHECKPOINT` at
+successive `step_<N>` directories and compare the tables. The notebook caps
+scoring at `N_EVAL` samples (at most `MAX_PER_TASK` per task) for speed;
+raise `N_EVAL` to cover all of `val_shards/` before reporting numbers.
+
+## Known limitations
+
+- There is no command-line numeric evaluator yet; held-out forecast scoring
+  lives in the notebook.
+- `tools/universal_evaluator.py --mode verify_timeseries_scits` scores the
+  **text** answer of each SciTS sample. It ignores the numeric target and mask,
+  so it does not evaluate the forecast head.
+
+Back to the [tutorial index](README.md).

docs/tutorial/olmo_scits_forecast.ipynb

@@ -45,7 +45,7 @@
     "jupyter lab --no-browser --ip=\"$(hostname)\" --port 8888\n",
     "```\n",
     "\n",
-    "The venv is the one built on the [environment page](environment.md) with `tools/build_aurora_env.sh --intern-s2`. It does not include Jupyter; install it once from a login node with `uv pip install jupyterlab` while the venv is active. The model cache, extracted encoder, SciTS-forecast data, and trained `step_2000` checkpoint are already staged under `/flare/GenesisHackathonOct26/scits`.\n",
+    "The venv is the one built on the [environment page](environment.md) with `tools/build_aurora_env.sh --intern-s2`. It does not include Jupyter; install it once from a login node with `uv pip install jupyterlab ipywidgets` while the venv is active (`ipywidgets` lets progress bars render). The model cache, extracted encoder, SciTS-forecast data, and trained `step_2000` checkpoint are already staged under `/flare/GenesisHackathonOct26/scits`.\n",
     "\n",
     "Forward the port from your laptop (`ssh -L 8888:<compute-node>:8888 <user>@aurora.alcf.anl.gov` through the UAN), then open this notebook. The smoke job needs the whole node (12 XPUs), so run the evaluation section after it finishes."
    ]
@@ -234,7 +234,8 @@
     "    if re.match(r\"\\s*export (WEBDATASET_\\w+|LOCAL_SHARDS_DIR|VENV_PATH|HF_HOME)=\", line):\n",
     "        print(line.strip())\n",
     "for pattern in (r\"hydra\\.run\\.dir=\\S+\", r\"training\\.max_steps=\\S+\", r\"model=\\S+\"):\n",
-    "    print(*sorted(set(re.findall(pattern, script_text))))"
+    "    # The design value comes first and the CLI override last; the last one wins.\n",
+    "    print(re.findall(pattern, script_text)[-1])"
    ]
   },
   {
@@ -341,7 +342,7 @@
    "source": [
     "## 4. Evaluate the `step_2000` checkpoint\n",
     "\n",
-    "As explained on the [evaluation page](evaluation.md), the trainer's periodic `eval` records under HSDP are running training loss, and the text evaluator ignores `.gt_ts.npy` / `.gt_ts_mask.npy`. This section therefore scores the forecast head directly: it runs the checkpoint on held-out `val_shards/` and compares the predicted quantiles with the numeric targets, using only cells where the target mask is valid.\n",
+    "As explained on the [evaluation page](evaluation.md), the trainer's periodic `eval` records repeat the running training loss, and the text evaluator ignores `.gt_ts.npy` / `.gt_ts_mask.npy`. This section therefore scores the forecast head directly: it runs the checkpoint on held-out `val_shards/` and compares the predicted quantiles with the numeric targets, using only cells where the target mask is valid.\n",
     "\n",
     "First, the training loss of the full run up to the evaluated checkpoint:"
    ]
@@ -384,12 +385,15 @@
     "import torch\n",
     "from omegaconf import OmegaConf\n",
     "from safetensors.torch import load_file\n",
+    "from transformers.utils import logging as hf_logging\n",
     "\n",
     "from src.config import ModelConfig\n",
     "from src.decoders import remap_legacy_decoder_keys\n",
     "from src.model import UnifiedTransformer\n",
     "from tools.universal_evaluator import _align_checkpoint_backbone_vocab\n",
     "\n",
+    "hf_logging.disable_progress_bar()  # keep weight-loading progress out of the cell output\n",
+    "\n",
     "preset = OmegaConf.to_container(OmegaConf.load(MODEL_PRESET), resolve=True)\n",
     "preset[\"llm_backbone_id\"] = preset.pop(\"backbone_id\")\n",
     "preset[\"llm_tokenizer_id\"] = preset.pop(\"tokenizer_id\")\n",

docs/tutorial/training.md

@@ -30,6 +30,7 @@ python tools/launch_aurora_web.py \
     --file experiments/LLM_timeseries_scaling.yaml \
     --design PRISM-OLMO-1B-SCITS-FORECAST \
     --nodes 1 \
+    --project GenesisHackathonOct26 \
     --batch \
     --queue debug \
     --walltime 00:30:00 \
@@ -67,7 +68,9 @@ every 500 steps under
 `outputs/OLMO-SCITS-FORECAST/<date>/<time>/checkpoints/`. It consumes numeric
 targets and their masks; text generation is not the forecast output.
 
-The command uses HSDP to match the branch's capacity launch recipe. See
-[evaluation](evaluation.md) for what its `eval` records do and do not measure.
+`--project` sets the PBS allocation (`#PBS -A`) and defaults to `ModCon`;
+use your own allocation outside the hackathon. See
+[evaluation](evaluation.md) for how to check the run and score its
+checkpoints.
 
 Next: [evaluate the run](evaluation.md).