W-Twin

Detect progressive neural network training degradation before it shows up in your loss curves.

You are training a model for 3 days. On day 2 something went wrong — but the loss looks normal.
You find out on day 3, when it is too late.

W-Twin catches this 200+ steps earlier by comparing your actual loss trajectory against a scaling-law forecast. When the two diverge, it alerts — before CUSUM or threshold monitors notice anything.

---

30-second example

from wtwin import WTwinMonitor

monitor = WTwinMonitor()

for step, loss in enumerate(your_training_losses, 1):
    state = monitor.update(step, loss)
    if state.alert:
        print(f"⚠ Degradation at step {step}  (W={state.W:.2f})")
        # → rollback checkpoint, reduce LR, or stop run

Or from the command line — no code needed:

pip install git+https://github.com/Kretski/WTwin.git
wtwin monitor training_log.csv
wtwin demo

---

Try it instantly (no install)

Live browser demo: https://kretski.github.io/WTwin/

Drop a CSV with step and loss columns. Everything runs locally — no upload, no server.

---

Results

From the paper — real nano-GPT training runs:

Experiment	Runs	W-Twin	Threshold	CUSUM
Progressive drift	9	9/9 (100%)	0/9 (0%)	0/9 (0%)
Mean detection delay	—	257 steps (SD=67, CI: [209, 305], n=10)	—	—
False alarm rate	30 clean runs	0/30 (0%)	0/30	0/30
Abrupt spike	2	2/2 (+5 steps)	0/2	2/2 (+1 step, faster)

W-Twin is the only method that detects progressive drift.
For sudden spikes, CUSUM is faster — the two are complementary. The CUSUM baseline here uses a simple rolling-mean implementation — a tuned CUSUM may perform better on abrupt spikes.

Scope: Results are from controlled nano-GPT experiments (842K parameters) with synthetically injected failures.

---

Ablation results

Controlled experiments — char-level Transformer (200K params, AdamW + cosine):

Mode	Scenario	Final loss	vs none	Compute saved
none	clean	1.29	—	—
none	degraded	3.03	baseline	—
active (LR×0.5)	degraded	3.05	+0.7% worse	—
early stop	degraded	2.57	−15.3%	27.4%

Early stopping on W-Twin alert: −15% final loss, 27% compute saved.
LR reduction on alert: no improvement (honest negative).

Synthetic label-noise degradation. N=8 seeds. Not production scale.

---

External blind replay validation

W-Twin was evaluated using a blind replay protocol on real EleutherAI Pythia training logs — trajectories not generated by W-Twin itself.

Protocol: Baseline fit on first 20% of trajectory only. Future 80% withheld from detector. Results compared against CUSUM and threshold baselines.

Case	Run	Steps	W-Twin	CUSUM	Threshold	Notes
A — Clean	Pythia-14M normal run	143,000	0 false alarms	0	0	r=0.939 forecast correlation
B — Anomalous	Pythia resumed checkpoint (loss stagnated ~2.8)	143,000	Alert @ step 2,051	no signal	no signal	150 steps post-onset

Key findings:

• W-Twin is the only method that detected the anomalous trajectory
• Clean run: W stable negative throughout (range [−105, −1.7]), r=0.939 forecast correlation
• Anomalous run: alert at step 2,051, ~150 steps after estimated plateau onset (step 1,901)
• Convergence rate post-alert: 440× slower than pre-alert
• Reproduced on two independent machines with independent wtwin installations

Validated on wtwin v1.2.0 without parameter tuning between cases.
Anomaly onset estimated from rolling slope analysis — not ground-truth labeled.
Scripts: wtwin_external_validation.py, fetch_anomalous.py, find_onset.py

---

Scale-free warmup experiment

W-Twin warmup parameter was tested across a wide range on real Pythia logs:

Config	Warmup	Clean FA	Anomaly detected	Lead time
fixed_100	100	✅ 0	✅ step 2,051	+150 steps
fixed_500	500	✅ 0	✅ step 2,051	+150 steps
scale_free (0.5% × T)	71	✅ 0	✅ step 2,051	+150 steps
scale_free (1.0% × T)	143	✅ 0	✅ step 2,051	+150 steps
scale_free (0.2% × T)	50	✅ 0	✅ step 2,051	+150 steps

Key finding: W-Twin results are robust across warmup values from 50 to 500 steps.
Scale-free warmup = 0.5% × T_train produces identical results to fixed warmup=100.
No manual tuning required per model size.

Tested on real Pythia-14M logs (2 runs). use_adaptive_T=True produced false alarms on clean run — not recommended without further tuning.

---

Cosine annealing restarts — pilot test

Scenario	Restarts	False alarms	Detection
No restarts	—	✅ 0/3	—
3 restarts (@200,400,600)	yes	✅ 0/3	—
Restarts + degradation @500	yes	0/3	✅ 3/3 (delay 75-105 steps)

W-Twin with fixed threshold (use_adaptive_T=False, default) shows 0 false alarms at restart points in this pilot test.

N=3 seeds. Pilot signal only — not a robust claim. use_adaptive_T=True NOT tested here.

---

How it works

W-Twin fits a power-law baseline to early training steps, then at every step computes:

W(t) = Q(t) · (D(t) − α)

D(t) = (L_obs(t) − L_pred(t)) / σ_local(t)   ← deviation from expected curve
Q(t) = exp(−MSE_fit / τ)                        ← confidence in the baseline
α    = 2.0                                       ← detection threshold (z-score)

An alert fires when W(t) > 0 for 5 consecutive steps.

---

Drop-in optimizer wrapper

from wtwin_optimizer import WTwinAdamW

optimizer = WTwinAdamW(
    model.parameters(),
    lr=3e-4,
    betas=(0.9, 0.95),
    preset='pretraining',
    wtwin_on_alert=lambda step, W, state: save_checkpoint(step)
)

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step(loss=loss)

Presets: pretraining, adaptive_pretraining, finetuning (experimental), custom

---

HuggingFace Trainer

from wtwin_trainer_callback import WTwinCallback

trainer = Trainer(
    model=model,
    args=training_args,
    callbacks=[WTwinCallback(preset="pretraining")]
)

Note: Requires step-level logging (logging_steps=1).
For short fine-tuning runs (<500 steps), use WTwinAdamW directly.

---

Installation

pip install git+https://github.com/Kretski/WTwin.git

Dependencies: numpy, scipy only. No framework lock-in.

---

Limitations

• Validated on nano-GPT (842K params, synthetic) and Pythia-14M (real, 143K steps)
• Power-law baseline assumes monotonically decreasing loss
• use_adaptive_T=True not recommended with cosine annealing restarts — adaptive threshold history is contaminated by loss jump at restart, producing false alarms. Use fixed threshold (default).
• Scale behavior at 1M+ steps: unknown
• RLHF, LoRA, multi-phase schedules: not validated
• suggest() classifier not validated on labeled real failures

Full details in the paper.

---

Did it work for you?

If you ran W-Twin on a real training log — whether it detected something or missed it — please open an issue.

One real training log is worth more than 100 synthetic benchmarks.

---

Citation

@software{kretski2026wtwin,
  author    = {Kretski, Dimitar},
  title     = {W-Twin: Forecast-Based Detection of Progressive
               Neural Network Training Degradation},
  year      = {2026},
  doi       = {10.5281/zenodo.21951979},
  url       = {https://zenodo.org/records/21951979},
  publisher = {Zenodo}
}

---

License

MIT — see LICENSE.

Author: Dimitar Kretski, Center for Hydro- and Aerodynamics, Varna, Bulgaria
ORCID: 0000-0001-5108-2243