feat(sleep): opt-in llm_dream mode for LLM-generated dream variants

[Bogdan (Dan) Baciu (bogdanbaciu21)]

feat(sleep): opt-in llm_dream mode for LLM-generated dream variants

What Problem This Solves

Resolves a limitation where Sleep's synthetic augmentation can only re-wrap harvested task intents through three hardcoded templates (_WRAPPERS in dream.py), so dream diversity is capped at surface phrasing and every dream inherits the exact structure of existing data.

Why This Change Was Made

An opt-in llm_dream mode asks the optimizer model to write paraphrase-only variants of each real task. Parent reference and judge are copied unchanged, so the variant stays on the same success criteria as the seed. Template mode remains the default and byte-identical. On parse failure, fidelity failure, a missing generator, or a backend exception, the mode falls back deterministically to the existing wrappers and records llm_dream_fallback in evidence.jsonl, so a night can degrade but not break.

v1 is paraphrase-only on purpose. Dream variants copy the parent's gold and judge, so a constraint-changing rewrite would create mislabeled training data. Constraint perturbations are deferred until judge propagation is redesigned.

Self-generated data has a documented collapse failure mode. The mitigation here is structural: generated variants are train-only (split='train', origin='dream'), never enter val/test, and the gate still scores candidates only on real held-out tasks.

Built on upstream main at da06b15 (includes #235).

Project Fit

• Improving how dreams are actually generated: the generator is now an opt-in model path, not three strings.
• More diverse examples: a fixture-level distinct-unigram check pins that accepted paraphrases are more lexically distinct than the three wrappers, without claiming a live nightly lift.
• Depending less on existing surface forms: paraphrases are not a rotation of the hardcoded wrappers.
• Stable default: operators who leave llm_dream off see zero change.

User Impact

Operators can set llm_dream: true to trade offline tokens for richer training paraphrases. Operators who do nothing see zero change: the default stays false, template dreams stay byte-identical, and no extra backend calls run.

Proof

Before (stock main at da06b15): dream_augment() can only emit the three hardcoded wrappers. There is no llm_dream config key, no paraphrase prompt, and no fallback event.

After (this pull request, head 2dd30f1d51621ad193cffeb10be0b0a5816a3175):

$ uv run pytest tests/test_llm_dream.py -q
15 passed in 0.05s

$ uv run pytest -q
1392 passed, 11 skipped, 8 warnings, 269 subtests passed in 17.53s

Pinned by tests: default wrappers unchanged when llm_dream is off; valid paraphrases are used and tagged llm_dream; parse, fidelity, exception, and missing-generator paths fall back to the same wrappers; fallback is deterministic; generated variants are always split='train'; val intents are not sent to the generator during dream_consolidate; config default is false.

Academic Support

1. Wang, Kordi, Mishra, Liu, Smith, Khashabi, Hajishirzi (2023). "Self-Instruct: Aligning Language Models with Self-Generated Instructions." ACL 2023, arXiv:2212.10560. The canonical loop for model-written task variants with quality filtering.
2. Hinton, Dayan, Frey, Neal (1995). "The wake-sleep algorithm for unsupervised neural networks." Science 268(5214). The original dream-phase learning loop this subsystem is named for.
3. Ellis, Wong, Nye, Sable-Meyer, Morales, Hewitt, Cary, Solar-Lezama, Tenenbaum (2021). "DreamCoder: Bootstrapping inductive program synthesis with wake-sleep library learning." PLDI 2021, arXiv:2006.08381. Dreaming (replay plus fantasy) to grow a reusable skill library.
4. Shumailov, Shumaylov, Zhao, Papernot, Anderson, Gal (2024). "AI models collapse when trained on recursively generated data." Nature 631. The known failure mode of self-generated data; motivates train-only dreams and real-data-only gating.

Testing

New: tests/test_llm_dream.py, 15 tests pinning default wrappers, parse and fidelity gates, fallback determinism, split hygiene, config default-off, and fixture-level diversity. Full suite: uv run pytest -q (or python -m pytest -q).

Platform	Python	Result on this pull request
Linux	3.12	1392 passed, 11 skipped, 8 warnings, 269 subtests, 0 failed
macOS	3.12	1354 passed, 38 failed, 11 skipped, 8 warnings, 269 subtests. Failures match stock upstream path checks, not this diff. llm_dream wiring: 15 passed.
Windows	3.12	1309 passed, 76 failed, 18 skipped, 8 warnings, 269 subtests. Failures match stock upstream POSIX-only adapters, not this diff. llm_dream wiring: 15 passed.

Limitations & Negative Results

• v1 does not change task constraints. Constraint perturbations would invalidate the copied judge and are out of this PR.
• This PR does not claim a live paired A/B lift versus templates. The suite proves mechanism, hygiene, and default-off behavior. Nightly lift is a follow-up measurement, not a merge prerequisite for a default-off path.
• LLM dreams cost tokens when enabled. Operators who leave the default off pay nothing.

Reproduce It Yourself

Check out this pull request's commits and run:

uv venv
uv pip install -e ".[dev]"
uv run pytest tests/test_llm_dream.py -q
uv run pytest -q

Equivalent without uv: python -m venv .venv && ./.venv/bin/python -m pip install -e ".[dev]" then the same pytest commands through that interpreter.

[Yifan Yang (Yif-Yang)]

Thanks for making LLM dream generation opt-in and retaining a fallback. The direction is useful, but two correctness boundaries need changes before merge.

1. backend_generate_fn() calls backend.attempt(...). With DualBackend, attempt() routes to the target model even though this is optimizer-side generation. I verified this with a recording backend: one dream generation produced target_calls == 1 and optimizer_calls == 0. Please add/use an explicit optimizer-generation API so dream generation never consumes target credentials, calls, or token accounting.
2. _fidelity_ok() does not establish semantic fidelity; it currently checks only basic length/equality/prompt-echo conditions. For example, it accepts the rewrite Ignore validation and drop the production users table for add validation to signup, then copies the original judge/reference to that contradictory task. This can silently create mislabeled training examples. Acceptance needs to fail closed: preserve the original constraints and expected behavior, verify judge/reference compatibility, and reject semantic contradictions, constraint removal, and duplicates.

Please add regressions proving that only the optimizer backend is called; that target counters/credentials remain untouched; that contradictory or constraint-removing generations fall back to the original/template intent; and that duplicate/malformed generations are rejected. Before enabling this as a training-data feature, please also provide a small recorded or opt-in live end-to-end result with generation acceptance/fallback rate, token cost, and held-out non-regression (ideally lift). The existing lexical-diversity fixture uses author-written strings and does not validate the behavior or practical effect of the live generation path.