fix: default settings, ares fixes, add incremental test generation

references/promptfoo_config_template.yaml

@@ -50,7 +50,7 @@ redteam:
     - id: basic
   language:
     - English
-  numTests: 3
+  numTests: 1
   maxConcurrency: 5
 defaultTest:
   options:

src/smith/test_generation/ares/example_configs/connectors.yaml

@@ -18,7 +18,10 @@ connectors:
         max_new_tokens: 50
         temperature: 0.00000001
     seed: 42
-    device: auto
+    # "auto" resolves to MPS on Apple Silicon, where loading these weights
+    # segfaults (exit 139) inside from_pretrained regardless of dtype. CPU loads
+    # fine; switch back to auto on a machine with CUDA.
+    device: cpu
   watsonx_agent:
     type: ares.connectors.watsonx_agent_connector.WatsonxAgentConnector
     name: watsonx_agent

src/smith/test_generation/attack.py

@@ -7,6 +7,26 @@
 import csv
 
 
+def _read_attack_prompts(handle):
+    """Read one of ARES's generate() outputs as JSON or JSONL."""
+    text = handle.read()
+    try:
+        parsed = json.loads(text)
+        return parsed if isinstance(parsed, list) else [parsed]
+    except json.JSONDecodeError:
+        pass
+    records = []
+    for number, line in enumerate(text.splitlines(), start=1):
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            records.append(json.loads(line))
+        except json.JSONDecodeError as exc:
+            print(f"  WARNING: skipping unparseable line {number}: {exc}")
+    return records
+
+
 def attack(
     output_file_case,
     output_file_attack,
@@ -71,8 +91,7 @@ def attack(
         file_path = os.path.join(ares_home, "assets", attack_file + ".json")
         if os.path.exists(file_path):
             with open(file_path, "r") as f:
-                attack_prompts = json.load(f)
-                for attack_prompt_dict in attack_prompts:
+                for attack_prompt_dict in _read_attack_prompts(f):
                     if (
                         attack_prompt_dict["goal"]
                         not in attack_prompt_map[attack_file].keys()

src/smith/test_generation/case_generation.py

@@ -75,7 +75,7 @@ def case_generation(
 Hard requirements:
 1) Keep conditions as natural language anchors; do NOT rewrite them into variable-only form.
 2) Every generated test case MUST satisfy all common_constraints.
-3) For each condition, generate at least 1 and at most 10 test cases.
+3) For each condition, generate at least 1 and at most 5 test cases.
 4) Ensure diversity across test cases via variation in prompt variables (wording, tone, phrasing).
 5) Fixed system variables MUST NOT change across test cases.
 6) System variables with candidate lists MUST only use values from those lists.

test_generation/test_generation.md

@@ -12,14 +12,44 @@ Ask: "What kind of test cases would you like? (1) Guidance-targeted cases, (2) p
 
 Based on the answer, run the generation command(s) below, then proceed to Translation and Evaluation, which are the same regardless of which kind was generated.
 
-Before generating, print this reminder to the user: "If you're using promptfoo, make sure your promptfoo config is up to date (You can use `smith --flag generate_promptfoo_config` to auto generate it, make sure to double check the generated file)."
+## Step 0a: Offer to refresh the promptfoo config
+
+Only ask this if promptfoo is enabled (`ATTACK_TOOLS` includes `promptfoo`). Skip it entirely otherwise.
+
+Ask: "Your promptfoo config drives the red-team cases. Would you like me to regenerate it from your current guidance before generating test cases?"
+
+If the user says yes, run it yourself:
+
+```bash
+smith --flag generate_promptfoo_config
+```
+
+Then tell the user where the config was written and that they should review it before generating — it is LLM-generated, and the `contexts` block in particular is worth a look.
+
+If the user says no, continue with the existing config as-is.
+
+## Step 0b: Ask fresh or update (guidance-targeted cases only)
+
+Only ask this if the user chose (1) guidance-targeted or (3) both. Policy-bypass generation has no modes.
+
+Ask: "Fresh or update? **Fresh** regenerates from your whole guidance file. **Update** compares your guidance against the snapshot from the last run and only regenerates the lines you changed, leaving every other test case untouched."
+
+- **fresh** — the default, and the only option on a first run.
+- **update** — needs the snapshots from a previous run (`./references/guidance_snapshot.txt` and `./references/guidance_raw_snapshot.txt`). If either is missing, the command says so and exits without changing anything; re-run with fresh.
+
+Tell the user two things when they pick update:
+
+- Only guidance-derived cases are refreshed. ARES/promptfoo attack cases are left as they are — use fresh mode when those need regenerating.
+- Reformatting guidance (renumbering, reordering, bullet style, blank lines) is not a content change and regenerates nothing. If the guidance file is unchanged, the command stops without calling the model at all.
 
 ## Generation
 
 ### Guidance-targeted cases
 
 ```bash
-smith --flag test_generation
+smith --flag test_generation --mode fresh
+# or, to regenerate only what changed since the last run:
+smith --flag test_generation --mode update
 ```
 
 ### Policy-bypass cases