Expand benchmarks across classification and generative data

Author: voidstackloop

benchmarks/README.md

@@ -1,37 +1,103 @@
 # Reproducible real-dataset benchmark
 
-This benchmark downloads fixed stratified slices from `fancyzhx/ag_news` and
-`stanfordnlp/imdb`, generates disclosed dirty variants, applies BuffData's local
-validation/task-detection/exact-deduplication stages, and trains identical PyTorch
-classifiers across three fixed seeds.
+## Non-classification matrix
+
+The generative matrix complements the label-classification suites with six large,
+real Hugging Face datasets covering instruction/SFT, multi-turn chat, preference/DPO,
+extractive QA, summarization, and raw language-modeling text. It compares every
+unmodified source slice with BuffData's output, then appends 40% exact duplicates and
+10% invalid records and repeats the comparison at 10,000 and 30,000 source rows.
+
+```bash
+python benchmarks/benchmark_generative_matrix.py \
+  --scales 10000 30000 \
+  --output-dir benchmarks/results-generative
+```
+
+The generated `REPORT.md` and `results.json` include rows retained/deleted, source
+rows lost or changed, exact duplicates found and removed, invalid rows removed,
+defect leakage, cleaning-decision accuracy, deletion precision/recall/F1, exact
+output-content accuracy, clean-vs-dirty output parity, character preservation,
+per-stage rejections, and throughput. Because these datasets are open-ended rather
+than class-labeled, accuracy is defined against the clean optimized control instead
+of inventing an inapplicable classifier label. LLM quality scoring and classification
+are disabled, and provider token usage must remain zero.
+
+For downstream before → after movement on disjoint held-out rows, run:
+
+```bash
+python benchmarks/benchmark_generative_utility.py \
+  --scales 10000 30000 --eval-rows 250 \
+  --output-dir benchmarks/results-generative
+```
+
+This writes `UTILITY_REPORT.md` and `utility_results.json`. Metrics are task-specific:
+response token-F1 for instruction/chat, preference accuracy for DPO, answer exact
+match/F1 for QA, ROUGE-L for summarization, and next-token accuracy/perplexity for
+raw text. Each report shows clean raw → clean optimized and dirty raw → dirty optimized.
+
+This benchmark downloads fixed stratified slices from **19 Hugging Face datasets**,
+generates disclosed dirty variants, applies BuffData's local validation/task-detection/
+exact-deduplication stages, and trains identical PyTorch classifiers across three fixed
+seeds. The catalog spans sentiment, topic, emotion, hate/irony, question type, and
+subjectivity tasks with 2 to 20 classes.
+
+| Family | Datasets |
+|---|---|
+| Binary | `imdb`, `yelp_polarity`, `amazon_polarity`, `rotten_tomatoes`, `sst2`, `subj`, `tweet_eval_irony`, `tweet_eval_hate`, `cr`, `amazon_counterfactual` |
+| Multi-class | `ag_news`, `dbpedia_14`, `emotion`, `tweet_eval_sentiment`, `yahoo_answers_topics`, `tweet_eval_emotion`, `newsgroups_20`, `trec_coarse`, `tweet_sentiment_extraction` |
 
 ```bash
 cd ~/projects/buffdata
 source .venv/bin/activate
 python benchmarks/benchmark_buffdata.py
 ```
 
+The default run is local after the Hugging Face downloads complete; it makes no LLM
+API calls. To run a smaller slice while developing:
+
+```bash
+python benchmarks/benchmark_buffdata.py \
+  --datasets ag_news imdb tweet_eval_sentiment \
+  --train-rows 1000 --test-rows 500 --epochs 2
+```
+
 Outputs are written to `benchmarks/results/`, including the exact JSONL inputs,
-per-seed metrics in `results.json`, and a concise `REPORT.md`.
+per-seed metrics in `results.json`, and a concise `REPORT.md`. The report and JSON now
+include both model-quality metrics and explicit data-hygiene accounting:
+
+- input, retained, and deleted row counts plus deletion rate;
+- duplicate rows and duplicate groups present in the pipeline input;
+- conflicting-label duplicate rows/groups;
+- invalid rows deleted by validation and duplicates deleted by exact dedup;
+- clean source rows accidentally deleted;
+- injected defects removed versus retained, with a per-category/per-stage cross-tab;
+- unique-content and empty-text counts, plus exact rejection-reason totals.
+
+`duplicate_rows_in_input` follows BuffData's own exact-dedup equivalence rule: trimmed
+classification text, independent of label. `duplicate_rows_deleted` is the observed
+dedup-stage result, so the report distinguishes duplicates that exist from rows that
+were actually removed.
 
 For the larger Gemini-audited suite (12,000 training records per dataset):
 
 ```bash
 python benchmarks/benchmark_buffdata.py \
   --datasets dbpedia_14 yelp_polarity emotion \
   --train-rows 12000 --test-rows 2000 --epochs 6 \
-  --gemini-audit-rows 20 --output-dir benchmarks/results-large
+  --gemini-audit-rows 20 --audit-provider gemini \
+  --output-dir benchmarks/results-large
 ```
 
 `--gemini-audit-rows` uses BuffData's scalable batched-sample quality mode. Validation,
 deduplication, and task detection still process every row; Gemini judges only a
 deterministic representative sample (20 rows per structured request) and its token
 usage is recorded in the report.
 
-The local benchmark does not require an API key. After `GEMINI_API_KEY` is set,
-the provider-backed optimizer can be tested separately on generative datasets;
-it is deliberately not mixed into this classification benchmark because LLM
-scoring/refinement and deterministic validation/deduplication test different claims.
+The local benchmark does not require an API key. Setting `--gemini-audit-rows` above
+zero enables the optional provider-backed quality audit and requires the corresponding
+provider credential. Audit scoring is informational: deterministic validation and
+deduplication remain the only stages deciding the row-retention numbers.
 
 ## Accuracy regression gate
 

benchmarks/benchmark_accuracy_regression.py

@@ -20,10 +20,13 @@
 from buffdata.engine.pipeline import OptimizationPipeline
 from buffdata.models.schemas import DatasetItem, PipelineConfig
 from benchmark_buffdata import (
+    DEFECT_CLEAN,
     build_vocab,
     make_dirty,
     stratified_rows,
     summarize,
+    summarize_data_hygiene,
+    summarize_defect_breakdown,
     train_once,
     write_jsonl,
 )
@@ -65,7 +68,13 @@ async def optimize(
         {"id": item.id, "text": item.get_classification_text(), "label": int(item.labels)}
         for item in result.accepted
     ]
-    return generated, result.metrics
+    metrics = dict(result.metrics)
+    # Real per-category, per-stage accept/reject cross-tab from the pipeline's own
+    # decisions (not the injection ratios) -- how many of each injected defect type
+    # were actually caught vs. slipped through, and which stage caught them.
+    metrics["defect_breakdown"] = summarize_defect_breakdown(result.accepted, result.rejected)
+    metrics["hygiene"] = summarize_data_hygiene(rows, result.accepted, result.rejected)
+    return generated, metrics
 
 
 def evaluate(
@@ -146,6 +155,40 @@ def report(payload: dict[str, Any]) -> str:
         "",
         f"Generated optimized data improves accuracy by **{gain['accuracy'] * 100:+.2f} points** "
         f"and macro-F1 by **{gain['macro_f1'] * 100:+.2f} points**.",
+        "",
+        "### Data hygiene: how many rows were actually retained vs. lost",
+        "",
+        "Cross-tabulated against the pipeline's real per-row decisions (`item.metadata` after the "
+        "run), not the injection ratios -- 'Lost' is a row BuffData actually rejected, broken down "
+        "by the exact stage that caught it.",
+        "",
+        "| Category | Injected | Retained | Lost | Lost via validate | Lost via dedup |",
+        "|---|---:|---:|---:|---:|---:|",
+    ])
+    breakdown = value.get("defect_breakdown", {})
+    category_label = {
+        DEFECT_CLEAN: "clean (uncontaminated source)",
+        "conflicting_duplicate": "conflicting-label duplicate",
+        "class_skew_duplicate": "class-skew duplicate",
+        "empty": "empty row",
+    }
+    totals = {"input": 0, "retained": 0, "lost": 0}
+    stage_totals: dict[str, int] = {}
+    for category, label in category_label.items():
+        entry = breakdown.get(category, {"input": 0, "retained": 0, "lost": 0, "lost_by_stage": {}})
+        for key in ("input", "retained", "lost"):
+            totals[key] += entry[key]
+        for stage, count in entry["lost_by_stage"].items():
+            stage_totals[stage] = stage_totals.get(stage, 0) + count
+        lines.append(
+            f"| {label} | {entry['input']:,} | {entry['retained']:,} | {entry['lost']:,} | "
+            f"{entry['lost_by_stage'].get('validate', 0):,} | {entry['lost_by_stage'].get('dedup', 0):,} |"
+        )
+    lines.append(
+        f"| **Total** | **{totals['input']:,}** | **{totals['retained']:,}** | **{totals['lost']:,}** | "
+        f"**{stage_totals.get('validate', 0):,}** | **{stage_totals.get('dedup', 0):,}** |"
+    )
+    lines.extend([
         "",
         "## Gates",
         "",
@@ -264,6 +307,7 @@ async def main(args: argparse.Namespace) -> None:
             "conditions": value_conditions,
             "generated_minus_original": value_gain,
             "defects": defects,
+            "defect_breakdown": generated_metrics.get("defect_breakdown", {}),
             "pipeline_metrics": generated_metrics,
         },
         "gates": {