Skip to content

Tune the delivery knobs against real recordings #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MattModeCode wants to merge 1 commit into
base: feat/liquid-glass-studio
Choose a base branch
from
Open

Tune the delivery knobs against real recordings #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,21 @@ double-chin train yourvoice path/to/recordings/ --manifest path/to/recordings/ma
double-chin say "Hello, world." --voice yourvoice -o out.wav --verify
```

Then tune how that voice is *delivered*:

```bash
double-chin tune yourvoice --real-dir path/to/recordings/ --write-defaults
```

`tune` searches the four delivery controls — expression, reference adherence,
variation and speaking rate — synthesizing several takes per candidate and
scoring each set against your own recordings with the indistinguishability
gate. It prints the winning values and the run-to-run noise floor, so a tie
never gets reported as a win, and `--write-defaults` saves the winner as the
values the app opens with. In the window, **Match my voice** puts the sliders
back on that profile; **Reset to neutral** returns to the engine's own
baseline. Method and results: [docs/tuning.md](docs/tuning.md).

Every enrolled voice lives only in `~/.double-chin/voices/`; deleting that folder revokes it entirely. Don't clone a voice you don't have the right to clone — every output carries an inaudible [Perth watermark](https://github.com/resemble-ai/chatterbox#watermarking), but it's a provenance signal, not a control.

## Licences
Expand Down
48 changes: 48 additions & 0 deletions docs/design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,3 +257,51 @@ Load-bearing decisions, each verifiable in `tests/test_studio.py`:
**Gate integration:** Studio will eventually embed a gate-running UI; for now, `double-chin gate VOICE` runs the full suite from the CLI.

Known limits, honestly: single-process, single-user by design; no job cancellation (kill the server); no ASR/content check on outputs (inherited from §8 — the upgrade path stands); SSE drops don't kill a job (state is polled/replayed from `/api/jobs/{id}`) but the UI tells you to check History rather than pretending nothing happened. Every claim in this section survived a dedicated second-round red team; its findings and the code fixes are in [red-team.md](red-team.md).

## 11. Delivery tuning

The engine exposes four delivery controls — `exaggeration` (Expression),
`cfg_weight` (Reference adherence), `temperature` (Variation) and `rate`
(Speaking rate). Until now their defaults were Chatterbox's own, which are
not the values that best match any particular voice. `double-chin tune`
searches them.

Method, and the reasoning behind each choice:

- **Score with the existing gate, not a new metric.** Each candidate is
synthesized several times and the resulting *set* of clips is scored against
a set of the owner's real recordings with `indistinguishability_gate`.
- **Rank on the components that respond.** With eight real clips against three
clone clips, the gate's `discrimination` component sits at 0.0 for every
candidate — the classifier separates the sets outright, and a component that
is constant cannot order anything, it only makes the composite unreadable.
The search therefore ranks on speaker similarity, naturalness and prosody
combined with the gate's own weights, and records the full composite for
every candidate anyway. The objective is a parameter of the search
(`SweepContext.objective`), not a hardcoded assumption.
- **Repeats, and a noise floor.** Generation is stochastic, so the baseline is
re-measured several times with different seeds before the search starts. A
winner that does not beat the baseline by more than that spread is reported
as a tie, not a win.
- **Plain scripts.** The eval scripts carry no prosody markup: `*emphasis*`
adjusts exaggeration and cfg_weight per chunk and clamps at the knob's range,
which would distort exactly the candidates near the ends of the grid.
- **A held-out check.** The winner is re-scored on a second script and a
disjoint set of real clips, which is what catches a profile fitted to one
passage.
- **A resumable ledger.** Every measurement is appended to
`<home>/tuning/<voice>/ledger.jsonl`, keyed by profile, script and repeat, so
an interrupted run continues rather than restarting.

The winner is stored as `delivery.TUNED_PROFILE`, which is what Studio opens
with and what "Match my voice" restores; `GET /api/delivery` serves it (or the
user's saved `~/.double-chin/delivery.json` over it) alongside the neutral
baseline, so the numbers live in one place instead of being duplicated in the
frontend.

**Outcome on the owner's voice:** a tie for three of the four knobs — no
setting of expression, adherence or variation beat the defaults on held-out
text, so the tuned profile is the baseline. Speaking rate is the one clear
result, and it is negative: any value other than 1.00x drops speaker
similarity from ~0.66 to ~0.40, because the time-stretch is applied to
finished audio. Run, results and honest limits: [tuning.md](tuning.md).
157 changes: 157 additions & 0 deletions docs/tuning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Delivery tuning — method and results

`double-chin tune` searches the four delivery controls for the values that
make synthesized takes least distinguishable from your own recordings. This
page records the run against the owner's voice, and what it found.

**Headline: three of the four knobs came back a tie, and the fourth has a
large, one-sided result.** Expression, reference adherence and variation could
not be shown to beat the engine's own defaults on held-out text — every
difference sat inside run-to-run noise. Speaking rate is different: any value
other than 1.00× measurably damages the voice.

## Setup

| | |
|---|---|
| Voice | `owner` (LoRA fine-tuned, 20 s reference) |
| Machine | M-series Mac, MPS, 2026-09-09 |
| Real clips | 8 recordings for the search, 6 disjoint ones for the held-out check |
| Scripts | two plain-prose passages, ~10 s of audio each |
| Takes per candidate | 3, deterministic seeds derived from the candidate key |
| Candidates measured | 24 on the tuning script, plus 2 held-out checks |
| Cost | ~5.5 min per candidate (3 syntheses + one gate run) |

Recordings 001–010 are excluded from both sets: `reference.wav` is built from
the start of the corpus and caps at 20 s, so scoring against them would compare
the clone to its own conditioning audio.

The eval scripts carry no prosody markup. `*emphasis*` raises exaggeration and
lowers cfg_weight for the chunk it spans and clamps at the knob's range, which
would distort exactly the candidates near the ends of the grid. Marks layer on
top of whatever profile wins; they are not part of what is being measured.

## What is being maximized

Each candidate's three takes are scored as a *set* against the real clips with
`verification.gate.indistinguishability_gate`. The gate's own composite turns
out to be unusable as a ranking signal here: its `discrimination` component is
**0.000 for every candidate**, because with eight real clips against three
clone clips the classifier separates the two sets outright. A weighted
geometric mean with a zero in it collapses — every candidate scores ~0.026 —
and a component that never varies cannot order anything.

So the search ranks on the three components that do respond, using the gate's
own weights:

```
objective = weighted geometric mean of
speaker_similarity (0.30), naturalness (0.25), prosody (0.20)
```

The full gate result is still recorded for every candidate. The objective is a
parameter of the search (`SweepContext.objective`), not a hidden assumption.

## Noise floor

The baseline profile was measured three times with different seeds before the
search started:

| repeat | objective | speaker | prosody |
|---|---|---|---|
| v0 | 0.7777 | 0.637 | 0.766 |
| v1 | 0.7698 | 0.632 | 0.747 |
| v2 | 0.7943 | 0.673 | 0.763 |

**mean 0.781, sd 0.012.** A candidate has to beat 0.781 by more than 0.012 to
be worth anything.

## Results

Best candidates on the tuning script (all at rate 1.00×):

| exaggeration | cfg_weight | temperature | objective | speaker | prosody |
|---|---|---|---|---|---|
| 0.50 | 0.80 | 0.45 | **0.8018** | 0.668 | 0.800 |
| 0.50 | 0.20 | 0.80 | 0.7991 | 0.664 | 0.798 |
| 0.50 | 0.80 | 0.50 | 0.7942 | 0.657 | 0.792 |
| 0.50 | 0.80 | 0.80 | 0.7881 | 0.656 | 0.771 |
| 0.50 | 0.50 | 0.80 | 0.7806 (baseline) | 0.647 | 0.759 |
| 0.50 | 0.80 | 0.65 | 0.7625 | 0.621 | 0.740 |

Every rate-1.00× candidate measured, best to worst, spans 0.7625 to 0.8018 — a
range of 0.039, or about three standard deviations of the noise floor, across
the entire grid. Individual differences are one to one and a half sd.

The search's winner (0.50 / 0.80 / 0.45 / 1.00×) beat the baseline by +0.021 on
the tuning script, which clears the noise floor. It then **failed the held-out
check**:

| profile | tuning script | held-out script + held-out clips |
|---|---|---|
| winner 0.50 / 0.80 / 0.45 | 0.8018 | 0.8104 |
| baseline 0.50 / 0.50 / 0.80 | 0.7806 | **0.8213** |

On unseen text and unseen real clips the baseline scores *higher* than the
winner. The +0.021 was fitted to the tuning passage. Note also that the
baseline moved 0.781 → 0.821 between conditions: the variance between
script/clip-set conditions (~0.04) is larger than any parameter effect measured
within one (~0.02). This measurement cannot resolve a winner among those three
knobs, and reporting one anyway would be reporting noise.

### Speaking rate is the exception

| rate | objective | speaker similarity |
|---|---|---|
| 1.00× | 0.7942 | 0.657 |
| 0.95× | 0.6556 | 0.421 |
| 1.05× | 0.6182 | 0.403 |
| 1.10× | 0.6483 | 0.389 |

Every departure from 1.00× costs 0.13–0.18 objective — roughly ten standard
deviations — and speaker similarity falls from ~0.66 to ~0.40. That is the
expected consequence of how the knob works: rate is a pitch-preserving
time-stretch applied to *finished* audio (`engine.synthesize`), so it resamples
away formant and timing detail the speaker embedding relies on. It is a real
control for pacing, and it is the wrong tool for sounding more like yourself.

## Conclusion

`delivery.TUNED_PROFILE` is set to **0.50 / 0.50 / 0.80 / 1.00×** — identical to
the neutral baseline, because nothing beat it out of sample. Studio opens on
that profile, "Match my voice" restores it, and "Reset to neutral" goes to the
same place today. The two buttons diverge as soon as a run on a different
corpus writes a different profile with `tune --write-defaults`.

The plausible reading of this result: the LoRA fine-tune is already doing the
voice matching, and the delivery knobs have little headroom left on top of it.
That is a finding about this voice, not a defect in the search.

## Limits

- 24 candidates, one coordinate pass plus refinement — a fuller search might
find a real effect, though the noise floor sets a hard limit on how small an
effect this method can resolve at all.
- Three takes per candidate. More repeats would shrink the noise floor; the
cost is linear.
- ~10 s clips. Longer clips give the prosody metric more to work with and cost
proportionally more.
- The gate composite is unreadable in this configuration (~0.026 for
everything) because of the collapsed `discrimination` component. Compare
candidates on the objective, and use `double-chin gate` — which uses a
balanced set — for an absolute verdict on a voice.
- The real clips were part of the fine-tune's training corpus, so the absolute
similarity numbers are optimistic. This affects every candidate equally, so
it does not bias the ranking.

## Re-running

```bash
double-chin tune owner --real-dir Recording-scripts-audio --takes 3 --budget 120
```

Add `--write-defaults` to save the winner to `~/.double-chin/delivery.json`,
which is what Studio then opens with. Measurements are appended to
`~/.double-chin/tuning/<voice>/ledger.jsonl` and reused on the next run, so an
interrupted sweep continues where it stopped and re-running costs only the
candidates it has not seen.
119 changes: 118 additions & 1 deletion src/double_chin/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# ahead of these, so `double-chin verify` (arbitrary wav files, no storage) and
# argparse-level exits (--help, unknown/missing command) never trigger it.
_STORAGE_COMMANDS = frozenset(
{"enroll", "say", "train", "voices", "doctor", "studio", "app", "gate"}
{"enroll", "say", "train", "voices", "doctor", "studio", "app", "gate", "tune"}
)

# Default script synthesized for `double-chin gate VOICE` when no clone clips are
Expand Down Expand Up @@ -142,6 +142,47 @@ def build_parser() -> argparse.ArgumentParser:
"--json", action="store_true", help="Emit the raw result dict as JSON."
)

tune_parser = subparsers.add_parser(
"tune",
help=(
"Search the four delivery knobs for the values that score best "
"against your own recordings, and print the winner."
),
)
tune_parser.add_argument("name", help="Name of an already-enrolled voice.")
tune_parser.add_argument(
"--real-dir", type=Path, required=True,
help="Directory of genuine recordings to score against.",
)
tune_parser.add_argument(
"--takes", type=int, default=3,
help="Clips synthesized per candidate (default: 3).",
)
tune_parser.add_argument(
"--budget", type=int, default=240,
help=(
"Stop after this many syntheses and report the best so far. "
"Counted across resumes, since the ledger persists."
),
)
tune_parser.add_argument(
"--passes", type=int, default=2, help="Coordinate-descent passes (default: 2)."
)
tune_parser.add_argument(
"--work-dir", type=Path, default=None,
help="Where takes and the resumable ledger live (default: <home>/tuning).",
)
tune_parser.add_argument(
"--write-defaults", action="store_true",
help="Save the winner as the defaults Studio opens with.",
)
tune_parser.add_argument(
"--device", default=None, help="Force a device (mps/cuda/cpu)."
)
tune_parser.add_argument(
"--json", action="store_true", help="Emit the full result as JSON."
)

subparsers.add_parser("doctor", help="Report environment diagnostics.")

studio_parser = subparsers.add_parser(
Expand Down Expand Up @@ -355,6 +396,81 @@ def _cmd_gate(args: argparse.Namespace) -> int:
return 0 if result["passed"] else 1


def _cmd_tune(args: argparse.Namespace) -> int:
import json

from double_chin.config import double_chin_home
from double_chin.delivery import NEUTRAL_PROFILE, save_defaults
from double_chin.tuning import run_sweep, score
from double_chin.tuning.runner import (
build_context,
delivery_objective,
format_report,
prepare_sets,
)
from double_chin.tuning.scripts import TUNING_SCRIPTS
from double_chin.voices import get_voice

voice = get_voice(args.name)
work_dir = args.work_dir or (double_chin_home() / "tuning" / voice.name)

print(f"double-chin: preparing real clips from {args.real_dir}", file=sys.stderr)
tuning_clips, holdout_clips = prepare_sets(args.real_dir, work_dir)

def on_event(kind: str, data: dict) -> None:
if kind == "scored":
values = " ".join(f"{k}={v:g}" for k, v in data["profile"].items())
print(
f" obj {delivery_objective(data):.4f} gate {data['score']:.3f} "
f"{values}",
file=sys.stderr,
)

ctx = build_context(
voice=voice,
real_clips=tuning_clips,
scripts=TUNING_SCRIPTS,
work_dir=work_dir,
takes=args.takes,
budget=args.budget,
device=args.device,
on_event=on_event,
)
result = run_sweep(ctx, start=NEUTRAL_PROFILE, script_id="tuning", passes=args.passes)

# Re-score winner and baseline on the held-out script and the held-out
# real clips: the only check that the winner is not overfit to one passage.
ctx.real_clips = holdout_clips
ctx.budget = None
holdout = {
"best": delivery_objective(score(ctx, result.best, "holdout")),
"baseline": delivery_objective(score(ctx, result.baseline, "holdout")),
}

if args.json:
print(json.dumps(
{
"best": result.best.as_dict(),
"best_score": result.best_score,
"baseline": result.baseline.as_dict(),
"baseline_score": result.baseline_score,
"noise_floor": result.noise_floor,
"beats_noise": result.beats_noise,
"holdout": holdout,
"synthesis_count": result.synthesis_count,
},
indent=2,
))
else:
print(format_report(result, holdout))

if args.write_defaults:
path = save_defaults(result.best)
print(f"\nSaved as the opening defaults: {path}")

return 0


def _which_in(directories: list[str], name: str) -> str | None:
for directory in directories:
candidate = Path(directory) / name
Expand Down Expand Up @@ -453,6 +569,7 @@ def _cmd_app(args: argparse.Namespace) -> int:
"voices": _cmd_voices,
"verify": _cmd_verify,
"gate": _cmd_gate,
"tune": _cmd_tune,
"doctor": _cmd_doctor,
"studio": _cmd_studio,
"app": _cmd_app,
Expand Down
Loading