Skip to content

[eval] Add an accuracy and latency eval harness - #29

Open
drycode wants to merge 3 commits into
mainfrom
dy/eval-harness
Open

drycode wants to merge 3 commits into
mainfrom
dy/eval-harness

Conversation

@drycode

@drycode drycode commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Nothing measured how accurate or how fast tacet's transcription is. That left the open questions without numbers: whether to prompt, which decoder settings, which model, whether Parakeet or an LLM cleanup pass is worth building. This adds the harness, and a way to feed it your own voice, which is the input that can actually answer those questions.

What it does

eval/tacet_eval.py (standard library only):

  • Server: starts its own whisper-server on a free loopback port, with install-server.sh's flags. It never touches the live services on 8910 and 8911.
  • Requests: POSTs each clip exactly as WhisperClient.swift does.
  • Report: WER (overall, per voice or recording session, per line kind), p50/p95 latency, and how many no-speech clips came back with text, including how many of those would pass the RMS gate.
  • Pass-throughs: --prompt, --beam-size, --server-arg=…, --model, --voices, --corpus.

It takes two kinds of input:

  • The shipped synthetic corpus, a smoke test. 44 sentences spoken by macOS say in four voices, about 410 words per voice. It answers "did this flag break something" and gives server latency.
    • It is too small and too clean to decide an accuracy question: one word moves a voice's WER by about 0.25 points.
    • Its first recorded number, 4.32%, didn't reproduce: two fresh runs today agree exactly with each other, at 3.83% under the same rules.
    • say has no microphone, room or hesitation, and some of its misses are its own pronunciation. It reads "p95" aloud as "page 95".
  • Your own recorded voice, for decisions.
    • Recording: eval/record.py records a script line by line through eval/recorder.swift. That copies Recorder.swift's capture path: AVAudioEngine's default input, a 4096-frame tap, AVAudioConverter to 16 kHz mono 16-bit, the same 44-byte header.
    • Takes: they go to ~/.local/share/tacet-eval/recordings/<session>/, outside the repo, with a note beside each saying what was read, where, and how loud. A take whose script line has changed since is refused.
    • Scoring: tacet_eval.py --recordings --split tune|holdout scores them. Script lines can carry a split (tune on one, score the other once) and a kind, and [[quiet]] takes record real non-speech (breath, typing, the room), unlike synthetic digital silence.

Scoring rules worth a look

  • normalize() reads symbols as spoken words (/etc/hosts = "slash etc slash hosts"), closes hyphenated compounds and turns spelled-out numbers into digits.
  • join_split_words() forgives whisper splitting a word: "CH migrate" scores the same as ch-migrate, and "TKT 1687" the same as TKT-1687. Gluing two words together ("Ontomain") still counts.

Both are judgment calls that move the number, so compare runs only under the same rules. tests/test_eval.py pins them.

Synthetic baseline (2026-11-05, M4 Max, large-v3-turbo, production flags, empty prompt)

WER 3.53%, 58 of 1,644 words (Samantha 3.16%, Daniel 2.68%, Karen 4.62%, Moira 3.65%)
latency p50 242 ms, p95 264 ms
silence 2 of 7 returned text, both digital silence under the RMS gate

This is a regression yardstick, not tacet's real accuracy. An early experiment combined a jargon --prompt, beam 5 and --no-fallback. It moved one voice by two words, which is noise. It also turned two more silence clips into text, which is worth checking on real quiet takes.

Tests

  • 190 pass: the scoring rules, corpus and recording loading, and the recorder's decisions about what to redo and warn about. None of them need macOS or a whisper-server, so CI on Linux is unaffected.
  • recorder.swift --selftest checks the WAV writer without a microphone.

🤖 Generated with Claude Code

Nothing measured how well whisper-server transcribes tacet's kind of
dictation, so no change to the prompt, decoding or model could be judged.
eval/tacet_eval.py is that yardstick. It speaks 44 jargon-heavy
utterances in four `say` voices, converts each with afconvert to the WAV
tacet records, and starts a private whisper-server on a free loopback
port with install-server.sh's flags. It then POSTs every clip the way
WhisperClient does. It reports WER after a documented normalization,
p50/p95 latency, and how many of 7 silence clips came back with text.

Normalization reads symbols as their spoken words. A path or flag said
aloud therefore scores the same as its written form, and the number
reflects recognition rather than formatting.

Baseline on an M4 Max: WER 4.32%, p50 239 ms, p95 253 ms, silence 2/7.
Both false texts are under tacet's RMS gate, so a user never sees them.

Solves: no measurement of tacet's transcription quality
Tests: tests/test_eval.py (54) plus the existing suite, 166 passed
Whisper often writes a compound or identifier in pieces: "CH migrate"
for ch-migrate, "DRY 1687" for DRY-1687, "large V3 turbo". Pasted into
an agent prompt those read fine, yet each scored two or three errors,
so the WER measured spacing as much as recognition. Two or three
adjacent hypothesis words are now joined when the joined form is a
reference word. Gluing two real words together ("Ontomain") still
counts, and WER still divides by the reference's own word count.

The synthetic baseline drops from 3.83% to 3.53% on the same
transcripts (two fresh runs agree exactly). The 4.32% first recorded
for this corpus did not reproduce under the old rules either.

Solves: WER counting split compounds and ids as misrecognition
Tests: tests/test_eval.py spacing cases; full suite passes
The synthetic corpus is 44 sentences of text-to-speech: about 410
words per voice, with no microphone, no room and no hesitation, and
some misses are say's own pronunciation (it reads "p95" as "page 95").
It can catch a flag that breaks something; it can't settle whether a
prompt, a decoder setting or a model is better for a real speaker.

eval/record.py records a script line by line through
eval/recorder.swift, which copies Recorder.swift's capture path
(AVAudioEngine default input, 4096-frame tap, AVAudioConverter to
16 kHz mono 16-bit, canonical 44-byte header). Press Enter and talk as
with the hotkey; keep, redo or play back each take. Takes go outside
the repo, to ~/.local/share/tacet-eval/recordings/<session>/, each
with a note of what was read, on which Mac and input, and how loud.

tacet_eval.py --recordings scores them, one session per microphone and
Mac in place of a voice. Corpus lines can carry a split (tune on one,
score the other once) and a kind, reported separately, and [[quiet]]
takes record real non-speech in place of digital silence. A missing
take, or one recorded for text that has since changed, stops the run.

Solves: no way to measure tacet on a real voice and microphone
Tests: tests/test_eval.py recordings, splits, quiet takes; tests/test_record.py; recorder.swift --selftest

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant