Repository navigation
Conversation
Nothing measured how well whisper-server transcribes tacet's kind of dictation, so no change to the prompt, decoding or model could be judged. eval/tacet_eval.py is that yardstick. It speaks 44 jargon-heavy utterances in four `say` voices, converts each with afconvert to the WAV tacet records, and starts a private whisper-server on a free loopback port with install-server.sh's flags. It then POSTs every clip the way WhisperClient does. It reports WER after a documented normalization, p50/p95 latency, and how many of 7 silence clips came back with text. Normalization reads symbols as their spoken words. A path or flag said aloud therefore scores the same as its written form, and the number reflects recognition rather than formatting. Baseline on an M4 Max: WER 4.32%, p50 239 ms, p95 253 ms, silence 2/7. Both false texts are under tacet's RMS gate, so a user never sees them. Solves: no measurement of tacet's transcription quality Tests: tests/test_eval.py (54) plus the existing suite, 166 passed
Whisper often writes a compound or identifier in pieces: "CH migrate"
for ch-migrate, "DRY 1687" for DRY-1687, "large V3 turbo". Pasted into
an agent prompt those read fine, yet each scored two or three errors,
so the WER measured spacing as much as recognition. Two or three
adjacent hypothesis words are now joined when the joined form is a
reference word. Gluing two real words together ("Ontomain") still
counts, and WER still divides by the reference's own word count.
The synthetic baseline drops from 3.83% to 3.53% on the same
transcripts (two fresh runs agree exactly). The 4.32% first recorded
for this corpus did not reproduce under the old rules either.
Solves: WER counting split compounds and ids as misrecognition
Tests: tests/test_eval.py spacing cases; full suite passes
The synthetic corpus is 44 sentences of text-to-speech: about 410 words per voice, with no microphone, no room and no hesitation, and some misses are say's own pronunciation (it reads "p95" as "page 95"). It can catch a flag that breaks something; it can't settle whether a prompt, a decoder setting or a model is better for a real speaker. eval/record.py records a script line by line through eval/recorder.swift, which copies Recorder.swift's capture path (AVAudioEngine default input, 4096-frame tap, AVAudioConverter to 16 kHz mono 16-bit, canonical 44-byte header). Press Enter and talk as with the hotkey; keep, redo or play back each take. Takes go outside the repo, to ~/.local/share/tacet-eval/recordings/<session>/, each with a note of what was read, on which Mac and input, and how loud. tacet_eval.py --recordings scores them, one session per microphone and Mac in place of a voice. Corpus lines can carry a split (tune on one, score the other once) and a kind, reported separately, and [[quiet]] takes record real non-speech in place of digital silence. A missing take, or one recorded for text that has since changed, stops the run. Solves: no way to measure tacet on a real voice and microphone Tests: tests/test_eval.py recordings, splits, quiet takes; tests/test_record.py; recorder.swift --selftest
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing measured how accurate or how fast tacet's transcription is. That left the open questions without numbers: whether to prompt, which decoder settings, which model, whether Parakeet or an LLM cleanup pass is worth building. This adds the harness, and a way to feed it your own voice, which is the input that can actually answer those questions.
What it does
eval/tacet_eval.py(standard library only):install-server.sh's flags. It never touches the live services on 8910 and 8911.WhisperClient.swiftdoes.--prompt,--beam-size,--server-arg=…,--model,--voices,--corpus.It takes two kinds of input:
sayin four voices, about 410 words per voice. It answers "did this flag break something" and gives server latency.sayhas no microphone, room or hesitation, and some of its misses are its own pronunciation. It reads "p95" aloud as "page 95".eval/record.pyrecords a script line by line througheval/recorder.swift. That copiesRecorder.swift's capture path: AVAudioEngine's default input, a 4096-frame tap, AVAudioConverter to 16 kHz mono 16-bit, the same 44-byte header.~/.local/share/tacet-eval/recordings/<session>/, outside the repo, with a note beside each saying what was read, where, and how loud. A take whose script line has changed since is refused.tacet_eval.py --recordings --split tune|holdoutscores them. Script lines can carry asplit(tune on one, score the other once) and akind, and[[quiet]]takes record real non-speech (breath, typing, the room), unlike synthetic digital silence.Scoring rules worth a look
normalize()reads symbols as spoken words (/etc/hosts= "slash etc slash hosts"), closes hyphenated compounds and turns spelled-out numbers into digits.join_split_words()forgives whisper splitting a word: "CH migrate" scores the same asch-migrate, and "TKT 1687" the same asTKT-1687. Gluing two words together ("Ontomain") still counts.Both are judgment calls that move the number, so compare runs only under the same rules.
tests/test_eval.pypins them.Synthetic baseline (2026-11-05, M4 Max, large-v3-turbo, production flags, empty prompt)
This is a regression yardstick, not tacet's real accuracy. An early experiment combined a jargon
--prompt, beam 5 and--no-fallback. It moved one voice by two words, which is noise. It also turned two more silence clips into text, which is worth checking on real quiet takes.Tests
recorder.swift --selftestchecks the WAV writer without a microphone.🤖 Generated with Claude Code