Repository navigation
audiofeatures: a voice activity flag per slice - #336
Merged
Merged
Conversation
Each slice now carries a flag saying whether it holds speech, from the frontend's own band levels against a per-band noise floor, held 100 ms. readinto() and readinto_raw() take an optional second buffer for the flags, and speaking() is the newest one. Integer only, so CPython, MicroPython and a board agree; about 0.7 KB of flash on an ESP32-P4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every slice the
audiofeaturesfrontend makes now says whether it holds speech. Pass a second buffer toreadinto()(orreadinto_raw()), one byte a slice, and it fills with the flags;speaking()gives the newest one. A wake word detector can skip its model while nothing is said, and an intercom can hold the floor while someone is talking. The flags travel with their slices through the queue, soattach()to a tap andfeed()from Python both get them, at any rate the frontend takes.How it decides
It reads the frontend's 40 band amplitudes before noise reduction, as integer log levels (a 64th of an octave), and tracks a noise floor per band that falls fast and rises slowly. The score is the mean of the four bands furthest over their floors, averaged over 50 ms; about 10 dB over is speech, held 100 ms. All integer arithmetic, no libm, so the parity gate covers it:
features_probe.pyprints the flags, and CPython and unix MicroPython agree byte for byte.Measured against Silero VAD
97 recordings (spoken phrases, longer recordings, alert sounds, steady noise), with the phrases also mixed with loud noise and turned down 30 dB, each after two seconds of its own background. Agreement per 10 ms slice:
Fed at 48 kHz the figures match 16 kHz to 0.1%. Known misses: an alarm or doorbell reads as speech, and speech under louder noise is mostly missed. The guide section has the details.
Cost
About 0.7 KB of RISC-V text at -O2 on top of the frontend (13.07 KB to 13.75 KB linked), against about 9.6 KB for WebRTC's detector. 176 bytes of state and a byte per queued slice.
Tests
Goldens of the flags for a synthetic signal and for tone bursts, at block sizes 1, 160 and 4096, which fail when the threshold moves or the hold is removed (checked by planting both); steady noise at three levels never flagged; a noise that steps up 20 dB is learnt; digital silence then noise; 48 kHz against 16 kHz; reset.
examples/audiofeatures_voice_activity.pyprints the speech in a WAV file.