diff --git a/ROADMAP.md b/ROADMAP.md index 5d2dda4..db51b91 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -15,10 +15,6 @@ audio pump holding up under live use such as a stompbox or a stage rig. - A level ramp on `audiomixer` voices: a level move ramps over a set number of frames instead of stepping, so an effect can fade its wet path in or out without a click when you switch it in mid-note. -- Voice activity in `audiofeatures`: a flag per slice saying whether it holds - speech, from the frontend's own noise estimate, so a wake word model need - only run while someone is talking. The frontend and resampler take about - 16 KB of flash on an ESP32-P4 today; this should add little to it. Bugs, and things you need that don't work yet, go to [issues](https://github.com/PyDevices/audiodsp/issues). diff --git a/docs/guides/speech-features.md b/docs/guides/speech-features.md index dcc64e5..86a6ecf 100644 --- a/docs/guides/speech-features.md +++ b/docs/guides/speech-features.md @@ -48,6 +48,85 @@ are the same as at 16 kHz and the mean difference is 0.27 of a step, and three microWakeWord models detected and missed exactly the same clips, with peak probabilities within 0.03. +## Is anyone speaking? + +Each slice also carries a voice activity flag: 1 when it holds speech, 0 +when it holds silence or steady noise. Pass a second buffer, one byte a +slice, and `readinto()` fills it beside the features; `speaking()` is the +newest slice's flag on its own. + +```python +voice = bytearray(10) +n = fe.readinto(buf, voice) # voice[i] is slice i's flag +if fe.speaking(): + ... # someone is talking right now +``` + +A wake word detector can skip the model while every flag is 0, and an +intercom can hold the floor while `speaking()` is true. +[`examples/audiofeatures_voice_activity.py`](../../examples/audiofeatures_voice_activity.py) +prints where the speech is in a WAV file. `readinto_raw()` takes the same +second buffer. Attached to a tap or fed from Python, at any rate, the flags +come with their slices through the same queue. + +### How it decides + +The detector reads the frontend's 40 band levels before noise reduction, +in steps of a 64th of an octave, and keeps a noise floor for each band. +The floor follows the quiet parts quickly and rises slowly, so a fan or +road noise that stays on is learnt within a few seconds and stops counting. +A slice's score is how far its four loudest bands stand above their floors, +averaged over the last 50 ms; over about 10 dB is speech. A flag stays on +for 100 ms after the score drops, so the gaps between words don't break a +sentence into pieces. All of it is integer arithmetic, so CPython, +MicroPython and a board give the same flags for the same audio. + +The first 160 ms after the frontend starts or `reset()` is spent learning +the floor and is never flagged, so reset it in quiet, not mid-word. +Digital silence (a stream of zeros) is left out of the floor, so it doesn't +make the noise that follows it look like speech. + +### How well it works + +We scored it against Silero VAD, a neural voice detector, on 97 +recordings: 72 spoken phrases, 23 longer recordings and sounds, and two +of steady noise, with the phrases also mixed with loud noise and turned +down 30 dB. Each clip had two seconds of its own +background first, as an always-on device would hear. The fraction of +10 ms slices on which the two agree: + +| Audio | Agreement | Speech found | Noise called speech | +| --- | --- | --- | --- | +| Clean spoken phrases | 92% | 99.7% | 14% | +| Phrases with noise 10 dB below the speech | 97% | 96% | 1.7% | +| Phrases with noise as loud as the speech | 90% | 78% | 0.3% | +| Phrases 30 dB quieter | 97% | 99% | 4% | +| Longer recordings | 92% | 97% | 28% | +| Short spoken replies and alert sounds | 80% | 100% | 28% | +| 40 seconds of steady noise | | | 0.3% | + +Most of the "noise called speech" on clean audio is the 100 ms hold +running past where Silero stops. Speech starts within 30 ms of Silero's +start. Fed at 48 kHz instead, every figure in the table is the same to the +nearest 0.1%. + +On the same clips WebRTC's voice detector at its most aggressive setting +agreed 96% on clean phrases, but called the noisy mixes speech throughout +(55% and 42%) and missed most of the quiet phrases (62%), which is why +this one is built on the frontend's own bands instead. + +What it gets wrong: a tone that comes and goes, like an alarm or a +doorbell, looks like speech, because nothing here knows what a voice +sounds like. Speech buried under noise louder than itself is mostly +missed. If you need either, run a model on the slices it flags. + +### What it costs + +About 0.7 KB of flash on an ESP32-P4 on top of the frontend, 176 bytes of +state, and a byte for each slice in the queue. Nothing to set up and no +settings: if you want a longer hold, keep a flag on in Python for as many +slices as you like. + ## The queue Slices wait in a queue of `slices` (100 by default, a second of audio) until @@ -56,8 +135,8 @@ counts it, so a reader that falls behind loses old audio, never new. `reset()` forgets the audio and the queue, for the start of a new utterance. A frontend takes about 17 to 26 KB of heap with the default queue, depending -on the rate: 8 KB is the queue, the rest the frontend's state and the -resampler's filter. +on the rate: 8 KB is the queue and its voice flags, the rest the frontend's +state and the resampler's filter. CircuitPython doesn't get `audiofeatures`; see [additions](../how-it-works/additions.md). diff --git a/docs/reference/modules.md b/docs/reference/modules.md index f9b0cbd..ce27f7d 100644 --- a/docs/reference/modules.md +++ b/docs/reference/modules.md @@ -25,7 +25,7 @@ It adds modules CircuitPython doesn't have. Each has a page here or a guide: | `audiograph` | A whole patch rendered by one C call per block | [reference](sources-and-synthesis.md#audiograph), [how it works](../how-it-works/audiograph.md) | | `audiosampler` | Recorded notes played at any pitch, looped, under an envelope | [guide](../guides/play-recorded-notes.md) | | `audiometer` | Band levels for a spectrum meter | [guide](../guides/meter-levels.md) | -| `audiofeatures` | Speech features for a wake word model | [guide](../guides/speech-features.md) | +| `audiofeatures` | Speech features and voice activity for a wake word model | [guide](../guides/speech-features.md) | | `audiovorbis` | Ogg Vorbis files | [guide](../guides/decode-vorbis.md), [C interface](vorbis.md) | | `audiopump` | The audio pull, off the interpreter thread | [guide](../guides/run-the-audio-pump.md), [reference](audiopump.md) | diff --git a/examples/audiofeatures_voice_activity.py b/examples/audiofeatures_voice_activity.py new file mode 100644 index 0000000..e3dcaf5 --- /dev/null +++ b/examples/audiofeatures_voice_activity.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: 2026 Brad Barnett +# +# SPDX-License-Identifier: MIT +"""Find where someone is speaking in a WAV file. + +An audiofeatures Frontend flags every 10 ms slice it makes as speech or not, +beside the features a wake word model reads. The program feeds it a WAV file +a block at a time, reads the flags, and prints each stretch of speech with +its start and end time, so a file of a few phrases prints a line for each. + +Needs: audiodsp, built into MicroPython or installed on CPython with +pip install pydevices-audiodsp, and a 16-bit PCM WAV file at 16 kHz or +higher, mono or stereo, that starts with a moment of quiet. + +Run it: + micropython audiofeatures_voice_activity.py + +Tested on: the unix port of MicroPython and CPython 3.12 on Linux. +""" + +import struct + +import audiofeatures + +# Change these to match your setup. +FILENAME = "speech.wav" +BLOCK = 480 # frames read and fed at a time + +# Walk the WAV's chunks to its format and the start of its samples. +f = open(FILENAME, "rb") +assert f.read(12)[8:] == b"WAVE", "not a WAV file" +while True: + tag, size = struct.unpack("<4sI", f.read(8)) + if tag == b"data": + break + body = f.read(size + (size & 1)) + if tag == b"fmt ": + kind, channels, rate = struct.unpack(" 0: + block = f.read(min(left, BLOCK * 2 * channels)) + left -= len(block) + frontend.feed(block, rate, channels) + count = frontend.readinto(features, voice) + for i in range(count): + voiced += voice[i] + if voice[i] and start is None: + start = slices + i + elif not voice[i] and start is not None: + print("speech from %.2f s to %.2f s" % (start / 100, (slices + i) / 100)) + start = None + slices += count +f.close() + +# Speech still going at the end of the file ends with it. +if start is not None: + print("speech from %.2f s to the end" % (start / 100)) +print("%.2f s of speech in %.2f s of audio" % (voiced / 100, slices / 100)) diff --git a/src/audiofeatures/module.c b/src/audiofeatures/module.c index 4dfb662..826f63d 100644 --- a/src/audiofeatures/module.c +++ b/src/audiofeatures/module.c @@ -5,6 +5,8 @@ // fe.feed(pcm, 48000, channel_count=2) # from Python, anywhere // fe.attach(tap, 48000) # an audiopump.Tap, read in C // n = fe.readinto(buf) # whole 10 ms slices of 40 int8 +// n = fe.readinto(buf, voice) # and a byte each: 1 for speech +// fe.speaking() # the newest slice holds speech // // As with audiometer.Meter, the two ways in differ in where the work runs. // feed() runs it in the caller. A tap is drained in C each time readinto() @@ -141,29 +143,45 @@ static void audiofeatures_drain(audiofeatures_frontend_obj_t *self, uint32_t wan } } -static mp_obj_t audiofeatures_read(mp_obj_t self_in, mp_obj_t buf_in, bool raw) { - audiofeatures_frontend_obj_t *self = audiofeatures_get(self_in); +static mp_obj_t audiofeatures_read(size_t n_args, const mp_obj_t *args, bool raw) { + audiofeatures_frontend_obj_t *self = audiofeatures_get(args[0]); mp_buffer_info_t buf; - mp_get_buffer_raise(buf_in, &buf, MP_BUFFER_WRITE); + mp_get_buffer_raise(args[1], &buf, MP_BUFFER_WRITE); size_t width = raw ? 2 * AUDIODSP_FEATURES_SIZE : AUDIODSP_FEATURES_SIZE; size_t max = buf.len / width; + uint8_t *voice = NULL; + if (n_args > 2 && args[2] != mp_const_none) { + mp_buffer_info_t vb; + mp_get_buffer_raise(args[2], &vb, MP_BUFFER_WRITE); + voice = (uint8_t *)vb.buf; + if (vb.len < max) { + max = vb.len; + } + } audiofeatures_drain(self, max > UINT32_MAX ? UINT32_MAX : (uint32_t)max); - size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)buf.buf, max) - : audiodsp_features_read_s8(self->f, (int8_t *)buf.buf, max); + size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)buf.buf, max, voice) + : audiodsp_features_read_s8(self->f, (int8_t *)buf.buf, max, voice); return MP_OBJ_NEW_SMALL_INT(n); } -// readinto(buf) -> slices: 40 int8 each, ready for a model -static mp_obj_t audiofeatures_frontend_readinto(mp_obj_t self_in, mp_obj_t buf_in) { - return audiofeatures_read(self_in, buf_in, false); +// readinto(buf, voice=None) -> slices: 40 int8 each, ready for a model, and +// a byte each into voice, 1 when the slice holds speech +static mp_obj_t audiofeatures_frontend_readinto(size_t n_args, const mp_obj_t *args) { + return audiofeatures_read(n_args, args, false); +} +static MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(audiofeatures_frontend_readinto_obj, 2, 3, audiofeatures_frontend_readinto); + +// readinto_raw(buf, voice=None) -> slices: 40 uint16 each, the frontend's own values +static mp_obj_t audiofeatures_frontend_readinto_raw(size_t n_args, const mp_obj_t *args) { + return audiofeatures_read(n_args, args, true); } -static MP_DEFINE_CONST_FUN_OBJ_2(audiofeatures_frontend_readinto_obj, audiofeatures_frontend_readinto); +static MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(audiofeatures_frontend_readinto_raw_obj, 2, 3, audiofeatures_frontend_readinto_raw); -// readinto_raw(buf) -> slices: 40 uint16 each, the frontend's own values -static mp_obj_t audiofeatures_frontend_readinto_raw(mp_obj_t self_in, mp_obj_t buf_in) { - return audiofeatures_read(self_in, buf_in, true); +// speaking() -> whether the newest slice made holds speech +static mp_obj_t audiofeatures_frontend_speaking(mp_obj_t self_in) { + return mp_obj_new_bool(audiodsp_features_speaking(audiofeatures_get(self_in)->f)); } -static MP_DEFINE_CONST_FUN_OBJ_2(audiofeatures_frontend_readinto_raw_obj, audiofeatures_frontend_readinto_raw); +static MP_DEFINE_CONST_FUN_OBJ_1(audiofeatures_frontend_speaking_obj, audiofeatures_frontend_speaking); static mp_obj_t audiofeatures_frontend_reset(mp_obj_t self_in) { audiofeatures_frontend_obj_t *self = audiofeatures_get(self_in); @@ -223,6 +241,7 @@ static const mp_rom_map_elem_t audiofeatures_frontend_locals_table[] = { { MP_ROM_QSTR(MP_QSTR_detach), MP_ROM_PTR(&audiofeatures_frontend_detach_obj) }, { MP_ROM_QSTR(MP_QSTR_readinto), MP_ROM_PTR(&audiofeatures_frontend_readinto_obj) }, { MP_ROM_QSTR(MP_QSTR_readinto_raw), MP_ROM_PTR(&audiofeatures_frontend_readinto_raw_obj) }, + { MP_ROM_QSTR(MP_QSTR_speaking), MP_ROM_PTR(&audiofeatures_frontend_speaking_obj) }, { MP_ROM_QSTR(MP_QSTR_reset), MP_ROM_PTR(&audiofeatures_frontend_reset_obj) }, { MP_ROM_QSTR(MP_QSTR_stats), MP_ROM_PTR(&audiofeatures_frontend_stats_obj) }, { MP_ROM_QSTR(MP_QSTR_deinit), MP_ROM_PTR(&audiofeatures_frontend_deinit_obj) }, diff --git a/src/cpython/_audiodsp.c b/src/cpython/_audiodsp.c index a50dda1..f30d635 100644 --- a/src/cpython/_audiodsp.c +++ b/src/cpython/_audiodsp.c @@ -3458,19 +3458,39 @@ static PyObject *features_state_feed(audiodsp_features_object_t *self, PyObject return PyLong_FromUnsignedLong(audiodsp_features_waiting(self->f)); } -// read(buf, raw): whole slices into a writable buffer, int8 or (raw) uint16 +// read(buf, raw, voice=None): whole slices into a writable buffer, int8 or +// (raw) uint16, and a byte a slice into voice when it is given static PyObject *features_state_read(audiodsp_features_object_t *self, PyObject *args) { - Py_buffer out = {0}; + Py_buffer out = {0}, vb = {0}; int raw; - if (!PyArg_ParseTuple(args, "w*p:read", &out, &raw)) return NULL; + PyObject *voice_obj = Py_None; + if (!PyArg_ParseTuple(args, "w*p|O:read", &out, &raw, &voice_obj)) return NULL; size_t width = raw ? 2 * AUDIODSP_FEATURES_SIZE : AUDIODSP_FEATURES_SIZE; size_t max = (size_t)out.len / width; - size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)out.buf, max) - : audiodsp_features_read_s8(self->f, (int8_t *)out.buf, max); + uint8_t *voice = NULL; + if (voice_obj != Py_None) { + if (PyObject_GetBuffer(voice_obj, &vb, PyBUF_WRITABLE) != 0) { + PyBuffer_Release(&out); + return NULL; + } + voice = (uint8_t *)vb.buf; + if ((size_t)vb.len < max) { + max = (size_t)vb.len; + } + } + size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)out.buf, max, voice) + : audiodsp_features_read_s8(self->f, (int8_t *)out.buf, max, voice); PyBuffer_Release(&out); + if (voice != NULL) { + PyBuffer_Release(&vb); + } return PyLong_FromSize_t(n); } +static PyObject *features_state_speaking(audiodsp_features_object_t *self, PyObject *unused) { + return PyBool_FromLong(audiodsp_features_speaking(self->f)); +} + static PyObject *features_state_reset(audiodsp_features_object_t *self, PyObject *unused) { audiodsp_features_reset(self->f); Py_RETURN_NONE; @@ -3487,11 +3507,13 @@ static PyObject *features_state_stats(audiodsp_features_object_t *self, PyObject AUDIODSP_LOCKED_ARGS(features_state_feed, audiodsp_features_object_t) AUDIODSP_LOCKED_ARGS(features_state_read, audiodsp_features_object_t) AUDIODSP_LOCKED_ARGS(features_state_reset, audiodsp_features_object_t) +AUDIODSP_LOCKED_ARGS(features_state_speaking, audiodsp_features_object_t) AUDIODSP_LOCKED_ARGS(features_state_stats, audiodsp_features_object_t) static PyMethodDef features_state_methods[] = { {"feed", (PyCFunction)features_state_feed_locked, METH_VARARGS, NULL}, {"read", (PyCFunction)features_state_read_locked, METH_VARARGS, NULL}, {"reset", (PyCFunction)features_state_reset_locked, METH_NOARGS, NULL}, + {"speaking", (PyCFunction)features_state_speaking_locked, METH_NOARGS, NULL}, {"stats", (PyCFunction)features_state_stats_locked, METH_NOARGS, NULL}, {NULL, NULL, 0, NULL}, }; diff --git a/src/cpython/audiofeatures.py b/src/cpython/audiofeatures.py index 583619b..5e4aed5 100644 --- a/src/cpython/audiofeatures.py +++ b/src/cpython/audiofeatures.py @@ -11,12 +11,18 @@ fe.feed(pcm, 48000, 2) # interleaved s16, any rate from 16 kHz buf = bytearray(40 * 10) n = fe.readinto(buf) # n whole slices of 40 int8 + voice = bytearray(10) + n = fe.readinto(buf, voice) # and voice[i] is 1 where slice i holds speech Audio that isn't 16 kHz is resampled to 16 kHz first, through a low-pass that is flat to 7.5 kHz; 16 kHz audio goes to the frontend untouched. Each 10 ms makes a slice, which waits in a queue of ``slices`` until it is read. When the queue is full the oldest slice is dropped, and ``stats()`` counts it. +Every slice also carries a voice activity flag, from the frontend's own +filterbank: ``readinto(buf, voice)`` puts one byte a slice into ``voice``, +and ``speaking()`` says whether the newest slice holds speech. + On a board the frontend can also listen without Python touching a sample: ``attach(tap, sample_rate)`` reads an ``audiopump.Tap`` in C. There is no pump to attach to on CPython, so ``attach()`` raises there. @@ -66,13 +72,21 @@ def attach(self, source, sample_rate=None): def detach(self): self._live() - def readinto(self, buf): - """Whole slices of 40 int8 into ``buf``, oldest first. Returns how many.""" - return self._live().read(buf, False) + def readinto(self, buf, voice=None): + """Whole slices of 40 int8 into ``buf``, oldest first. Returns how many. + + Given ``voice``, a writable buffer, each slice's voice flag goes into + it too, a byte each: 1 when the slice holds speech, 0 when not. + """ + return self._live().read(buf, False, voice) - def readinto_raw(self, buf): + def readinto_raw(self, buf, voice=None): """Whole slices of 40 uint16 (the frontend's own values) into ``buf``. Returns how many.""" - return self._live().read(buf, True) + return self._live().read(buf, True, voice) + + def speaking(self): + """Whether the newest slice made holds speech, read or not.""" + return self._live().speaking() def reset(self): """Forget the audio so far and every queued slice.""" diff --git a/src/shared/audiodsp_features.c b/src/shared/audiodsp_features.c index 8bca11c..cfc6212 100644 --- a/src/shared/audiodsp_features.c +++ b/src/shared/audiodsp_features.c @@ -14,6 +14,7 @@ #include "shared/audiodsp_microfrontend_names.h" #include "tensorflow/lite/experimental/microfrontend/lib/frontend.h" #include "tensorflow/lite/experimental/microfrontend/lib/frontend_util.h" +#include "tensorflow/lite/experimental/microfrontend/lib/bits.h" // --- the resampler ----------------------------------------------------------- // @@ -210,6 +211,134 @@ static const float *rs_row(rs_t *rs) { return rs->row; } +// --- voice activity ---------------------------------------------------------- +// +// Each slice gets a flag saying whether it holds speech, worked out from the +// filterbank's own band amplitudes, before noise reduction touches them. +// Everything is integer arithmetic, so every target flags the same slices. +// +// Per band, the level is log2 of the amplitude in 1/64 octave (0.094 dB) steps, +// and a noise floor follows it: down 2 steps a slice when the level is under +// it, up 1/4 step a slice when it is over. That holds the floor near the +// quietest tenth of what the band hears, so speech, which comes and goes, +// barely moves it, and a steady noise becomes the floor in a few seconds. A +// band quieter than any real microphone (digital silence between files, say) +// leaves its floor alone. +// +// The floor starts as the mean of the first VAD_PRIME slices, which are +// never voice. A slice's score is the mean, over the VAD_TOP bands furthest above their +// floors, of how far above they are, averaged over the last VAD_SMOOTH +// slices. Voice is a score of VAD_ON or more, and it holds for VAD_HOLD slices +// after the score drops, so a word's tail and the gaps between words stay in. +// While the flag is up the floor rises more slowly, and then faster the longer +// it stays up, so a noise that steps up and stays is taken in within a few +// seconds instead of reading as speech forever. + +#define VAD_TOP 4 // bands scored +#define VAD_ON 110 // score for speech: about 10 dB above the floor +#define VAD_CLIP 192 // a band counts at most 18 dB above its floor +#define VAD_SMOOTH 5 // slices the score is averaged over +#define VAD_HOLD 10 // slices voice holds after the score drops +#define VAD_LEVEL_MIN 200 // a level under this (about -80 dBFS) is no news +#define VAD_FALL 32 // floor steps, in 1/16 of a level step a slice +#define VAD_RISE 4 +#define VAD_RISE_VOICE 2 +#define VAD_RISE_SHIFT 4 // the rise grows by one every 2^4 slices of voice +#define VAD_PRIME 16 // slices whose mean starts the floor, never voice + +typedef struct { + int32_t floor[AUDIODSP_FEATURES_SIZE]; // each band's noise floor, 1/1024 octave + int16_t recent[VAD_SMOOTH]; // the last slices' scores + uint16_t run; // slices voiced in a row + uint8_t at, hold, voice; + uint8_t primed; // slices heard, up to VAD_PRIME +} audiodsp_vad_t; + +// log2(x) in 1/64 octave: the top bit's place, then six bits of mantissa, +// read as linear. 0 and 1 both give 0. +static int32_t vad_level(uint32_t x) { + if (x < 2) { + return 0; + } + int m = MostSignificantBit32(x) - 1; + uint32_t frac = m >= 6 ? (x >> (m - 6)) & 63 : (x << (6 - m)) & 63; + return m * 64 + (int32_t)frac; +} + +static void vad_reset(audiodsp_vad_t *v) { + memset(v, 0, sizeof(*v)); +} + +static uint8_t vad_slice(audiodsp_vad_t *v, const uint32_t *amp) { + int32_t level[AUDIODSP_FEATURES_SIZE], above[AUDIODSP_FEATURES_SIZE]; + for (int b = 0; b < AUDIODSP_FEATURES_SIZE; b++) { + level[b] = vad_level(amp[b]); + } + if (v->primed < VAD_PRIME) { + // The floor starts as the mean of the first slices: one slice alone + // could catch a band low and read the next ones as voice. + int32_t n = v->primed; + for (int b = 0; b < AUDIODSP_FEATURES_SIZE; b++) { + int32_t l = level[b] < VAD_LEVEL_MIN ? VAD_LEVEL_MIN : level[b]; + v->floor[b] = (v->floor[b] * n + (l << 4)) / (n + 1); + } + v->primed++; + v->voice = 0; + return 0; + } + for (int b = 0; b < AUDIODSP_FEATURES_SIZE; b++) { + int32_t d = level[b] - (v->floor[b] >> 4); + above[b] = d < 0 ? 0 : d > VAD_CLIP ? VAD_CLIP : d; + } + // The mean of the VAD_TOP largest, by selection. + int32_t sum = 0; + for (int i = 0; i < VAD_TOP; i++) { + int best = i; + for (int j = i + 1; j < AUDIODSP_FEATURES_SIZE; j++) { + if (above[j] > above[best]) { + best = j; + } + } + int32_t t = above[i]; + above[i] = above[best]; + above[best] = t; + sum += above[i]; + } + v->recent[v->at] = (int16_t)(sum / VAD_TOP); + v->at = (uint8_t)((v->at + 1) % VAD_SMOOTH); + int32_t score = 0; + for (int i = 0; i < VAD_SMOOTH; i++) { + score += v->recent[i]; + } + score /= VAD_SMOOTH; + uint8_t voice; + if (score >= VAD_ON) { + v->hold = VAD_HOLD; + voice = 1; + } else if (v->hold > 0) { + v->hold--; + voice = 1; + } else { + voice = 0; + } + int32_t rise = voice ? VAD_RISE_VOICE + (int32_t)(v->run >> VAD_RISE_SHIFT) : VAD_RISE; + for (int b = 0; b < AUDIODSP_FEATURES_SIZE; b++) { + if (level[b] < VAD_LEVEL_MIN) { + continue; + } + int32_t t = level[b] << 4; + int32_t d = t - v->floor[b]; + if (d < 0) { + v->floor[b] -= -d < VAD_FALL ? -d : VAD_FALL; + } else { + v->floor[b] += d < rise ? d : rise; + } + } + v->run = voice ? (v->run < 65535 ? v->run + 1 : v->run) : 0; + v->voice = voice; + return voice; +} + // --- the engine -------------------------------------------------------------- struct audiodsp_features { @@ -219,6 +348,8 @@ struct audiodsp_features { rs_t rs; uint32_t rate; // last rate fed uint16_t *queue; // capacity slices + uint8_t *voice; // a flag for each queued slice + audiodsp_vad_t vad; uint32_t capacity, head, count; uint32_t slices, dropped; }; @@ -259,8 +390,9 @@ audiodsp_features_t *audiodsp_features_new(uint32_t capacity) { } f->capacity = capacity; f->queue = malloc((size_t)capacity * AUDIODSP_FEATURES_SIZE * sizeof(uint16_t)); + f->voice = malloc(capacity); features_config(&f->config); - if (f->queue == NULL) { + if (f->queue == NULL || f->voice == NULL) { audiodsp_features_free(f); return NULL; } @@ -284,6 +416,7 @@ void audiodsp_features_free(audiodsp_features_t *f) { } rs_free(&f->rs); free(f->queue); + free(f->voice); free(f); } @@ -292,11 +425,12 @@ void audiodsp_features_reset(audiodsp_features_t *f) { if (f->rs.buf != NULL) { rs_restart(&f->rs); } + vad_reset(&f->vad); f->head = f->count = 0; f->slices = f->dropped = 0; } -static void features_push(audiodsp_features_t *f, const uint16_t *values) { +static void features_push(audiodsp_features_t *f, const uint16_t *values, uint8_t voice) { if (f->count == f->capacity) { f->head = (f->head + 1) % f->capacity; f->count--; @@ -304,23 +438,47 @@ static void features_push(audiodsp_features_t *f, const uint16_t *values) { } uint32_t tail = (f->head + f->count) % f->capacity; memcpy(f->queue + (size_t)tail * AUDIODSP_FEATURES_SIZE, values, AUDIODSP_FEATURES_SIZE * sizeof(uint16_t)); + f->voice[tail] = voice; f->count++; f->slices++; } +// One slice, when the window has filled: FrontendProcessSamples() step for +// step (frontend.c), so the values are the library's own, with a copy of the +// band amplitudes taken for voice activity before noise reduction changes +// them. Returns false while the window still wants samples. +static bool features_slice(audiodsp_features_t *f, const int16_t *x, size_t n, size_t *read) { + struct FrontendState *st = &f->fe; + if (!WindowProcessSamples(&st->window, x, n, read)) { + return false; + } + int input_shift = 15 - MostSignificantBit32(st->window.max_abs_output_value); + FftCompute(&st->fft, st->window.output, input_shift); + int32_t *energy = (int32_t *)st->fft.output; + FilterbankConvertFftComplexToEnergy(&st->filterbank, st->fft.output, energy); + FilterbankAccumulateChannels(&st->filterbank, energy); + uint32_t *scaled = FilterbankSqrt(&st->filterbank, input_shift); + uint8_t voice = vad_slice(&f->vad, scaled); + NoiseReductionApply(&st->noise_reduction, scaled); + if (st->pcan_gain_control.enable_pcan) { + PcanGainControlApply(&st->pcan_gain_control, scaled); + } + int correction_bits = MostSignificantBit32(st->fft.fft_size) - 1 - (kFilterbankBits / 2); + uint16_t *logged = LogScaleApply(&st->log_scale, scaled, st->filterbank.num_channels, correction_bits); + features_push(f, logged, voice); + return true; +} + // 16 kHz mono samples through the frontend. static void features_run(audiodsp_features_t *f, const int16_t *x, size_t n) { while (n > 0) { size_t read = 0; - struct FrontendOutput out = FrontendProcessSamples(&f->fe, x, n, &read); - if (read == 0) { + bool made = features_slice(f, x, n, &read); + if (read == 0 && !made) { break; } x += read; n -= read; - if (out.size == AUDIODSP_FEATURES_SIZE) { - features_push(f, out.values); - } } } @@ -427,9 +585,12 @@ uint32_t audiodsp_features_waiting(const audiodsp_features_t *f) { return f->count; } -size_t audiodsp_features_read_u16(audiodsp_features_t *f, uint16_t *out, size_t max_slices) { +size_t audiodsp_features_read_u16(audiodsp_features_t *f, uint16_t *out, size_t max_slices, uint8_t *voice) { size_t n = 0; while (n < max_slices && f->count > 0) { + if (voice != NULL) { + voice[n] = f->voice[f->head]; + } memcpy(out + n * AUDIODSP_FEATURES_SIZE, f->queue + (size_t)f->head * AUDIODSP_FEATURES_SIZE, AUDIODSP_FEATURES_SIZE * sizeof(uint16_t)); f->head = (f->head + 1) % f->capacity; @@ -439,9 +600,12 @@ size_t audiodsp_features_read_u16(audiodsp_features_t *f, uint16_t *out, size_t return n; } -size_t audiodsp_features_read_s8(audiodsp_features_t *f, int8_t *out, size_t max_slices) { +size_t audiodsp_features_read_s8(audiodsp_features_t *f, int8_t *out, size_t max_slices, uint8_t *voice) { size_t n = 0; while (n < max_slices && f->count > 0) { + if (voice != NULL) { + voice[n] = f->voice[f->head]; + } const uint16_t *v = f->queue + (size_t)f->head * AUDIODSP_FEATURES_SIZE; for (int i = 0; i < AUDIODSP_FEATURES_SIZE; i++) { out[n * AUDIODSP_FEATURES_SIZE + i] = audiodsp_features_quantize(v[i]); @@ -453,6 +617,10 @@ size_t audiodsp_features_read_s8(audiodsp_features_t *f, int8_t *out, size_t max return n; } +bool audiodsp_features_speaking(const audiodsp_features_t *f) { + return f->vad.voice != 0; +} + void audiodsp_features_stats(const audiodsp_features_t *f, audiodsp_features_stats_t *st) { st->slices = f->slices; st->dropped = f->dropped; diff --git a/src/shared/audiodsp_features.h b/src/shared/audiodsp_features.h index 083cdcd..ff79b0f 100644 --- a/src/shared/audiodsp_features.h +++ b/src/shared/audiodsp_features.h @@ -18,6 +18,7 @@ #pragma once +#include #include #include @@ -57,9 +58,13 @@ uint32_t audiodsp_features_room(const audiodsp_features_t *f); uint32_t audiodsp_features_waiting(const audiodsp_features_t *f); // Up to max_slices of the oldest queued slices, removed from the queue. -// Returns how many were written: AUDIODSP_FEATURES_SIZE values each. -size_t audiodsp_features_read_u16(audiodsp_features_t *f, uint16_t *out, size_t max_slices); -size_t audiodsp_features_read_s8(audiodsp_features_t *f, int8_t *out, size_t max_slices); +// Returns how many were written: AUDIODSP_FEATURES_SIZE values each. When +// voice isn't NULL it gets one byte a slice: 1 when the slice holds speech. +size_t audiodsp_features_read_u16(audiodsp_features_t *f, uint16_t *out, size_t max_slices, uint8_t *voice); +size_t audiodsp_features_read_s8(audiodsp_features_t *f, int8_t *out, size_t max_slices, uint8_t *voice); + +// Whether the newest slice made holds speech, read or not. +bool audiodsp_features_speaking(const audiodsp_features_t *f); // The int8 a microWakeWord model takes for one raw value: ESPHome's // value * 256 / 666 - 128, rounded, in integers. diff --git a/tests/data/audiofeatures/bursts_16k.voice b/tests/data/audiofeatures/bursts_16k.voice new file mode 100644 index 0000000..0962bda Binary files /dev/null and b/tests/data/audiofeatures/bursts_16k.voice differ diff --git a/tests/data/audiofeatures/signal_16k.voice b/tests/data/audiofeatures/signal_16k.voice new file mode 100644 index 0000000..a87f6b4 Binary files /dev/null and b/tests/data/audiofeatures/signal_16k.voice differ diff --git a/tests/parity/features_probe.py b/tests/parity/features_probe.py index 14d1f0d..91c6b5a 100644 --- a/tests/parity/features_probe.py +++ b/tests/parity/features_probe.py @@ -10,7 +10,8 @@ The signal is made with integers only (tone and noise bursts, as in tests/test_cpython_audiofeatures.py), then held at 48 kHz stereo and 44.1 kHz mono by repeating samples, which is integer too. Each case is fed in two block -sizes, and both must print the same. +sizes, and both must print the same. Each case also prints its voice activity +flags, one character a slice, which are integer arithmetic too. """ import sys @@ -23,12 +24,18 @@ SIZE = 40 -def signal(quarters=6): +PLAN = [(0, 0, 300), (7721, 9000, 300), (3861, 12000, 2000), (0, 0, 9000), + (25736, 6000, 600), (38604, 3000, 4000)] +# For voice activity: a quiet floor, a burst, a long quiet, two bursts, quiet. +QUIET = (0, 0, 300) +VOICE_PLAN = [QUIET] * 4 + [(3861, 12000, 300), (7721, 6000, 300)] + [QUIET] * 6 + \ + [(5148, 8000, 600)] + [QUIET] * 2 + [(0, 0, 6000)] + [QUIET] * 4 + + +def signal(quarters=6, plan=PLAN): out = array("h") s = 2024 seg = 4000 - plan = [(0, 0, 300), (7721, 9000, 300), (3861, 12000, 2000), (0, 0, 9000), - (25736, 6000, 600), (38604, 3000, 4000)] for i in range(quarters): step, level, noise = plan[i % len(plan)] x, y = level << 8, 0 @@ -57,28 +64,32 @@ def hold(pcm, rate, channels): def run(pcm, rate, channels, block): fe = features_module.Frontend(slices=64) raw = array("H", bytes(2 * SIZE * 64)) + voice = bytearray(64) got = array("H") + flags = [] mv = memoryview(pcm) step = block * channels for at in range(0, len(pcm), step): fe.feed(mv[at:at + step], rate, channels) - n = fe.readinto_raw(raw) + n = fe.readinto_raw(raw, voice) got.extend(raw[:n * SIZE]) + flags.extend("v" if voice[i] else "." for i in range(n)) dropped = fe.stats()["dropped"] fe.deinit() - return got, dropped + return got, dropped, "".join(flags) def main(): base = signal() cases = ( ("16k_mono", base, 16000, 1), + ("16k_voice", signal(len(VOICE_PLAN), VOICE_PLAN), 16000, 1), ("48k_stereo", hold(base, 48000, 2), 48000, 2), ("44k1_mono", hold(base, 44100, 1), 44100, 1), ) for name, pcm, rate, channels in cases: - a, dropped = run(pcm, rate, channels, 480) - b, _ = run(pcm, rate, channels, 37) + a, dropped, va = run(pcm, rate, channels, 480) + b, _, vb = run(pcm, rate, channels, 37) h = 2166136261 for v in a: h = ((h ^ v) * 16777619) & 0xFFFFFFFF @@ -86,7 +97,9 @@ def main(): print("%s slices %d dropped %d fnv %08x" % (name, slices, dropped, h)) for k in range(0, slices, 25): print("%s %d %s" % (name, k, " ".join("%d" % v for v in a[k * SIZE:(k + 1) * SIZE]))) - print("%s blocks %s" % (name, "same" if a == b else "DIFFERENT")) + print("%s blocks %s" % (name, "same" if a == b and va == vb else "DIFFERENT")) + for k in range(0, len(va), 50): + print("%s voice %d %s" % (name, k, va[k:k + 50])) print("quantize", " ".join("%d" % features_module.quantize(v) for v in (0, 1, 332, 333, 334, 665, 666, 670, 65535))) diff --git a/tests/test_cpython_audiofeatures.py b/tests/test_cpython_audiofeatures.py index 158e2cd..caa11da 100644 --- a/tests/test_cpython_audiofeatures.py +++ b/tests/test_cpython_audiofeatures.py @@ -34,7 +34,7 @@ SIZE = audiofeatures.FEATURE_SIZE -def signal(seconds=3): +def signal(seconds=3, plan=None): """Integer-only 16 kHz test audio: a noise floor, then tone and noise bursts. Integers only, so it is the same audio on every platform. Changing it @@ -44,10 +44,14 @@ def signal(seconds=3): s = 2024 seg = 16000 // 4 # (Minsky circle step, 2*pi*f/16000 in Q16; tone level; noise level) - plan = [(0, 0, 300), (0, 0, 300), (7721, 9000, 300), (15442, 4000, 300), - (3861, 12000, 2000), (0, 0, 9000), (25736, 6000, 600), (0, 0, 300), - (11581, 15000, 300), (38604, 3000, 4000), (0, 0, 150), (5148, 8000, 1200)] - for i in range(seconds * 4): + quarters = seconds * 4 + if plan is None: + plan = [(0, 0, 300), (0, 0, 300), (7721, 9000, 300), (15442, 4000, 300), + (3861, 12000, 2000), (0, 0, 9000), (25736, 6000, 600), (0, 0, 300), + (11581, 15000, 300), (38604, 3000, 4000), (0, 0, 150), (5148, 8000, 1200)] + else: + quarters = len(plan) + for i in range(quarters): step, level, noise = plan[i % len(plan)] x, y = level << 8, 0 for n in range(seg): @@ -65,8 +69,20 @@ def signal(seconds=3): return out -def golden(ext): - with open(os.path.join(DATA, "signal_16k." + ext), "rb") as f: +QUIET = (0, 0, 300) +#: Quarter-second segments for voice activity: a quiet floor, tone bursts +#: (rising onto the floor and dying away) and a burst of loud noise. +BURSTS = [QUIET] * 4 + [(3861, 12000, 300), (7721, 6000, 300)] + [QUIET] * 6 + \ + [(5148, 8000, 600)] + [QUIET] * 2 + [(0, 0, 6000)] + [QUIET] * 4 + + +def bursts(): + """signal()'s generator over BURSTS: 5 s, integers only.""" + return signal(plan=BURSTS) + + +def golden(ext, name="signal_16k"): + with open(os.path.join(DATA, name + "." + ext), "rb") as f: return f.read() @@ -189,6 +205,134 @@ def test_rate_change_restarts_cleanly(self): self.assertEqual(fe.stats()["rate"], 16000) +def voice(pcm, rate=16000, block=480, channels=1): + """Every slice's voice flag, as bytes, fed `block` frames at a time.""" + fe = audiofeatures.Frontend(slices=64) + buf = bytearray(SIZE * 64) + flags = bytearray(64) + out = bytearray() + mv = memoryview(pcm).cast("B") + step = block * 2 * channels + for at in range(0, len(mv), step): + fe.feed(mv[at:at + step], rate, channels) + n = fe.readinto(buf, flags) + out += flags[:n] + fe.deinit() + return bytes(out) + + +def noise(seconds, rms, seed=1): + """Gaussian noise at `rms`, from numpy's seeded generator.""" + rng = np.random.default_rng(seed) + return rng.normal(0, rms, int(16000 * seconds)) + + +def pcm(x): + return np.clip(np.round(x), -32768, 32767).astype(" 0 or noise_level > 1000 + before = q > 0 and (BURSTS[q - 1][1] > 0 or BURSTS[q - 1][2] > 1000) + if loud: + self.assertGreater(sum(quarter(q)), 15, q) + elif not before: + self.assertEqual(sum(quarter(q)), 0, q) + + def test_steady_noise_is_never_voice(self): + for rms in (30, 300, 3000): + with self.subTest(rms=rms): + flags = voice(pcm(noise(10, rms))) + self.assertEqual(sum(flags), 0) + + def test_a_noise_that_steps_up_is_learnt(self): + # 3 s of quiet noise, then 20 dB louder for 10 s: voice at first, + # since that is what a sudden sound is, then never again. + x = np.concatenate([noise(3, 100), noise(10, 1000, seed=2)]) + flags = voice(pcm(x)) + self.assertGreater(sum(flags[300:350]), 40) + self.assertEqual(sum(flags[800:]), 0) # measured: clear 3.5 s after the step + + def test_digital_silence_then_noise(self): + # A stream that starts silent and then meets a microphone's floor. + x = np.concatenate([np.zeros(32000), noise(10, 100)]) + flags = voice(pcm(x)) + self.assertEqual(sum(flags[:200]), 0) + self.assertEqual(sum(flags[700:]), 0) + + def test_48k_flags_match_16k(self): + x = np.frombuffer(bursts().tobytes(), "