audiofeatures: a voice activity flag per slice

ROADMAP.md

@@ -15,10 +15,6 @@ audio pump holding up under live use such as a stompbox or a stage rig.
 - A level ramp on `audiomixer` voices: a level move ramps over a set number of
   frames instead of stepping, so an effect can fade its wet path in or out
   without a click when you switch it in mid-note.
-- Voice activity in `audiofeatures`: a flag per slice saying whether it holds
-  speech, from the frontend's own noise estimate, so a wake word model need
-  only run while someone is talking. The frontend and resampler take about
-  16 KB of flash on an ESP32-P4 today; this should add little to it.
 
 Bugs, and things you need that don't work yet, go to
 [issues](https://github.com/PyDevices/audiodsp/issues).

docs/guides/speech-features.md

@@ -48,6 +48,85 @@ are the same as at 16 kHz and the mean difference is 0.27 of a step, and
 three microWakeWord models detected and missed exactly the same clips, with
 peak probabilities within 0.03.
 
+## Is anyone speaking?
+
+Each slice also carries a voice activity flag: 1 when it holds speech, 0
+when it holds silence or steady noise. Pass a second buffer, one byte a
+slice, and `readinto()` fills it beside the features; `speaking()` is the
+newest slice's flag on its own.
+
+```python
+voice = bytearray(10)
+n = fe.readinto(buf, voice)            # voice[i] is slice i's flag
+if fe.speaking():
+    ...                                # someone is talking right now
+```
+
+A wake word detector can skip the model while every flag is 0, and an
+intercom can hold the floor while `speaking()` is true.
+[`examples/audiofeatures_voice_activity.py`](../../examples/audiofeatures_voice_activity.py)
+prints where the speech is in a WAV file. `readinto_raw()` takes the same
+second buffer. Attached to a tap or fed from Python, at any rate, the flags
+come with their slices through the same queue.
+
+### How it decides
+
+The detector reads the frontend's 40 band levels before noise reduction,
+in steps of a 64th of an octave, and keeps a noise floor for each band.
+The floor follows the quiet parts quickly and rises slowly, so a fan or
+road noise that stays on is learnt within a few seconds and stops counting.
+A slice's score is how far its four loudest bands stand above their floors,
+averaged over the last 50 ms; over about 10 dB is speech. A flag stays on
+for 100 ms after the score drops, so the gaps between words don't break a
+sentence into pieces. All of it is integer arithmetic, so CPython,
+MicroPython and a board give the same flags for the same audio.
+
+The first 160 ms after the frontend starts or `reset()` is spent learning
+the floor and is never flagged, so reset it in quiet, not mid-word.
+Digital silence (a stream of zeros) is left out of the floor, so it doesn't
+make the noise that follows it look like speech.
+
+### How well it works
+
+We scored it against Silero VAD, a neural voice detector, on 97
+recordings: 72 spoken phrases, 23 longer recordings and sounds, and two
+of steady noise, with the phrases also mixed with loud noise and turned
+down 30 dB. Each clip had two seconds of its own
+background first, as an always-on device would hear. The fraction of
+10 ms slices on which the two agree:
+
+| Audio | Agreement | Speech found | Noise called speech |
+| --- | --- | --- | --- |
+| Clean spoken phrases | 92% | 99.7% | 14% |
+| Phrases with noise 10 dB below the speech | 97% | 96% | 1.7% |
+| Phrases with noise as loud as the speech | 90% | 78% | 0.3% |
+| Phrases 30 dB quieter | 97% | 99% | 4% |
+| Longer recordings | 92% | 97% | 28% |
+| Short spoken replies and alert sounds | 80% | 100% | 28% |
+| 40 seconds of steady noise | | | 0.3% |
+
+Most of the "noise called speech" on clean audio is the 100 ms hold
+running past where Silero stops. Speech starts within 30 ms of Silero's
+start. Fed at 48 kHz instead, every figure in the table is the same to the
+nearest 0.1%.
+
+On the same clips WebRTC's voice detector at its most aggressive setting
+agreed 96% on clean phrases, but called the noisy mixes speech throughout
+(55% and 42%) and missed most of the quiet phrases (62%), which is why
+this one is built on the frontend's own bands instead.
+
+What it gets wrong: a tone that comes and goes, like an alarm or a
+doorbell, looks like speech, because nothing here knows what a voice
+sounds like. Speech buried under noise louder than itself is mostly
+missed. If you need either, run a model on the slices it flags.
+
+### What it costs
+
+About 0.7 KB of flash on an ESP32-P4 on top of the frontend, 176 bytes of
+state, and a byte for each slice in the queue. Nothing to set up and no
+settings: if you want a longer hold, keep a flag on in Python for as many
+slices as you like.
+
 ## The queue
 
 Slices wait in a queue of `slices` (100 by default, a second of audio) until
@@ -56,8 +135,8 @@ counts it, so a reader that falls behind loses old audio, never new.
 `reset()` forgets the audio and the queue, for the start of a new utterance.
 
 A frontend takes about 17 to 26 KB of heap with the default queue, depending
-on the rate: 8 KB is the queue, the rest the frontend's state and the
-resampler's filter.
+on the rate: 8 KB is the queue and its voice flags, the rest the frontend's
+state and the resampler's filter.
 
 CircuitPython doesn't get `audiofeatures`; see
 [additions](../how-it-works/additions.md).

docs/reference/modules.md

@@ -25,7 +25,7 @@ It adds modules CircuitPython doesn't have. Each has a page here or a guide:
 | `audiograph` | A whole patch rendered by one C call per block | [reference](sources-and-synthesis.md#audiograph), [how it works](../how-it-works/audiograph.md) |
 | `audiosampler` | Recorded notes played at any pitch, looped, under an envelope | [guide](../guides/play-recorded-notes.md) |
 | `audiometer` | Band levels for a spectrum meter | [guide](../guides/meter-levels.md) |
-| `audiofeatures` | Speech features for a wake word model | [guide](../guides/speech-features.md) |
+| `audiofeatures` | Speech features and voice activity for a wake word model | [guide](../guides/speech-features.md) |
 | `audiovorbis` | Ogg Vorbis files | [guide](../guides/decode-vorbis.md), [C interface](vorbis.md) |
 | `audiopump` | The audio pull, off the interpreter thread | [guide](../guides/run-the-audio-pump.md), [reference](audiopump.md) |
 

examples/audiofeatures_voice_activity.py

@@ -0,0 +1,67 @@
+# SPDX-FileCopyrightText: 2026 Brad Barnett
+#
+# SPDX-License-Identifier: MIT
+"""Find where someone is speaking in a WAV file.
+
+An audiofeatures Frontend flags every 10 ms slice it makes as speech or not,
+beside the features a wake word model reads. The program feeds it a WAV file
+a block at a time, reads the flags, and prints each stretch of speech with
+its start and end time, so a file of a few phrases prints a line for each.
+
+Needs: audiodsp, built into MicroPython or installed on CPython with
+pip install pydevices-audiodsp, and a 16-bit PCM WAV file at 16 kHz or
+higher, mono or stereo, that starts with a moment of quiet.
+
+Run it:
+    micropython audiofeatures_voice_activity.py
+
+Tested on: the unix port of MicroPython and CPython 3.12 on Linux.
+"""
+
+import struct
+
+import audiofeatures
+
+# Change these to match your setup.
+FILENAME = "speech.wav"
+BLOCK = 480  # frames read and fed at a time
+
+# Walk the WAV's chunks to its format and the start of its samples.
+f = open(FILENAME, "rb")
+assert f.read(12)[8:] == b"WAVE", "not a WAV file"
+while True:
+    tag, size = struct.unpack("<4sI", f.read(8))
+    if tag == b"data":
+        break
+    body = f.read(size + (size & 1))
+    if tag == b"fmt ":
+        kind, channels, rate = struct.unpack("<HHI", body[:8])
+        assert kind == 1 and body[14] == 16, "16-bit PCM only"
+
+# The features go to a buffer this program ignores; the flags, one byte a
+# slice, go to their own.
+frontend = audiofeatures.Frontend()
+features = bytearray(audiofeatures.FEATURE_SIZE * 10)
+voice = bytearray(10)
+
+# Feed the samples and watch the flags change. Slice k starts k * 10 ms in.
+left, slices, start, voiced = size, 0, None, 0
+while left > 0:
+    block = f.read(min(left, BLOCK * 2 * channels))
+    left -= len(block)
+    frontend.feed(block, rate, channels)
+    count = frontend.readinto(features, voice)
+    for i in range(count):
+        voiced += voice[i]
+        if voice[i] and start is None:
+            start = slices + i
+        elif not voice[i] and start is not None:
+            print("speech from %.2f s to %.2f s" % (start / 100, (slices + i) / 100))
+            start = None
+    slices += count
+f.close()
+
+# Speech still going at the end of the file ends with it.
+if start is not None:
+    print("speech from %.2f s to the end" % (start / 100))
+print("%.2f s of speech in %.2f s of audio" % (voiced / 100, slices / 100))

src/audiofeatures/module.c

@@ -5,6 +5,8 @@
 //   fe.feed(pcm, 48000, channel_count=2)      # from Python, anywhere
 //   fe.attach(tap, 48000)                     # an audiopump.Tap, read in C
 //   n = fe.readinto(buf)                      # whole 10 ms slices of 40 int8
+//   n = fe.readinto(buf, voice)               # and a byte each: 1 for speech
+//   fe.speaking()                             # the newest slice holds speech
 //
 // As with audiometer.Meter, the two ways in differ in where the work runs.
 // feed() runs it in the caller. A tap is drained in C each time readinto()
@@ -141,29 +143,45 @@ static void audiofeatures_drain(audiofeatures_frontend_obj_t *self, uint32_t wan
     }
 }
 
-static mp_obj_t audiofeatures_read(mp_obj_t self_in, mp_obj_t buf_in, bool raw) {
-    audiofeatures_frontend_obj_t *self = audiofeatures_get(self_in);
+static mp_obj_t audiofeatures_read(size_t n_args, const mp_obj_t *args, bool raw) {
+    audiofeatures_frontend_obj_t *self = audiofeatures_get(args[0]);
     mp_buffer_info_t buf;
-    mp_get_buffer_raise(buf_in, &buf, MP_BUFFER_WRITE);
+    mp_get_buffer_raise(args[1], &buf, MP_BUFFER_WRITE);
     size_t width = raw ? 2 * AUDIODSP_FEATURES_SIZE : AUDIODSP_FEATURES_SIZE;
     size_t max = buf.len / width;
+    uint8_t *voice = NULL;
+    if (n_args > 2 && args[2] != mp_const_none) {
+        mp_buffer_info_t vb;
+        mp_get_buffer_raise(args[2], &vb, MP_BUFFER_WRITE);
+        voice = (uint8_t *)vb.buf;
+        if (vb.len < max) {
+            max = vb.len;
+        }
+    }
     audiofeatures_drain(self, max > UINT32_MAX ? UINT32_MAX : (uint32_t)max);
-    size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)buf.buf, max)
-        : audiodsp_features_read_s8(self->f, (int8_t *)buf.buf, max);
+    size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)buf.buf, max, voice)
+        : audiodsp_features_read_s8(self->f, (int8_t *)buf.buf, max, voice);
     return MP_OBJ_NEW_SMALL_INT(n);
 }
 
-// readinto(buf) -> slices: 40 int8 each, ready for a model
-static mp_obj_t audiofeatures_frontend_readinto(mp_obj_t self_in, mp_obj_t buf_in) {
-    return audiofeatures_read(self_in, buf_in, false);
+// readinto(buf, voice=None) -> slices: 40 int8 each, ready for a model, and
+// a byte each into voice, 1 when the slice holds speech
+static mp_obj_t audiofeatures_frontend_readinto(size_t n_args, const mp_obj_t *args) {
+    return audiofeatures_read(n_args, args, false);
+}
+static MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(audiofeatures_frontend_readinto_obj, 2, 3, audiofeatures_frontend_readinto);
+
+// readinto_raw(buf, voice=None) -> slices: 40 uint16 each, the frontend's own values
+static mp_obj_t audiofeatures_frontend_readinto_raw(size_t n_args, const mp_obj_t *args) {
+    return audiofeatures_read(n_args, args, true);
 }
-static MP_DEFINE_CONST_FUN_OBJ_2(audiofeatures_frontend_readinto_obj, audiofeatures_frontend_readinto);
+static MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(audiofeatures_frontend_readinto_raw_obj, 2, 3, audiofeatures_frontend_readinto_raw);
 
-// readinto_raw(buf) -> slices: 40 uint16 each, the frontend's own values
-static mp_obj_t audiofeatures_frontend_readinto_raw(mp_obj_t self_in, mp_obj_t buf_in) {
-    return audiofeatures_read(self_in, buf_in, true);
+// speaking() -> whether the newest slice made holds speech
+static mp_obj_t audiofeatures_frontend_speaking(mp_obj_t self_in) {
+    return mp_obj_new_bool(audiodsp_features_speaking(audiofeatures_get(self_in)->f));
 }
-static MP_DEFINE_CONST_FUN_OBJ_2(audiofeatures_frontend_readinto_raw_obj, audiofeatures_frontend_readinto_raw);
+static MP_DEFINE_CONST_FUN_OBJ_1(audiofeatures_frontend_speaking_obj, audiofeatures_frontend_speaking);
 
 static mp_obj_t audiofeatures_frontend_reset(mp_obj_t self_in) {
     audiofeatures_frontend_obj_t *self = audiofeatures_get(self_in);
@@ -223,6 +241,7 @@ static const mp_rom_map_elem_t audiofeatures_frontend_locals_table[] = {
     { MP_ROM_QSTR(MP_QSTR_detach), MP_ROM_PTR(&audiofeatures_frontend_detach_obj) },
     { MP_ROM_QSTR(MP_QSTR_readinto), MP_ROM_PTR(&audiofeatures_frontend_readinto_obj) },
     { MP_ROM_QSTR(MP_QSTR_readinto_raw), MP_ROM_PTR(&audiofeatures_frontend_readinto_raw_obj) },
+    { MP_ROM_QSTR(MP_QSTR_speaking), MP_ROM_PTR(&audiofeatures_frontend_speaking_obj) },
     { MP_ROM_QSTR(MP_QSTR_reset), MP_ROM_PTR(&audiofeatures_frontend_reset_obj) },
     { MP_ROM_QSTR(MP_QSTR_stats), MP_ROM_PTR(&audiofeatures_frontend_stats_obj) },
     { MP_ROM_QSTR(MP_QSTR_deinit), MP_ROM_PTR(&audiofeatures_frontend_deinit_obj) },

src/cpython/_audiodsp.c

@@ -3458,19 +3458,39 @@ static PyObject *features_state_feed(audiodsp_features_object_t *self, PyObject
     return PyLong_FromUnsignedLong(audiodsp_features_waiting(self->f));
 }
 
-// read(buf, raw): whole slices into a writable buffer, int8 or (raw) uint16
+// read(buf, raw, voice=None): whole slices into a writable buffer, int8 or
+// (raw) uint16, and a byte a slice into voice when it is given
 static PyObject *features_state_read(audiodsp_features_object_t *self, PyObject *args) {
-    Py_buffer out = {0};
+    Py_buffer out = {0}, vb = {0};
     int raw;
-    if (!PyArg_ParseTuple(args, "w*p:read", &out, &raw)) return NULL;
+    PyObject *voice_obj = Py_None;
+    if (!PyArg_ParseTuple(args, "w*p|O:read", &out, &raw, &voice_obj)) return NULL;
     size_t width = raw ? 2 * AUDIODSP_FEATURES_SIZE : AUDIODSP_FEATURES_SIZE;
     size_t max = (size_t)out.len / width;
-    size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)out.buf, max)
-        : audiodsp_features_read_s8(self->f, (int8_t *)out.buf, max);
+    uint8_t *voice = NULL;
+    if (voice_obj != Py_None) {
+        if (PyObject_GetBuffer(voice_obj, &vb, PyBUF_WRITABLE) != 0) {
+            PyBuffer_Release(&out);
+            return NULL;
+        }
+        voice = (uint8_t *)vb.buf;
+        if ((size_t)vb.len < max) {
+            max = (size_t)vb.len;
+        }
+    }
+    size_t n = raw ? audiodsp_features_read_u16(self->f, (uint16_t *)out.buf, max, voice)
+        : audiodsp_features_read_s8(self->f, (int8_t *)out.buf, max, voice);
     PyBuffer_Release(&out);
+    if (voice != NULL) {
+        PyBuffer_Release(&vb);
+    }
     return PyLong_FromSize_t(n);
 }
 
+static PyObject *features_state_speaking(audiodsp_features_object_t *self, PyObject *unused) {
+    return PyBool_FromLong(audiodsp_features_speaking(self->f));
+}
+
 static PyObject *features_state_reset(audiodsp_features_object_t *self, PyObject *unused) {
     audiodsp_features_reset(self->f);
     Py_RETURN_NONE;
@@ -3487,11 +3507,13 @@ static PyObject *features_state_stats(audiodsp_features_object_t *self, PyObject
 AUDIODSP_LOCKED_ARGS(features_state_feed, audiodsp_features_object_t)
 AUDIODSP_LOCKED_ARGS(features_state_read, audiodsp_features_object_t)
 AUDIODSP_LOCKED_ARGS(features_state_reset, audiodsp_features_object_t)
+AUDIODSP_LOCKED_ARGS(features_state_speaking, audiodsp_features_object_t)
 AUDIODSP_LOCKED_ARGS(features_state_stats, audiodsp_features_object_t)
 static PyMethodDef features_state_methods[] = {
     {"feed", (PyCFunction)features_state_feed_locked, METH_VARARGS, NULL},
     {"read", (PyCFunction)features_state_read_locked, METH_VARARGS, NULL},
     {"reset", (PyCFunction)features_state_reset_locked, METH_NOARGS, NULL},
+    {"speaking", (PyCFunction)features_state_speaking_locked, METH_NOARGS, NULL},
     {"stats", (PyCFunction)features_state_stats_locked, METH_NOARGS, NULL},
     {NULL, NULL, 0, NULL},
 };

src/cpython/audiofeatures.py

@@ -11,12 +11,18 @@
     fe.feed(pcm, 48000, 2)                 # interleaved s16, any rate from 16 kHz
     buf = bytearray(40 * 10)
     n = fe.readinto(buf)                   # n whole slices of 40 int8
+    voice = bytearray(10)
+    n = fe.readinto(buf, voice)            # and voice[i] is 1 where slice i holds speech
 
 Audio that isn't 16 kHz is resampled to 16 kHz first, through a low-pass that
 is flat to 7.5 kHz; 16 kHz audio goes to the frontend untouched. Each 10 ms
 makes a slice, which waits in a queue of ``slices`` until it is read. When
 the queue is full the oldest slice is dropped, and ``stats()`` counts it.
 
+Every slice also carries a voice activity flag, from the frontend's own
+filterbank: ``readinto(buf, voice)`` puts one byte a slice into ``voice``,
+and ``speaking()`` says whether the newest slice holds speech.
+
 On a board the frontend can also listen without Python touching a sample:
 ``attach(tap, sample_rate)`` reads an ``audiopump.Tap`` in C. There is no pump
 to attach to on CPython, so ``attach()`` raises there.
@@ -66,13 +72,21 @@ def attach(self, source, sample_rate=None):
     def detach(self):
         self._live()
 
-    def readinto(self, buf):
-        """Whole slices of 40 int8 into ``buf``, oldest first. Returns how many."""
-        return self._live().read(buf, False)
+    def readinto(self, buf, voice=None):
+        """Whole slices of 40 int8 into ``buf``, oldest first. Returns how many.
+
+        Given ``voice``, a writable buffer, each slice's voice flag goes into
+        it too, a byte each: 1 when the slice holds speech, 0 when not.
+        """
+        return self._live().read(buf, False, voice)
 
-    def readinto_raw(self, buf):
+    def readinto_raw(self, buf, voice=None):
         """Whole slices of 40 uint16 (the frontend's own values) into ``buf``. Returns how many."""
-        return self._live().read(buf, True)
+        return self._live().read(buf, True, voice)
+
+    def speaking(self):
+        """Whether the newest slice made holds speech, read or not."""
+        return self._live().speaking()
 
     def reset(self):
         """Forget the audio so far and every queued slice."""