audiofeatures: speech features for a wake word model

MANIFEST.in

@@ -3,3 +3,5 @@ recursive-include src/shared *.h
 recursive-include lib/audiorender *.py *.md
 recursive-include lib/audiodsp_util *.py
 include third_party/tremor/*.c third_party/tremor/*.h third_party/tremor/COPYING third_party/tremor/ORIGIN.txt third_party/tremor/README
+recursive-include third_party/microfrontend *.c *.cc *.h LICENSE ORIGIN.txt
+recursive-include third_party/kissfft *.c *.h COPYING ORIGIN.txt

NOTICE

@@ -17,3 +17,10 @@ third_party/tremor/ is Xiph.Org's Tremor, an integer Ogg Vorbis decoder,
 used under the BSD-3-Clause license in third_party/tremor/COPYING
 (Copyright (c) 2002, Xiph.org Foundation). third_party/tremor/ORIGIN.txt names
 the upstream commit and the changes made to it.
+
+third_party/microfrontend/ is TensorFlow Lite Micro's audio microfrontend,
+used under the Apache-2.0 license in third_party/microfrontend/LICENSE
+(Copyright The TensorFlow Authors). third_party/kissfft/ is KISS FFT, used
+under the BSD-3-Clause license in third_party/kissfft/COPYING (Copyright (c)
+2003-2010 Mark Borgerding). Each directory's ORIGIN.txt names the upstream
+commit it came from and how it is built.

ROADMAP.md

@@ -15,6 +15,10 @@ audio pump holding up under live use such as a stompbox or a stage rig.
 - A level ramp on `audiomixer` voices: a level move ramps over a set number of
   frames instead of stepping, so an effect can fade its wet path in or out
   without a click when you switch it in mid-note.
+- Voice activity in `audiofeatures`: a flag per slice saying whether it holds
+  speech, from the frontend's own noise estimate, so a wake word model need
+  only run while someone is talking. The frontend and resampler take about
+  16 KB of flash on an ESP32-P4 today; this should add little to it.
 
 Bugs, and things you need that don't work yet, go to
 [issues](https://github.com/PyDevices/audiodsp/issues).

docs/get-started/install.md

@@ -18,7 +18,7 @@ This gets you `audiocore`, `synthio`, `audiomixer`, `audiofilters`,
 `audiodelays`, `audiofreeverb`, `audiospeed`, `audiodynamics`, `audioroute`,
 `audiomath`, `audioecho`, `audioshaper`, `audioladder`, `audiooscillator`,
 `audioconvolve`, `audiobiquad`, `audioverb`, `audiomodal`, `audiosampler`, `audiometer`,
-`audiopump`, `audiovorbis`, `audiograph`, and the `audiorender` package.
+`audiofeatures`, `audiopump`, `audiovorbis`, `audiograph`, and the `audiorender` package.
 `audiomp3` remains firmware-only. The distribution declares no *required*
 runtime dependencies and does not itself publish an `audiodsp` import; its
 version is the `VERSION` file, which is also what `_audiodsp.__version__`
@@ -105,5 +105,5 @@ own `audiobusio`, `audioio` and `audiopwmio`; `audiocore`, `synthio`,
 `audiomixer` and the effects modules stay CircuitPython's. Two modules differ
 there: `audiopump` carries only `Ring` (see
 [Play computed audio](../guides/play-computed-audio.md)), and `audiometer`
-isn't built, because its sources are a board's pump and a USB sound card, and
-CircuitPython has neither.
+and `audiofeatures` aren't built, because their sources are a board's pump and
+a USB sound card, and CircuitPython has neither.

docs/guides/speech-features.md

@@ -0,0 +1,63 @@
+# Speech features
+
+`audiofeatures` turns audio into what a microWakeWord model listens to: every
+10 ms, a slice of 40 values, one per mel band from 125 Hz to 7.5 kHz. Hand
+the slices to a TensorFlow Lite Micro model and you have a wake word
+detector; the model and the interpreter that runs it aren't part of audiodsp.
+
+```python
+fe = audiofeatures.Frontend()
+fe.feed(pcm, 48000, 2)                 # interleaved s16, any rate from 16 kHz
+buf = bytearray(audiofeatures.FEATURE_SIZE * 10)
+n = fe.readinto(buf)                   # n whole slices of 40 int8
+```
+
+[`examples/audiofeatures_simpletest.py`](../../examples/audiofeatures_simpletest.py)
+prints the slices of a WAV file, one row of characters each.
+
+On a board it can listen without Python touching a sample.
+`fe.attach(tap, 48000)` reads an `audiopump.Tap` in C each time you call
+`readinto()`, so whatever the pump is playing or recording, an I2S
+microphone or a USB sound card, becomes slices. A tap runs at the pump's
+rate and channel count; give `attach()` the rate.
+
+## What a slice is
+
+The frontend is TensorFlow Lite Micro's audio microfrontend, carried in
+[`third_party/microfrontend`](../../third_party/microfrontend/ORIGIN.txt),
+with the settings ESPHome's micro_wake_word models were trained on: a 30 ms
+window every 10 ms, 40 channels, noise reduction, PCAN gain control and a
+log scale. `readinto()` gives each value as the int8 those models take,
+`(v * 256 + 333) // 666 - 128`, and `audiofeatures.quantize()` is that sum.
+`readinto_raw()` gives the frontend's own uint16 values instead, about 0 to
+670.
+
+At 16 kHz the samples go to the frontend untouched, and the bytes are the
+ones TensorFlow Lite Micro's own C++ build makes, in any block size.
+[`tests/test_cpython_audiofeatures.py`](../../tests/test_cpython_audiofeatures.py)
+holds goldens that build wrote.
+
+## Other rates
+
+Audio at any other rate up to 384 kHz is resampled to 16 kHz first, through
+a windowed-sinc low-pass that is flat to 7.5 kHz, where the top mel band
+ends, and about 70 dB down from 8.5 kHz up. Stereo and wider audio is mixed
+to mono before it. The features can't be byte-identical to the 16 kHz ones,
+but they are close: on 74 spoken clips fed at 48 kHz, 95% of the int8 values
+are the same as at 16 kHz and the mean difference is 0.27 of a step, and
+three microWakeWord models detected and missed exactly the same clips, with
+peak probabilities within 0.03.
+
+## The queue
+
+Slices wait in a queue of `slices` (100 by default, a second of audio) until
+you read them. When it is full, the oldest slice goes and `stats()["dropped"]`
+counts it, so a reader that falls behind loses old audio, never new.
+`reset()` forgets the audio and the queue, for the start of a new utterance.
+
+A frontend takes about 17 to 26 KB of heap with the default queue, depending
+on the rate: 8 KB is the queue, the rest the frontend's state and the
+resampler's filter.
+
+CircuitPython doesn't get `audiofeatures`; see
+[additions](../how-it-works/additions.md).

docs/how-it-works/additions.md

@@ -40,10 +40,10 @@ These have no ancestor anywhere and are audiodsp's own:
 Every one of them builds into CircuitPython 11 too, as a user C module; see
 [Install](../get-started/install.md#circuitpython-11).
 
-Three more are for cases CircuitPython doesn't cover. `audiometer` (band
-levels for a spectrum meter) is the one CircuitPython doesn't get: its
-sources are a board's pump and a USB sound card, and CircuitPython has
-neither. `audiovorbis` plays Ogg Vorbis files, which CircuitPython has no
+Four more are for cases CircuitPython doesn't cover. `audiometer` (band
+levels for a spectrum meter) and `audiofeatures` (speech features for a wake
+word model) are the two CircuitPython doesn't get: their sources are a board's
+pump and a USB sound card, and CircuitPython has neither. `audiovorbis` plays Ogg Vorbis files, which CircuitPython has no
 module for. And `audiopump.Ring` lets CircuitPython play audio your own code
 computes.
 

docs/index.md

@@ -21,6 +21,7 @@ have. Start with install, then your first program.
 - [Filter a sound](guides/filter-a-sound.md) with `audiobiquad`.
 - [Play recorded notes](guides/play-recorded-notes.md) with `audiosampler`.
 - [Meter levels](guides/meter-levels.md) for a spectrum display.
+- [Speech features](guides/speech-features.md) for a wake word model.
 - [Render on several threads](guides/use-threads.md), including free-threaded Python.
 
 ## Reference

docs/reference/modules.md

@@ -25,6 +25,7 @@ It adds modules CircuitPython doesn't have. Each has a page here or a guide:
 | `audiograph` | A whole patch rendered by one C call per block | [reference](sources-and-synthesis.md#audiograph), [how it works](../how-it-works/audiograph.md) |
 | `audiosampler` | Recorded notes played at any pitch, looped, under an envelope | [guide](../guides/play-recorded-notes.md) |
 | `audiometer` | Band levels for a spectrum meter | [guide](../guides/meter-levels.md) |
+| `audiofeatures` | Speech features for a wake word model | [guide](../guides/speech-features.md) |
 | `audiovorbis` | Ogg Vorbis files | [guide](../guides/decode-vorbis.md), [C interface](vorbis.md) |
 | `audiopump` | The audio pull, off the interpreter thread | [guide](../guides/run-the-audio-pump.md), [reference](audiopump.md) |
 

examples/audiofeatures_simpletest.py

@@ -0,0 +1,67 @@
+# SPDX-FileCopyrightText: 2026 Brad Barnett
+#
+# SPDX-License-Identifier: MIT
+"""Print the speech features a wake word model sees in a WAV file.
+
+An audiofeatures Frontend turns every 10 ms of audio into a slice of 40
+values, one per mel band from 125 Hz to 7.5 kHz. The program feeds it a WAV
+file a block at a time and prints each slice as a row of characters, low
+bands on the left, darker for more energy, so speech shows up as dense rows
+and silence as blank ones.
+
+Needs: audiodsp, built into MicroPython or installed on CPython with
+pip install pydevices-audiodsp, and a 16-bit PCM WAV file at 16 kHz or
+higher, mono or stereo.
+
+Run it:
+    micropython audiofeatures_simpletest.py
+
+Tested on: the unix port of MicroPython and CPython 3.12 on Linux.
+"""
+
+import struct
+
+import audiofeatures
+
+# Change these to match your setup.
+FILENAME = "speech.wav"
+BLOCK = 480  # frames read and fed at a time
+SHADES = " .:-=+*#%@"  # from no energy to the most
+
+
+def main():
+    with open(FILENAME, "rb") as f:
+        # Walk the WAV's chunks to its format and the start of its samples.
+        assert f.read(12)[8:] == b"WAVE", "not a WAV file"
+        while True:
+            tag, size = struct.unpack("<4sI", f.read(8))
+            if tag == b"data":
+                break
+            body = f.read(size + (size & 1))
+            if tag == b"fmt ":
+                kind, channels, rate = struct.unpack("<HHI", body[:8])
+                assert kind == 1 and body[14] == 16, "16-bit PCM only"
+
+        # Feed the samples to the frontend and print each slice it makes.
+        frontend = audiofeatures.Frontend()
+        features = bytearray(audiofeatures.FEATURE_SIZE * 10)
+        left, rows = size, 0
+        while left > 0:
+            block = f.read(min(left, BLOCK * 2 * channels))
+            left -= len(block)
+            frontend.feed(block, rate, channels)
+            count = frontend.readinto(features)
+            for i in range(count * audiofeatures.FEATURE_SIZE):
+                # Each value is a signed byte, -128 to 127, held unsigned here.
+                value = (features[i] + 128) & 0xFF
+                print(SHADES[value * len(SHADES) // 256], end="")
+                if i % audiofeatures.FEATURE_SIZE == audiofeatures.FEATURE_SIZE - 1:
+                    print()
+            rows += count
+
+    # Every 10 ms after the first 30 ms makes a slice.
+    print("%d slices from %d Hz audio, %d dropped" % (rows, rate, frontend.stats()["dropped"]))
+
+
+if __name__ == "__main__":
+    main()

micropython.cmake

@@ -249,6 +249,22 @@ target_sources(usermod_mpaudio INTERFACE
     ${MPAUDIO_SRC_DIR}/audiovorbis/module.c
 )
 
+# --- audiofeatures (Frontend) ---
+#
+# See micropython.mk: TensorFlow Lite Micro's microfrontend and kissfft from
+# third_party, compiled as one translation unit by
+# src/shared/audiodsp_microfrontend.c, which renames their symbols and silences
+# their warnings for their own code only.
+target_sources(usermod_mpaudio INTERFACE
+    ${MPAUDIO_SRC_DIR}/shared/audiodsp_microfrontend.c
+    ${MPAUDIO_SRC_DIR}/shared/audiodsp_features.c
+    ${MPAUDIO_SRC_DIR}/audiofeatures/module.c
+)
+target_include_directories(usermod_mpaudio INTERFACE
+    ${MPAUDIO_MOD_DIR}/third_party/microfrontend
+    ${MPAUDIO_MOD_DIR}/third_party/kissfft
+)
+
 
 # CIRCUITPY_SYNTHIO_MAX_CHANNELS (phase 8e decision, made while actually
 # building the ESP32-P4 firmware -- see docs/porting-plan.md "8e"): the

micropython.mk

@@ -388,4 +388,18 @@ SRC_USERMOD_C += \
     $(MPAUDIO_SRC_DIR)/audiovorbis/VorbisDecoder.c \
     $(MPAUDIO_SRC_DIR)/audiovorbis/module.c
 
+# --- audiofeatures (Frontend) ---
+#
+# TensorFlow Lite Micro's audio microfrontend and the kissfft it calls, carried
+# in third_party/microfrontend (Apache-2.0) and third_party/kissfft
+# (BSD-3-Clause), compiled as one translation unit by
+# src/shared/audiodsp_microfrontend.c the way Tremor is. The library names its
+# headers by their full tensorflow/... path, hence the two include directories.
+# docs/guides/speech-features.md has what it does.
+CFLAGS_USERMOD += -I$(MPAUDIO_MOD_DIR)/third_party/microfrontend -I$(MPAUDIO_MOD_DIR)/third_party/kissfft
+SRC_USERMOD_LIB_C += \
+    $(MPAUDIO_SRC_DIR)/shared/audiodsp_microfrontend.c \
+    $(MPAUDIO_SRC_DIR)/shared/audiodsp_features.c
+SRC_USERMOD_C += $(MPAUDIO_SRC_DIR)/audiofeatures/module.c
+
 endif # not CircuitPython

pyproject.toml

@@ -9,9 +9,11 @@ description = "CircuitPython-compatible audio synthesis and effects for PyDevice
 readme = "README.md"
 requires-python = ">=3.11"
 # audiovorbis compiles Xiph's Tremor into the extension, under its own
-# BSD-3-Clause licence, whose text travels with the wheel.
-license = "MIT AND BSD-3-Clause"
-license-files = ["LICENSE", "NOTICE", "third_party/tremor/COPYING"]
+# BSD-3-Clause licence, and audiofeatures compiles TensorFlow Lite Micro's
+# microfrontend (Apache-2.0) and kissfft (BSD-3-Clause). Their texts travel
+# with the wheel.
+license = "MIT AND BSD-3-Clause AND Apache-2.0"
+license-files = ["LICENSE", "NOTICE", "third_party/tremor/COPYING", "third_party/microfrontend/LICENSE", "third_party/kissfft/COPYING"]
 authors = [{name = "PyDevices contributors"}]
 classifiers = [
   "Programming Language :: Python :: 3",
@@ -41,7 +43,7 @@ Repository = "https://github.com/PyDevices/audiodsp"
 Issues = "https://github.com/PyDevices/audiodsp/issues"
 
 [tool.setuptools]
-py-modules = ["audiocore", "synthio", "audiomixer", "audiofilters", "audiodelays", "audiofreeverb", "audiospeed", "audiodynamics", "audioroute", "audiomath", "audioecho", "audioconvolve", "audiobiquad", "audioshaper", "audioladder", "audiooscillator", "audioverb", "audiomodal", "audiosampler", "audiometer", "audiopump", "audiovorbis", "audiograph"]
+py-modules = ["audiocore", "synthio", "audiomixer", "audiofilters", "audiodelays", "audiofreeverb", "audiospeed", "audiodynamics", "audioroute", "audiomath", "audioecho", "audioconvolve", "audiobiquad", "audioshaper", "audioladder", "audiooscillator", "audioverb", "audiomodal", "audiosampler", "audiometer", "audiofeatures", "audiopump", "audiovorbis", "audiograph"]
 # Keep py-modules explicit: src/cpython is the root package directory now, and
 # an automatic scan there would also pick up anything else that lands in it.
 packages = ["audiorender", "audiodsp_util"]

setup.py

@@ -75,8 +75,12 @@
                 # Xiph's Tremor, included into this one file from
                 # third_party/tremor; docs/reference/vorbis.md.
                 "src/shared/audiodsp_vorbis.c",
+                # TensorFlow Lite Micro's microfrontend and kissfft, included
+                # into this one file from third_party; audiofeatures' engine.
+                "src/shared/audiodsp_microfrontend.c",
+                "src/shared/audiodsp_features.c",
             ],
-            include_dirs=["src"],
+            include_dirs=["src", "third_party/microfrontend", "third_party/kissfft"],
             define_macros=[("AUDIODSP_VERSION", '"%s"' % VERSION),
                            ("AUDIODSP_REVISION", '"%s"' % REVISION)],
             extra_compile_args=MACOS_COMPILE_ARGS,