audiosampler: a resonant low-pass, a modulation envelope and two LFOs per voice

docs/guides/play-recorded-notes.md

@@ -42,6 +42,51 @@ bend is played. `release()` enters the release stage from wherever the
 envelope is; `stop()` is silence from the next block; `playing` says whether
 the voice still sounds.
 
+## Filter, modulation envelope and LFOs
+
+A voice can also do what a SoundFont region asks for besides its amplitude
+envelope. Give it a resonant low-pass, a second envelope that sweeps the
+cutoff and the pitch, a vibrato, and a slower LFO on pitch, cutoff and
+volume:
+
+```python
+sampler.voice[0].play(brass_c4, rate=44100, loop_start=2000, loop_end=6000,
+                      loop_mode=audiosampler.LOOP_UNTIL_RELEASE,
+                      attack=0.02, release=0.2,
+                      cutoff=800, resonance=6,             # Hz, dB
+                      mod_attack=0.05, mod_decay=0.4, mod_sustain=0.3,
+                      mod_to_cutoff=2400,                  # cents at full scale
+                      vibrato_delay=0.3, vibrato_rate=5.5, vibrato_depth=15)
+...
+sampler.voice[0].vibrato_depth = 50                        # a mod wheel
+sampler.voice[0].cutoff = 1200
+```
+
+`cutoff` is in Hz and `resonance` in dB: the filter's response at the cutoff
+sits that far above its response at DC, and DC drops by half of it, as the
+SoundFont specification puts it. Zero resonance is a Butterworth. A cutoff of
+19.9 kHz or more, with no resonance and nothing moving it, is no filter at
+all, and so is leaving it out.
+
+The modulation envelope has the amplitude envelope's six stages, prefixed
+`mod_`, and moves pitch by `mod_to_pitch` and the cutoff by `mod_to_cutoff`
+cents at full scale. The vibrato LFO (`vibrato_delay`, `vibrato_rate` in Hz,
+`vibrato_depth` in cents) moves pitch. The modulation LFO (`lfo_delay`,
+`lfo_rate`) moves pitch by `lfo_to_pitch` cents, the cutoff by
+`lfo_to_cutoff` cents and the level by `lfo_to_volume` dB at its peak. Both
+LFOs are triangles that start at zero, rising, once their delay is over.
+`cutoff` and `vibrato_depth` are live properties.
+
+These are worked out every 32 frames, which at 48 kHz is about fourteen
+times a cycle of the fastest LFO a SoundFont can ask for (110 Hz). A voice
+that uses none of them renders exactly as before and costs the same. One
+with a filter costs about 1.7 times as much on a desktop and twice as much
+on an ESP32-P4, where 48 filtered voices fit a 256-frame block and 64 only
+just do: see
+[what the filter and the LFOs cost](../how-it-works/upstream-diff.md#what-the-filter-and-the-lfos-cost).
+
+## How it renders
+
 Every voice is mixed in one C call per block, summed in float and quantised
 once, through a 4-point Hermite interpolator, so a loop at pitch 1.0 is a
 wire. For why the position is fixed point and how the pitch is verified by

docs/how-it-works/upstream-diff.md

@@ -70,8 +70,8 @@ run). On an ESP32-P4 at 400 MHz, 64 voices of a dense General MIDI file
 took about 5 ms of the block before the frames between a voice's events got
 a loop of their own (state in registers, two frames' interpolation worked
 out together for the in-order core, the same arithmetic in the same order,
-so the same bits): about 2.4 ms after. Memory is the voice table (about 100 bytes a voice, two
-slots each), a float accumulator of one block, and the output block. The
+so the same bits): about 2.4 ms after. Memory is the voice table (256 bytes a voice on a
+64-bit desktop since the filter and LFOs arrived, 112 before, two slots each), a float accumulator of one block, and the output block. The
 buffers are the caller's, held by reference while a voice sounds.
 
 A `Sampler` carries up to 128 voices (`audiosampler.MAX_VOICES`). Cost
@@ -84,6 +84,47 @@ renders voices at its own pace, so scale the P4's 64 voices (2.4 ms of a
 5.3 ms block) to what you ask of it, and give `voice_count` the number it
 can keep up with.
 
+### What the filter and the LFOs cost
+
+A voice that sets none of `cutoff`, `resonance`, the modulation envelope's
+depths or the LFOs' takes the path every voice took before they existed,
+and renders the same bits (`sampler_probe.py`'s blocks from before them
+match). One that does renders in 32-frame chunks: once a chunk it works out
+its pitch, its filter's coefficients and its volume, then runs the chunk
+through a two-pole state-variable low-pass. The coefficients take a
+polynomial tangent and `audiodsp_exp2_poly`, never `tanf` or `powf`, so
+every target renders the same bits.
+
+[`tests/parity/sampler_cost.py`](../../tests/parity/sampler_cost.py) times
+it on any target, in microseconds per 256-frame stereo block at 48 kHz
+(5333 µs), every voice looping: *plain* is no filter, *filter* a 2 kHz
+low-pass with 6 dB of resonance, *full* the filter with the modulation
+envelope on pitch and cutoff, the vibrato LFO, and the modulation LFO on
+pitch, cutoff and volume.
+
+| Voices | CPython plain | filter | full | MicroPython unix plain | filter | full |
+|---|---|---|---|---|---|---|
+| 32 | 32 | 56 | 65 | 79 | 104 | 112 |
+| 48 | 47 | 83 | 95 | 111 | 154 | 163 |
+| 64 | 62 | 102 | 122 | 152 | 207 | 225 |
+| 128 | 115 | 208 | 247 | 308 | 407 | 452 |
+
+(x86-64, one core, gcc -O2.)
+
+On an ESP32-P4 at 400 MHz (MicroPython, the script run over the REPL):
+
+| Voices | plain before | plain | filter | full |
+|---|---|---|---|---|
+| 32 | 1383 | 1384 | 2626 | 3465 |
+| 48 | 2039 | 2040 | 3904 | 5179 |
+| 64 | 2745 | 2804 | 5290 | 6970 |
+
+*Plain before* is the same firmware built from the commit before the
+filter. A board fits 48 filtered voices with room to spare, and 64 with
+almost none; with every LFO and the modulation envelope moving as well, 48
+still fit and 64 don't. A bank rarely asks for all of it at once on every
+voice, but size a board's `voice_count` with this table in mind.
+
 ### How it is verified
 
 [`tests/parity/sampler_probe.py`](../../tests/parity/sampler_probe.py) in `verify_dsp`'s set, for the three-target

src/audiosampler/SamplerVoice.c

@@ -19,6 +19,8 @@ mp_obj_t audiosampler_samplervoice_new(audiosampler_sampler_obj_t *parent,
     self->level = mp_obj_new_float(1.0);
     self->pan = mp_obj_new_float(0.0);
     self->pitch = mp_obj_new_float(1.0);
+    self->cutoff = mp_const_none;
+    self->vibrato_depth = mp_obj_new_float(0.0);
     return MP_OBJ_FROM_PTR(self);
 }
 
@@ -33,7 +35,8 @@ static mp_obj_t audiosampler_samplervoice_play(size_t n_args,
     const mp_obj_t *pos_args, mp_map_t *kw_args) {
     enum { ARG_buffer, ARG_rate, ARG_pitch, ARG_loop_start, ARG_loop_end,
            ARG_loop_mode, ARG_level, ARG_pan, ARG_delay, ARG_attack, ARG_hold,
-           ARG_decay, ARG_sustain, ARG_release };
+           ARG_decay, ARG_sustain, ARG_release,
+           ARG_cutoff, ARG_resonance, ARG_mod_delay, ARG_mod_attack, ARG_mod_hold, ARG_mod_decay, ARG_mod_sustain, ARG_mod_release, ARG_mod_to_pitch, ARG_mod_to_cutoff, ARG_vibrato_delay, ARG_vibrato_rate, ARG_vibrato_depth, ARG_lfo_delay, ARG_lfo_rate, ARG_lfo_to_pitch, ARG_lfo_to_cutoff, ARG_lfo_to_volume };
     static const mp_arg_t allowed_args[] = {
         { MP_QSTR_buffer, MP_ARG_REQUIRED | MP_ARG_OBJ, {.u_obj = MP_OBJ_NULL} },
         { MP_QSTR_rate, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
@@ -49,6 +52,24 @@ static mp_obj_t audiosampler_samplervoice_play(size_t n_args,
         { MP_QSTR_decay, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
         { MP_QSTR_sustain, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
         { MP_QSTR_release, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_cutoff, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_resonance, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_delay, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_attack, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_hold, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_decay, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_sustain, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_release, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_to_pitch, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_mod_to_cutoff, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_vibrato_delay, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_vibrato_rate, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_vibrato_depth, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_lfo_delay, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_lfo_rate, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_lfo_to_pitch, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_lfo_to_cutoff, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
+        { MP_QSTR_lfo_to_volume, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
     };
     audiosampler_samplervoice_obj_t *self = MP_OBJ_TO_PTR(pos_args[0]);
     audiosample_check_for_deinit(&self->parent->base);
@@ -88,6 +109,24 @@ static mp_obj_t audiosampler_samplervoice_play(size_t n_args,
     note.decay = float_arg(args[ARG_decay].u_obj, 0.0f);
     note.sustain = float_arg(args[ARG_sustain].u_obj, 1.0f);
     note.release = float_arg(args[ARG_release].u_obj, 0.0f);
+    note.cutoff = float_arg(args[ARG_cutoff].u_obj, 0.0f);
+    note.resonance = float_arg(args[ARG_resonance].u_obj, 0.0f);
+    note.mod_delay = float_arg(args[ARG_mod_delay].u_obj, 0.0f);
+    note.mod_attack = float_arg(args[ARG_mod_attack].u_obj, 0.0f);
+    note.mod_hold = float_arg(args[ARG_mod_hold].u_obj, 0.0f);
+    note.mod_decay = float_arg(args[ARG_mod_decay].u_obj, 0.0f);
+    note.mod_sustain = float_arg(args[ARG_mod_sustain].u_obj, 1.0f);
+    note.mod_release = float_arg(args[ARG_mod_release].u_obj, 0.0f);
+    note.mod_to_pitch = float_arg(args[ARG_mod_to_pitch].u_obj, 0.0f);
+    note.mod_to_cutoff = float_arg(args[ARG_mod_to_cutoff].u_obj, 0.0f);
+    note.vibrato_delay = float_arg(args[ARG_vibrato_delay].u_obj, 0.0f);
+    note.vibrato_rate = float_arg(args[ARG_vibrato_rate].u_obj, 0.0f);
+    note.vibrato_depth = float_arg(args[ARG_vibrato_depth].u_obj, 0.0f);
+    note.lfo_delay = float_arg(args[ARG_lfo_delay].u_obj, 0.0f);
+    note.lfo_rate = float_arg(args[ARG_lfo_rate].u_obj, 0.0f);
+    note.lfo_to_pitch = float_arg(args[ARG_lfo_to_pitch].u_obj, 0.0f);
+    note.lfo_to_cutoff = float_arg(args[ARG_lfo_to_cutoff].u_obj, 0.0f);
+    note.lfo_to_volume = float_arg(args[ARG_lfo_to_volume].u_obj, 0.0f);
 
     // Read and checked first, stored under the lock: a pull landing between
     // the kernel's play() and the reference store would read a buffer the
@@ -115,6 +154,9 @@ static mp_obj_t audiosampler_samplervoice_play(size_t n_args,
         ? mp_obj_new_float(0.0) : args[ARG_pan].u_obj;
     self->pitch = args[ARG_pitch].u_obj == mp_const_none
         ? mp_obj_new_float(1.0) : args[ARG_pitch].u_obj;
+    self->cutoff = args[ARG_cutoff].u_obj;
+    self->vibrato_depth = args[ARG_vibrato_depth].u_obj == mp_const_none
+        ? mp_obj_new_float(0.0) : args[ARG_vibrato_depth].u_obj;
     return mp_const_none;
 }
 static MP_DEFINE_CONST_FUN_OBJ_KW(audiosampler_samplervoice_play_obj, 1,
@@ -236,6 +278,61 @@ MP_PROPERTY_GETSET(audiosampler_samplervoice_pitch_obj,
     (mp_obj_t)&audiosampler_samplervoice_get_pitch_obj,
     (mp_obj_t)&audiosampler_samplervoice_set_pitch_obj);
 
+// The low-pass cutoff in Hz, None for no filter. Live: a change lands
+// within 32 frames, and a filter switched on starts from rest.
+static mp_obj_t audiosampler_samplervoice_obj_get_cutoff(mp_obj_t self_in) {
+    audiosampler_samplervoice_obj_t *self = MP_OBJ_TO_PTR(self_in);
+    audiosample_check_for_deinit(&self->parent->base);
+    return self->cutoff;
+}
+static MP_DEFINE_CONST_FUN_OBJ_1(audiosampler_samplervoice_get_cutoff_obj,
+    audiosampler_samplervoice_obj_get_cutoff);
+static mp_obj_t audiosampler_samplervoice_obj_set_cutoff(mp_obj_t self_in,
+    mp_obj_t value) {
+    audiosampler_samplervoice_obj_t *self = MP_OBJ_TO_PTR(self_in);
+    audiosample_check_for_deinit(&self->parent->base);
+    float cutoff = value == mp_const_none ? 0.0f
+        : (float)mp_obj_get_float(value);
+    audiodsp_pump_lock_acquire();
+    audiodsp_sampler_set_cutoff(&self->parent->sampler, self->index, cutoff);
+    self->cutoff = value;
+    audiodsp_pump_lock_release();
+    return mp_const_none;
+}
+static MP_DEFINE_CONST_FUN_OBJ_2(audiosampler_samplervoice_set_cutoff_obj,
+    audiosampler_samplervoice_obj_set_cutoff);
+MP_PROPERTY_GETSET(audiosampler_samplervoice_cutoff_obj,
+    (mp_obj_t)&audiosampler_samplervoice_get_cutoff_obj,
+    (mp_obj_t)&audiosampler_samplervoice_set_cutoff_obj);
+
+// The vibrato LFO's depth in cents. Live, so a mod wheel is a series of
+// writes here; each lands within 32 frames.
+static mp_obj_t audiosampler_samplervoice_obj_get_vibrato_depth(
+    mp_obj_t self_in) {
+    audiosampler_samplervoice_obj_t *self = MP_OBJ_TO_PTR(self_in);
+    audiosample_check_for_deinit(&self->parent->base);
+    return self->vibrato_depth;
+}
+static MP_DEFINE_CONST_FUN_OBJ_1(audiosampler_samplervoice_get_vibrato_depth_obj,
+    audiosampler_samplervoice_obj_get_vibrato_depth);
+static mp_obj_t audiosampler_samplervoice_obj_set_vibrato_depth(
+    mp_obj_t self_in, mp_obj_t value) {
+    audiosampler_samplervoice_obj_t *self = MP_OBJ_TO_PTR(self_in);
+    audiosample_check_for_deinit(&self->parent->base);
+    float cents = (float)mp_obj_get_float(value);
+    audiodsp_pump_lock_acquire();
+    audiodsp_sampler_set_vibrato_depth(&self->parent->sampler, self->index,
+        cents);
+    self->vibrato_depth = value;
+    audiodsp_pump_lock_release();
+    return mp_const_none;
+}
+static MP_DEFINE_CONST_FUN_OBJ_2(audiosampler_samplervoice_set_vibrato_depth_obj,
+    audiosampler_samplervoice_obj_set_vibrato_depth);
+MP_PROPERTY_GETSET(audiosampler_samplervoice_vibrato_depth_obj,
+    (mp_obj_t)&audiosampler_samplervoice_get_vibrato_depth_obj,
+    (mp_obj_t)&audiosampler_samplervoice_set_vibrato_depth_obj);
+
 // `deinit()` on a voice is `stop()`, with no fade, plus letting go of the
 // buffer: a voice
 // has nothing of its own to release, it belongs to its Sampler, but every
@@ -267,6 +364,9 @@ static const mp_rom_map_elem_t audiosampler_samplervoice_locals_table[] = {
     { MP_ROM_QSTR(MP_QSTR_level), MP_ROM_PTR(&audiosampler_samplervoice_level_obj) },
     { MP_ROM_QSTR(MP_QSTR_pan), MP_ROM_PTR(&audiosampler_samplervoice_pan_obj) },
     { MP_ROM_QSTR(MP_QSTR_pitch), MP_ROM_PTR(&audiosampler_samplervoice_pitch_obj) },
+    { MP_ROM_QSTR(MP_QSTR_cutoff), MP_ROM_PTR(&audiosampler_samplervoice_cutoff_obj) },
+    { MP_ROM_QSTR(MP_QSTR_vibrato_depth),
+      MP_ROM_PTR(&audiosampler_samplervoice_vibrato_depth_obj) },
 };
 static MP_DEFINE_CONST_DICT(audiosampler_samplervoice_locals,
     audiosampler_samplervoice_locals_table);

src/audiosampler/SamplerVoice.h

@@ -27,6 +27,8 @@ typedef struct {
     mp_obj_t level;
     mp_obj_t pan;
     mp_obj_t pitch;
+    mp_obj_t cutoff;
+    mp_obj_t vibrato_depth;
 } audiosampler_samplervoice_obj_t;
 
 extern const mp_obj_type_t audiosampler_samplervoice_type;