From e068e71592c8e97564ace8a08b242118f3c3fd54 Mon Sep 17 00:00:00 2001 From: Brad Barnett <127794626+bdbarnett@users.noreply.github.com> Date: Wed, 28 Oct 2026 01:08:47 -0500 Subject: [PATCH] Shorten AGENTS.md to tips; the parity-gate and oracle rules move to a docs page --- AGENTS.md | 231 ++---------------------------- CIRCUITPYTHON_ORACLE | 6 +- CLAUDE.md | 1 + docs/how-it-works/index.md | 1 + docs/how-it-works/parity-gates.md | 90 ++++++++++++ docs/index.md | 1 + docs/porting-plan.md | 2 +- 7 files changed, 108 insertions(+), 224 deletions(-) create mode 100644 CLAUDE.md create mode 100644 docs/how-it-works/parity-gates.md diff --git a/AGENTS.md b/AGENTS.md index cddb643..5d98fb4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,220 +1,11 @@ -# AGENTS.md — audiodsp - -CircuitPython's audio system (`audiocore`, `synthio`, `audiomixer`, effects, -`audiomp3`), ported to MicroPython as `USER_C_MODULES` usermods. Import -names match CircuitPython exactly (`import audiocore`, `import synthio`, …) -for source compatibility; only this repo's own name differs. - -## Layout - -- Root: `micropython.mk` (unix/windows, Make-based ports), `micropython.cmake` - (esp32/rp2, CMake-based ports) — build glue for `USER_C_MODULES` discovery -- `src/` — one directory per module (`audiocore/`, `synthio/`, `audiomixer/`, - `audiospeed/`, `audiofreeverb/`, `audiofilters/`, `audiodelays/`, - `audiomp3/`, `audiodynamics/`, `audioroute/`, `audiomath/`, `audioecho/`, - `audioconvolve/`, `audiobiquad/`, `audioshaper/`, `audioladder/`, - `audiooscillator/`, - `audioverb/`, `audiomodal/`, `audiosampler/`, `audiovorbis/`), plus - `src/cp_compat/` - (CircuitPython-only core primitives ported as standalone compat shims, each - individually verified against mainline MicroPython before use — not assumed - missing) and `src/shared/` (runtime-neutral DSP the MicroPython usermod and - the CPython extension both compile) -- `src/cpython/` — the whole CPython target: `_audiodsp.c`, the extension built - in place, and the thirteen modules that wrap it (`audiocore.py`, - `synthio.py`, - …). They install as top-level modules, so it is `import audiocore` no matter - which of the three runtimes is underneath. Nothing puts this directory on - `sys.path`: audiodsp is a dependency, imported from wherever it is installed. -- `lib/` — the pure-Python tier: `lib/audiorender/` (whole-composition - offline rendering — numpy, desktop-only, never frozen) and - `lib/audiodsp_util/` (`float32`, the round trip that makes a setting derived - in Python the same number on a board as on a desktop — see - [docs/how-it-works/correctness-standard.md](docs/how-it-works/correctness-standard.md)). Both ship - inside the `omnipython-audiodsp` wheel. The instrument and effect libraries that - used to sit beside it — `audioinstruments` (55 `synthio` instruments) and - `audioeffects` (45 effect classes, racks included) — live in - [audiocomponents](https://github.com/PyDevices/audiocomponents) now, as - their own distributions depending on `omnipython-audiodsp`; nothing in this - repository builds, tests, publishes or freezes them. -- `circuitpython.mk` + `src/circuitpython_spike/` — build `audiodynamics`, - `audioroute`, `audiomath`, `audioecho`, `audioshaper`, `audioladder`, - `audioconvolve`, `audiobiquad`, `audioverb`, `audiomodal`, `audiosampler`, - `audiooscillator`, `audiovorbis`, `audiopump` and `audiograph` into - CircuitPython 11 as a user C module (`micropython.mk` hands a CircuitPython - tree to `circuitpython.mk`). None of the fifteen is a CircuitPython port: the - first two come from micropython-vst3's `vstaudio` engine and the last - thirteen are audiodsp's own, - so CircuitPython gains them - here rather than the other way round. -- `docs/porting-plan.md` — the full phased porting history, architecture, - and target layout -- `docs/how-it-works/upstream-sync.md` — when and how to resync the CircuitPython-derived - modules with upstream, what the last resync took, and the list of every - deliberate difference (the pump, `deinit`, the recorded deviations) -- `docs/how-it-works/upstream-diff.md` — the reasons and measurements behind each - deviation from upstream CircuitPython (verbatim-kept quirks vs. genuine port - bugs found and fixed) -- `tests/parity/` — oracle-diff scripts, run unchanged against both this - port and `bin/circuitpython` -- `tests/vendor/` — vendored third-party test fixtures (e.g. `synthtools`) - -- `third_party/tremor/` — Xiph's Tremor (lowmem), the integer Vorbis decoder - `audiovorbis` wraps, BSD-3-Clause, vendored (see its `ORIGIN.txt`). Never - listed in a build on its own: `src/shared/audiodsp_vorbis.c` includes it - into one translation unit with every symbol renamed. See - [docs/reference/vorbis.md](docs/reference/vorbis.md). - -## Sibling dependencies (cloned, not vendored) - -- `mp3` (upstream `adafruit/Adafruit_MP3`, the Helix MP3 decoder core - `audiomp3` wraps) — RPSL 1.0/RCSL 1.0 licensed, *not* MIT; carried - unmodified per upstream's own terms, same as CircuitPython itself. Kept - as a separate sibling clone rather than folded into this (MIT) repo's own - tree. - -Both are expected under `.deps/` (`scripts/fetch_deps.sh`), or as siblings -of this repository in the parent workspace, same pattern as -`pygraphics`/`displayif`. - -## Testing - -- Oracle diffing is the core discipline: every parity script under - `tests/parity/` runs unchanged against this port and `bin/circuitpython`, - rendering PCM and diffing byte-for-byte (or documenting the exact, - bounded exception in `docs/how-it-works/upstream-diff.md`). -- The nine modules that are **ours** have no oracle, by decision - ([docs/how-it-works/correctness-standard.md](docs/how-it-works/correctness-standard.md), 2026-09-09). - Upstream CircuitPython has no counterpart to them, so they are held to two - things: every target renders them identically, and their numeric traits hold. - `.venv/bin/python tests/parity/verify_dsp.py --micropython ../micropython/ports/unix/build-pydevices/micropython` - is the first. There is **no stored digest** — the gate is the comparison, and - it **refuses a run with fewer than two interpreters** rather than passing one - that cannot fail. Add `--circuitpython ../bin/circuitpython-oracle-11.0.0-alpha.2` - for the three-way. The arithmetic is all in `src/shared/`, so two - interpreters disagreeing is never a difference of intent: it is a width, an - undefined shift, a compiler's choice or an architecture. -- The traits are the other half, one file per module — - `tests/test_cpython_.py`, each opening with its trait table and the - bar for every row. Agreement cannot see a change that moves all three targets - together; the traits can, because they are measured against arithmetic rather - than against our own last answer. `audiobiquad`'s probe also prints two - invariants as integers beside its PCM (the block at which a tail reaches - exact zero, and the null depth at each feedback value), because a comparison - over PCM alone would not say whether either still held. -- **The vstaudio oracle is retired.** `audiodynamics` and `audioroute` came - from micropython-vst3's engine and used to be held to `vstaudio_dsp.c` - compiled unmodified. That file was deleted from micropython-vst3 in `6ea60d3` - and the plug-in links audiodsp now: the relationship reversed, so the engine is - a consumer of this package rather than a grader of it. The build script, the - usermod and `golden/dsp_nodes.json` are gone. -- The instruments parity gate — `run_instruments_parity.py`, its two probes, - `instrument_sequences.py` and the `instruments_*.json` digests — lives in - [audiocomponents](https://github.com/PyDevices/audiocomponents) now, under - its `tests/parity/`, beside the packages it renders. Its `REBUILT` rule (a - name there records a sound changed **on purpose**, and adding one is - the maintainer's call, never an agent's) is documented in that repository's - AGENTS.md. It is not run from here. -- `python3 -m flake8` is the lint gate (`.flake8`, defect checks only — - layout is deliberately not gated). It runs in CI on every push. -- Full regression after any change: rebuild interpreters - (`build_interpreters.sh` in the parent workspace), run the tier 0-5 - parity suite plus `tests/parity/synthtools_acceptance.py`, and the LVGL - smoke test. -- See `docs/porting-plan.md`'s "Testing strategy" section for the complete - methodology. -- **The parity/oracle gates are workspace-local by design.** They need - built interpreters (`bin/circuitpython`, the workspace MicroPython) and - golden captures that live outside this repository, so external - contributors cannot run them and CI does not try. **Since 2026-09-03 the - scripts' zero-argument defaults no longer point into the workspace** — a - standalone user should not be steered into a directory only this - workspace has — so in *this* workspace every parity command needs the - explicit `--micropython`/`--circuitpython` (or `CP_DIR`, `MP_UNIX`, - `ULAB_DIR`) override shown above. `verify_dsp.py` used to print `skipping - micropython (not built at ...)` and carry on with whatever was left, which - was not a pass; it now **refuses** a run with fewer than two interpreters, - because one interpreter cannot disagree with itself and a run that cannot - fail is worse than no run. What CI - covers instead is what needs only the wheel: the CPython fixture tests in - `tests/test_cpython_*.py` and the five in-repo parity gates - (`verify_acceptance`, `verify_effects`, `verify_streaming`, - `verify_biquad`, `verify_mixdown_knee`), whose goldens are committed - here. The fifth (2026-09-06, audiodsp#27) is the only one whose material - crosses the mix-down limiter's +/-28000 knee, so it is the only one that - can see a synthio voice-ceiling change at all — the other four are - byte-identical at every ceiling value. It sees exactly one of the - ceiling's five sites, `src/cpython/synthio.py`'s `max_polyphony`, because - that is the only one the CPython target reads; `tests/ - test_voice_ceiling_consistency.py` is still the guard for a half-applied - change. Its above-knee half is enforced against **this port**, not - against the oracle: the port is built at 64 and the pinned oracle at 14, - and no above-knee material can agree across those two. The component - contract tests (`test_audio_component_api`, `test_metadata_contract`, - `tools/validate_api.py`) went to audiocomponents with the packages they - check. -- **Two kinds of golden, two rules (2026-09-03).** The instruments - digests — `instruments_*.json`, now under audiocomponents' - `tests/parity/golden/` — record that *the port matches the pre-rewrite - original script, within one interpreter*: `run_instruments_parity.py` - there renders the originals from micropython-vst3 at `DEFAULT_OLD_REV` and - never consults the CircuitPython oracle. So a change to `src/cpython/` - *here* that is *right* still stales those cpython digests (audiodsp#25: - `b420dac` did exactly this). Rule: a CPython-target fix may re-capture the - affected cpython digests **only if it carries independent evidence against - the built oracle** — a test in `tests/test_cpython_*.py` run against - `bin/circuitpython` and cited in the message. The digests living in the - other repository changes only the mechanics: the audiocomponents - re-capture names the audiodsp commit that carries that evidence. A fix that - merely *asserts* oracle intent does not qualify; that would let it rewrite - its own reference. The accuracy program's *listening* goldens - (audiocomponents) are a different authority — a listener's ear — and move only - at his phrase. The stored digest is also the only thing in that gate that - notices the engine moving under both original and port — which is why it - stays: a live original-vs-port comparison was measured, approved and then - reversed the same day (audiodsp#26), because it would have stayed green - through `b420dac`. Each alarm costs one adjudication; that is the price of - the signal. - -## The CircuitPython oracle — extend, never modify - -CircuitPython at the tag in `CIRCUITPYTHON_ORACLE` is the **oracle** every -parity golden is measured against. The rule, for any agent working here: - -- **The oracle binary is `bin/circuitpython-oracle-` beside the sibling checkouts**, - built only by the interpreter build script's `--only cp-oracle` (CircuitPython's unix - coverage variant at `CIRCUITPY_SYNTHIO_MAX_CHANNELS=64`, with this repository as - its only audio user C module, through micropython-pydevices' `build_mp.py - --interpreter circuitpython`) and re-pinned in - `tests/test_voice_ceiling_consistency.py` in the same change that builds it. - **`bin/circuitpython` is not the oracle** — it is what that script's - `cp-unix` target installs, at the coverage variant's own 14-voice ceiling, - and it changes under you whenever anyone refreshes the interpreters - (audiodsp#89, twice in eight days). -- **Never edit CircuitPython's own sources for the oracle.** A modified oracle - silently redefines what "parity" means and invalidates every golden without - failing anything. Its *pin* is a different matter: it moves when this port - moves to a new CircuitPython release, deliberately, in a change that re-reads - every CP-shared gate and re-pins the binary's hash with the reason written - down. **Every pin move is a resync**: take every audio fix and every - addition (a new property, argument or class) upstream made since the old - pin, following `docs/how-it-works/upstream-sync.md`, and update its record of what was - taken and what differs on purpose. The oracle is built at the same version and the same ceiling this port - ships — an oracle at a different configuration cannot answer the only question - worth asking. See `docs/how-it-works/correctness-standard.md` and - `tests/test_voice_ceiling_consistency.py`. -- **Extending CircuitPython is fine and is the designed path**: new modules - live in this repo under `src/circuitpython_spike/` and build, with - `src/shared/` DSP, as a user C module (`circuitpython.mk`). That is - **additive-only by design**: it adds modules CircuitPython lacks and touches - none of its own. Never make it shadow or rewrite a CircuitPython module, - because the oracle is built with it, so a behavioral change would leak - *into* the oracle. -- Fixes to bugs that also exist upstream go in **this repo's targets only** - (MicroPython/CPython/`src/shared/`), recorded in `docs/how-it-works/upstream-diff.md` - — never into the CP tree. Approved deviations from the oracle are - enumerated there; **ask before adding one**. -- Quick self-check after any CP-adjacent work: `circuitpython.mk` compiles - only the ten modules above and `src/shared/`, never a file under - CircuitPython's own `shared-module/` or `shared-bindings/` for `synthio`, - `audiofilters`, `audiocore`, `audiomixer` or `audiodelays`. +# AGENTS.md + +- CPython tests need the native extension built (`python -m build`, then install the wheel, or an editable install). Run the module you changed: `python -m unittest tests/test_cpython_.py`. Don't run the whole suite for a small change; it is long. +- The five gates CI runs: `python tests/parity/verify_acceptance.py`, `verify_effects.py`, `verify_streaming.py`, `verify_biquad.py` and `verify_mixdown_knee.py`. Lint is `python -m flake8` (defect checks only). +- The deeper parity gates need built interpreters and the pinned CircuitPython oracle. Pass `--micropython` and `--circuitpython` explicitly; `verify_dsp.py` refuses a run with fewer than two interpreters. A stock `circuitpython` build is not the oracle (it has 14 voices; the oracle has 64). See [Running the parity gates](docs/how-it-works/parity-gates.md). +- Never edit CircuitPython's own sources for the oracle, and never let a new module shadow one of CircuitPython's. Fixes for bugs that also exist upstream go in this repo's targets only, recorded in [upstream-diff](docs/how-it-works/upstream-diff.md); ask before adding a deviation. +- Don't re-capture a golden, or move `CIRCUITPYTHON_ORACLE` or `DEPENDENCIES.lock`, without independent evidence and the reason written down ([CONTRIBUTING.md](CONTRIBUTING.md#rules-that-keep-the-port-honest)). +- The `audiomp3` decoder is fetched, not vendored: `scripts/fetch_deps.sh` puts it under `.deps/`. Tremor (`audiovorbis`) is vendored in `third_party/tremor`. +- Import names match CircuitPython (`import audiocore`, `import synthio`), whichever runtime is underneath. The instrument and effect libraries live in [audiocomponents](https://github.com/PyDevices/audiocomponents), not here. +- A frozen module wins over `/lib` on MicroPython and CircuitPython, so a copied or mip-installed `audiodsp` never runs until you rebuild or put `/lib` first. On a source checkout set `PYTHONPATH` (CPython) or `MICROPYPATH` (MicroPython). +- What each module is held to: [the correctness standard](docs/how-it-works/correctness-standard.md). Where everything is: [How audiodsp works](docs/how-it-works/index.md). diff --git a/CIRCUITPYTHON_ORACLE b/CIRCUITPYTHON_ORACLE index 04424c4..b940bef 100644 --- a/CIRCUITPYTHON_ORACLE +++ b/CIRCUITPYTHON_ORACLE @@ -4,7 +4,7 @@ # 10.3.0 on 2026-09-09). # Moving it is a re-port, not a bump: every move is a resync with upstream # (docs/how-it-works/upstream-sync.md; the one before this was audiodsp#220). See also -# AGENTS.md "The CircuitPython oracle" and docs/how-it-works/upstream-diff.md. The 64-voice oracle binary built from -# it lives at bin/circuitpython-oracle-11.0.0-alpha.2 (build_interpreters.sh -# cp-oracle); tests/test_voice_ceiling_consistency.py pins its sha256. +# docs/how-it-works/parity-gates.md (how the 64-voice oracle binary is built) +# and docs/how-it-works/upstream-diff.md; tests/test_voice_ceiling_consistency.py +# pins the binary's sha256. 11.0.0-alpha.2 d53b2277cfad71589028e5da77c984c045db990d diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/docs/how-it-works/index.md b/docs/how-it-works/index.md index 5100e91..cfd6d71 100644 --- a/docs/how-it-works/index.md +++ b/docs/how-it-works/index.md @@ -132,6 +132,7 @@ push-PCM transports. - [Additions beyond CircuitPython](additions.md): why each added module exists. - [Sound stability](sound-stability.md): what is stable across releases, and what isn't. - [The correctness standard](correctness-standard.md): how each module is held to a reference. +- [Running the parity gates](parity-gates.md): the oracle binary, the deeper gates and the golden rules. - [Keeping in step with upstream](upstream-sync.md): the resync procedure, and every deliberate difference. - [Differences from upstream, with measurements](upstream-diff.md): the reasons behind each one. - [Upstream reports](../upstream-reports/README.md): the fixes sent to CircuitPython. diff --git a/docs/how-it-works/parity-gates.md b/docs/how-it-works/parity-gates.md new file mode 100644 index 0000000..e48607c --- /dev/null +++ b/docs/how-it-works/parity-gates.md @@ -0,0 +1,90 @@ +# Running the parity gates + +CI runs the CPython fixture tests and the five parity gates whose goldens live +in this repository (`verify_acceptance`, `verify_effects`, `verify_streaming`, +`verify_biquad`, `verify_mixdown_knee`; see [Contributing](https://github.com/PyDevices/audiodsp/blob/main/CONTRIBUTING.md#test)). +The deeper gates need built interpreters and the pinned CircuitPython oracle, +so CI doesn't run them. This page is for when you do. What each module is held +to is in [the correctness standard](correctness-standard.md). + +## The oracle binary + +CircuitPython at the tag in `CIRCUITPYTHON_ORACLE` is the oracle every +CircuitPython-shared parity golden is measured against. Build it as +CircuitPython's unix coverage variant with `CIRCUITPY_SYNTHIO_MAX_CHANNELS=64` +and this repository as its only audio user C module (`circuitpython.mk`), at +the same version and voice ceiling this port ships. An oracle at another +configuration can't answer the only question worth asking. Re-pin its hash in +`tests/test_voice_ceiling_consistency.py` in the same change that builds it. + +A stock `circuitpython` unix build is not the oracle: the coverage variant's own +ceiling is 14 voices, and it changes whenever the interpreter is rebuilt. + +Never edit CircuitPython's own sources for the oracle. A modified oracle +redefines what parity means and invalidates every golden without failing +anything. The pin moves only when the port moves to a new CircuitPython +release, in a change that re-reads every CircuitPython-shared gate and writes +the reason down. Every pin move is a resync: take every audio fix and addition +upstream made since the old pin, following [upstream-sync](upstream-sync.md). + +Extending CircuitPython is the designed path, and it is additive only. New +modules live under `src/circuitpython_spike/` and build with `src/shared/` as a +user C module. Never make one shadow or rewrite a CircuitPython module, because +the oracle is built with it and a behavioral change would leak into the oracle. +After any CircuitPython-adjacent work, check that `circuitpython.mk` compiles +only this repository's modules and `src/shared/`, never a file under +CircuitPython's `shared-module/` or `shared-bindings/` for `synthio`, +`audiofilters`, `audiocore`, `audiomixer` or `audiodelays`. + +## Running them + +Every script under `tests/parity/` runs unchanged against this port and the +oracle, renders PCM and diffs it byte for byte, or documents the exact bounded +exception in [upstream-diff](upstream-diff.md). The scripts' zero-argument +defaults don't point at any directory, so pass `--micropython` and +`--circuitpython`, or set `CP_DIR`, `MP_UNIX` or `ULAB_DIR`. + +The nine modules that are ours have no oracle +([the standard](correctness-standard.md)). They are held to two things: every +target renders them identically, and their numeric traits hold. + +```sh +python tests/parity/verify_dsp.py --micropython +``` + +Add `--circuitpython ` for the three-way. There is no stored digest: the +gate is the comparison, and it refuses a run with fewer than two interpreters, +because one interpreter can't disagree with itself. The arithmetic is all in +`src/shared/`, so two interpreters disagreeing is never a difference of intent; +it is a width, an undefined shift, a compiler's choice or an architecture. + +The traits are the other half, one file per module: +`tests/test_cpython_.py`, each opening with its trait table and the bar +for every row. Agreement can't see a change that moves all three targets +together; the traits can, because they are measured against arithmetic rather +than against our own last answer. + +`verify_mixdown_knee` is the only gate whose material crosses the mix-down +limiter's +/-28000 knee, so it is the only one that can see a synthio voice +ceiling change. It sees one of the ceiling's five sites, `max_polyphony` in +`src/cpython/synthio.py`, because that is the only one the CPython target reads; +`tests/test_voice_ceiling_consistency.py` guards against a half-applied change. +Its above-knee half is enforced against this port, not the oracle: the port is +built at 64 voices and the pinned oracle at the same ceiling it ships. + +## Two kinds of golden + +The instruments digests (`instruments_*.json`, under +[audiocomponents](https://github.com/PyDevices/audiocomponents)' +`tests/parity/golden/`) record that the port matches the pre-rewrite original +script within one interpreter. They never consult the oracle. So a correct +change to `src/cpython/` here can still stale those CPython digests. + +A CPython-target fix may re-capture the affected digests only if it carries +independent evidence against the built oracle: a test in +`tests/test_cpython_*.py` run against the oracle and cited in the commit +message. The audiocomponents re-capture names the commit here that carries that +evidence. A fix that merely asserts oracle intent doesn't qualify, since that +would let it rewrite its own reference. The stored digest is the only thing in +that gate that notices the engine moving under both the original and the port, +so it stays. Each alarm costs one adjudication; that is the price of the signal. diff --git a/docs/index.md b/docs/index.md index b6cd301..f161373 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,6 +37,7 @@ have. Start with install, then your first program. - [Additions beyond CircuitPython](how-it-works/additions.md) - [Sound stability](how-it-works/sound-stability.md) - [The correctness standard](how-it-works/correctness-standard.md) +- [Running the parity gates](how-it-works/parity-gates.md) - [Keeping in step with upstream](how-it-works/upstream-sync.md) - [Differences from upstream, with measurements](how-it-works/upstream-diff.md) - [Upstream reports](upstream-reports/README.md): the fixes sent to CircuitPython. diff --git a/docs/porting-plan.md b/docs/porting-plan.md index 292b6fe..82e03b6 100644 --- a/docs/porting-plan.md +++ b/docs/porting-plan.md @@ -182,7 +182,7 @@ per-port DMA drivers — none port directly. Parity means reproducing their - **Adapter (optional):** audiosample → `audiodev.PCMOutput.write()` for wasm/web and any host audiodev already supports. -Teardown follows workspace rules (`displayif/AGENTS.md`): any output owning +Teardown follows displayif's soft-reset rules: any output owning host resources (SDL device, I2S instance) registers soft-reset teardown and supports idempotent `deinit`.