feat: embed feature — minizlib-backed deflate family for embedded targets - #133
Merged
Merged
Conversation
…argets Adds an `embed` Cargo feature: a mode under which the algorithms that have a small variant are built as that variant, under the same names, so downstream code compiles either way. The first (and so far only) such family is deflate/zlib/gzip, backed by the `minizlib` crate (same author): its push decoder (puff-style, ~1.1 KiB of state) and greedy fixed-Huffman compressor, wrapped onto compcol's `RawEncoder` / `RawDecoder` traits. - No allocation: `deflate`, `zlib` and `gzip` no longer imply `alloc`. The standard build still needs it and says so in a `compile_error!` when neither `alloc` nor `embed` is on (a breaking change for `--no-default-features --features gzip` builds that relied on the implied `alloc`). - The wrappers own everything (32 KiB window, 4 KiB block buffer, output buffer, 1024-entry hash table), are `const`-constructible with an all-zero state so a `static` lands in `.bss`, index nothing that could panic, and bound the input fed per call so the worst-case deflate expansion (129 bytes per bit) can never overrun the window. - `compcol::embed` documents what differs (fixed ratio, no sync flush, no preset dictionary, first gzip member only) and the sizes CI enforces. - `tools/footprint/`: a bare-metal thumbv7em-none-eabi harness and `check.sh`, which fails if code, static RAM or stack (deepest call path, read from the disassembly by `stack.py`) exceed the documented ceilings, if panic machinery is linked, or if the codec `static` leaves `.bss`. Measured: decoders 4.0–4.4 KB of code, encoders 1.6–1.9 KB, under 100 bytes of stack. - `tools/embed-crosscheck.sh`: the embed and standard CLI builds must read each other's gzip/zlib/deflate streams, both ways. - `tests/embed.rs`: reference streams from CPython's zlib plus hand-made ones (a back-reference at exactly 32768, a block whose every two bits stand for 258 bytes), every chunking, error reporting, reset/reuse, struct-size ceilings and a 16 KiB-stack thread. - CI: `--features all` for the standard runs, `--all-features` (which turns `embed` on) as separate steps, plus footprint and cross-check jobs. docs.rs and the release binaries build with `all`, not `--all-features`, so they keep the standard codecs. - tests/cli.rs feeds stdin from a thread: with the embed ratio the child's output exceeded the pipe buffer and the old helper deadlocked. TEMPORARY: minizlib's new API (output accessors, owned tables, const constructors) is on its master but not yet released, so `[patch.crates-io]` pins that commit. Once minizlib 0.1.2 is on crates.io, drop the patch in Cargo.toml and tools/footprint/Cargo.toml and require "0.1.2". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dependency The `embed` deflate/zlib/gzip codecs are now compcol's own code under `src/embed/` — `inflate` (bit reader, canonical Huffman, history window), `decoder` (the push state machine), `encoder` (greedy LZ77 + fixed Huffman), `format` (the three containers) and `checksum` (64-byte-table CRC-32, Adler-32) — written straight against the crate's error type and `RawEncoder` / `RawDecoder` traits. The code was absorbed from the author's minizlib crate, which is being retired; the dependency and the temporary `[patch.crates-io]` are gone, and compcol is back to tokio as its only (optional) dependency. Being native lets both halves drop what the wrapper had to work around from outside: - The decoder writes straight into the caller's slice. A match copy that runs out of output keeps what is left as its next state, so no input feed bound and no ring-to-caller copy are needed, and gzip members concatenate exactly as in the standard build (parked between members until the next byte or `finish` decides). - The encoder queues bits in a 64-bit buffer and moves whole bytes out as room allows, so any non-empty output buffer makes progress and the 4.6 KiB worst-case output buffer is gone: an encoder is 6.2 KB. - Errors are the standard build's: `Unsupported` for a bad CM or reserved gzip flags, `TrailerMismatch` for a wrong ISIZE, `Corrupt` for a stored block whose length check fails. Eight standard-suite tests that were gated off under `embed` run again. Footprint on thumbv7em-none-eabi: decoders 3 672–4 036 bytes of code (was 4 037–4 389), encoders 1 514–1 844 (was 1 555–1 871), both 5 642 (was 6 147), stack under 100 bytes, no panics, statics in `.bss`. The ceilings in `tools/footprint/check.sh`, `compcol::embed` and the README are tightened to match. Also stops tracking the harness's build directory and lockfile, which the first commit swept in. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An
embedCargo feature: a mode under which algorithms with a small variant are built as that variant under the same names, so the same downstream code compiles either way. First family:deflate/zlib/gzip, as native compcol code undersrc/embed/(the code was absorbed from the author's minizlib crate, which is being retired): a push decoder in the manner of zlib'spuff(~1.1 KiB of state between pieces, Huffman codes decoded bit by bit from the count of codes per length, no tables to build) and a greedy fixed-Huffman LZ77 encoder, both written straight against theRawEncoder/RawDecodertraits. No new dependency.deflate/zlib/gzipno longer implyalloc; the standard buildcompile_error!s with guidance when neitherallocnorembedis on. Breaking for--no-default-features --features gzipbuilds that relied on the impliedalloc.const-constructible with an all-zero state (astaticlands in.bss), and index nothing that can panic.compcol::embeddocuments the trade-offs (fixed ratio, no sync flush, no preset dictionary) and the size contract.Sizes CI now enforces (thumbv7em-none-eabi,
opt-level = "z", LTO)Zero panic symbols, zero
.data, no static RAM beyond the codec struct.tools/footprint/check.sh(new CI job) measures code/RAM/stack and fails on any ceiling; stack is the deepest call path from the entry point, read off the disassembly (stack.py), since-stack-size-sectionis not reachable from stable rustc.Tests
tests/embed.rs: CPython-zlib reference streams (dynamic/fixed/stored blocks, gzip with every optional header field, incompressible, empty), hand-built streams (a back-reference at exactly 32768, which zlib never emits; a "bomb" block where every two bits decode to 258 bytes), every input/output chunking, error variants, poison/reset/reuse,discard_output, concatenated gzip members, struct-size ceilings, and a 16 KiB-stack thread.tools/embed-crosscheck.sh(new CI job): the embed and standard CLI builds decompress each other's gzip/zlib/deflate output, both ways.--features alland--all-features); standard-only tests (levels, FLEVEL/XFL, sync flush, dictionaries, one fixture with an RFC-invalidCINFO=15) are gated onnot(embed).tests/cli.rsnow feeds stdin from a thread: the embed ratio pushed the child's output past the pipe buffer and the old helper deadlocked.Also: docs.rs and the release-binaries workflow now build with
features = ["all"]rather than--all-features, which would otherwise ship the embed codecs.🤖 Generated with Claude Code