Skip to content

feat: embed feature — minizlib-backed deflate family for embedded targets - #133

Merged
MagicalTux merged 2 commits into
masterfrom
embed
Oct 3, 2026
Merged

MagicalTux merged 2 commits into
masterfrom
embed

Conversation

@MagicalTux

@MagicalTux MagicalTux commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

What

An embed Cargo feature: a mode under which algorithms with a small variant are built as that variant under the same names, so the same downstream code compiles either way. First family: deflate / zlib / gzip, as native compcol code under src/embed/ (the code was absorbed from the author's minizlib crate, which is being retired): a push decoder in the manner of zlib's puff (~1.1 KiB of state between pieces, Huffman codes decoded bit by bit from the count of codes per length, no tables to build) and a greedy fixed-Huffman LZ77 encoder, both written straight against the RawEncoder / RawDecoder traits. No new dependency.

compcol = { version = "0.6", default-features = false, features = ["embed", "gzip"] }
  • No allocation. deflate/zlib/gzip no longer imply alloc; the standard build compile_error!s with guidance when neither alloc nor embed is on. Breaking for --no-default-features --features gzip builds that relied on the implied alloc.
  • Codec structs own everything (decoder ≈ 34 KB, encoder ≈ 6 KB), are const-constructible with an all-zero state (a static lands in .bss), and index nothing that can panic.
  • The decoder writes straight into the caller's slice (a match copy interrupted by a full output resumes as its own state); the encoder queues bits in a 64-bit buffer so any non-empty output buffer makes progress.
  • Same error variants and gzip multi-member semantics as the standard build. compcol::embed documents the trade-offs (fixed ratio, no sync flush, no preset dictionary) and the size contract.

Sizes CI now enforces (thumbv7em-none-eabi, opt-level = "z", LTO)

configuration code max stack max
gzip decode 3964 4200 88 256
zlib decode 4036 4300 92 256
raw deflate decode 3672 3900 84 256
gzip encode 1844 2000 96 256
zlib encode 1804 2000 96 256
raw deflate encode 1514 1700 76 256
gzip encode + decode 5642 6000 96 256

Zero panic symbols, zero .data, no static RAM beyond the codec struct. tools/footprint/check.sh (new CI job) measures code/RAM/stack and fails on any ceiling; stack is the deepest call path from the entry point, read off the disassembly (stack.py), since -stack-size-section is not reachable from stable rustc.

Tests

  • tests/embed.rs: CPython-zlib reference streams (dynamic/fixed/stored blocks, gzip with every optional header field, incompressible, empty), hand-built streams (a back-reference at exactly 32768, which zlib never emits; a "bomb" block where every two bits decode to 258 bytes), every input/output chunking, error variants, poison/reset/reuse, discard_output, concatenated gzip members, struct-size ceilings, and a 16 KiB-stack thread.
  • tools/embed-crosscheck.sh (new CI job): the embed and standard CLI builds decompress each other's gzip/zlib/deflate output, both ways.
  • Existing suites run in both modes (--features all and --all-features); standard-only tests (levels, FLEVEL/XFL, sync flush, dictionaries, one fixture with an RFC-invalid CINFO=15) are gated on not(embed).
  • tests/cli.rs now feeds stdin from a thread: the embed ratio pushed the child's output past the pipe buffer and the old helper deadlocked.

Also: docs.rs and the release-binaries workflow now build with features = ["all"] rather than --all-features, which would otherwise ship the embed codecs.

🤖 Generated with Claude Code

MagicalTux and others added 2 commits October 3, 2026 14:50
…argets

Adds an `embed` Cargo feature: a mode under which the algorithms that
have a small variant are built as that variant, under the same names, so
downstream code compiles either way. The first (and so far only) such
family is deflate/zlib/gzip, backed by the `minizlib` crate (same author):
its push decoder (puff-style, ~1.1 KiB of state) and greedy fixed-Huffman
compressor, wrapped onto compcol's `RawEncoder` / `RawDecoder` traits.

- No allocation: `deflate`, `zlib` and `gzip` no longer imply `alloc`.
  The standard build still needs it and says so in a `compile_error!`
  when neither `alloc` nor `embed` is on (a breaking change for
  `--no-default-features --features gzip` builds that relied on the
  implied `alloc`).
- The wrappers own everything (32 KiB window, 4 KiB block buffer, output
  buffer, 1024-entry hash table), are `const`-constructible with an
  all-zero state so a `static` lands in `.bss`, index nothing that could
  panic, and bound the input fed per call so the worst-case deflate
  expansion (129 bytes per bit) can never overrun the window.
- `compcol::embed` documents what differs (fixed ratio, no sync flush, no
  preset dictionary, first gzip member only) and the sizes CI enforces.
- `tools/footprint/`: a bare-metal thumbv7em-none-eabi harness and
  `check.sh`, which fails if code, static RAM or stack (deepest call path,
  read from the disassembly by `stack.py`) exceed the documented
  ceilings, if panic machinery is linked, or if the codec `static` leaves
  `.bss`. Measured: decoders 4.0–4.4 KB of code, encoders 1.6–1.9 KB,
  under 100 bytes of stack.
- `tools/embed-crosscheck.sh`: the embed and standard CLI builds must read
  each other's gzip/zlib/deflate streams, both ways.
- `tests/embed.rs`: reference streams from CPython's zlib plus hand-made
  ones (a back-reference at exactly 32768, a block whose every two bits
  stand for 258 bytes), every chunking, error reporting, reset/reuse,
  struct-size ceilings and a 16 KiB-stack thread.
- CI: `--features all` for the standard runs, `--all-features` (which
  turns `embed` on) as separate steps, plus footprint and cross-check
  jobs. docs.rs and the release binaries build with `all`, not
  `--all-features`, so they keep the standard codecs.
- tests/cli.rs feeds stdin from a thread: with the embed ratio the child's
  output exceeded the pipe buffer and the old helper deadlocked.

TEMPORARY: minizlib's new API (output accessors, owned tables, const
constructors) is on its master but not yet released, so `[patch.crates-io]`
pins that commit. Once minizlib 0.1.2 is on crates.io, drop the patch in
Cargo.toml and tools/footprint/Cargo.toml and require "0.1.2".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dependency

The `embed` deflate/zlib/gzip codecs are now compcol's own code under
`src/embed/` — `inflate` (bit reader, canonical Huffman, history window),
`decoder` (the push state machine), `encoder` (greedy LZ77 + fixed
Huffman), `format` (the three containers) and `checksum` (64-byte-table
CRC-32, Adler-32) — written straight against the crate's error type and
`RawEncoder` / `RawDecoder` traits. The code was absorbed from the
author's minizlib crate, which is being retired; the dependency and the
temporary `[patch.crates-io]` are gone, and compcol is back to tokio as
its only (optional) dependency.

Being native lets both halves drop what the wrapper had to work around
from outside:

- The decoder writes straight into the caller's slice. A match copy that
  runs out of output keeps what is left as its next state, so no input
  feed bound and no ring-to-caller copy are needed, and gzip members
  concatenate exactly as in the standard build (parked between members
  until the next byte or `finish` decides).
- The encoder queues bits in a 64-bit buffer and moves whole bytes out as
  room allows, so any non-empty output buffer makes progress and the
  4.6 KiB worst-case output buffer is gone: an encoder is 6.2 KB.
- Errors are the standard build's: `Unsupported` for a bad CM or reserved
  gzip flags, `TrailerMismatch` for a wrong ISIZE, `Corrupt` for a stored
  block whose length check fails. Eight standard-suite tests that were
  gated off under `embed` run again.

Footprint on thumbv7em-none-eabi: decoders 3 672–4 036 bytes of code
(was 4 037–4 389), encoders 1 514–1 844 (was 1 555–1 871), both 5 642
(was 6 147), stack under 100 bytes, no panics, statics in `.bss`. The
ceilings in `tools/footprint/check.sh`, `compcol::embed` and the README
are tightened to match. Also stops tracking the harness's build
directory and lockfile, which the first commit swept in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MagicalTux
MagicalTux merged commit 32e2e99 into master Oct 3, 2026
48 checks passed
@MagicalTux
MagicalTux deleted the embed branch October 3, 2026 06:39
@MagicalTux MagicalTux mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant