[FEATURE] Add compact binary model format (.namb) for fast embedded loading by jfsantos · Pull Request #222 · sdatkinson/NeuralAmpModelerCore

jfsantos · 2026-02-11T23:32:17Z

Introduces a binary serialization format that eliminates the need for
JSON parsing (no nlohmann/json dependency in the loader), achieving
80-89% file size reduction. Supports all architectures including
WaveNet with nested condition_dsp.

New files:

NAM/namb_format.h: format constants, CRC32, binary reader/writer
NAM/get_dsp_namb.h/.cpp: binary loader (zero JSON dependency)
tools/nam2namb.cpp: JSON-to-binary converter CLI tool
tools/test/test_namb.cpp: round-trip, validation, and size tests

Developed with support and sponsorship from TONE3000

…oading Introduces a binary serialization format that eliminates the need for JSON parsing (no nlohmann/json dependency in the loader), achieving 80-89% file size reduction. Supports all architectures including WaveNet with nested condition_dsp. New files: - NAM/namb_format.h: format constants, CRC32, binary reader/writer - NAM/get_dsp_namb.h/.cpp: binary loader (zero JSON dependency) - tools/nam2namb.cpp: JSON-to-binary converter CLI tool - tools/test/test_namb.cpp: round-trip, validation, and size tests

mikeoliphant · 2026-02-12T01:13:56Z

Is this intended as a new public distribution format? Or just as a tool for situations where using json is impractical?

A huge benefit of the json format is that it makes the .nam format very open and easy to integrate with. A custom binary format is much less so.

sdatkinson · 2026-02-12T01:21:38Z

tools/test/test_namb.cpp

+  uint32_t crc_empty = nam::namb::crc32(nullptr, 0);
+  assert(crc_empty == 0x00000000u);
+
+  std::cout << "  PASS" << std::endl;


Can you make the tests silent on happy path?

oops, I did it again. I always mean to delete those after making sure everything works, and always forget.

sdatkinson · 2026-02-12T01:22:49Z

tools/test/test_namb.cpp

+              << std::fixed << std::setprecision(1) << reduction << "% reduction)" << std::endl;
+
+    // .namb should always be smaller than .nam
+    assert(namb_size < nam_size);


Hmm...while it's true, it feels wrong to me to be asserting this.

The assertions were to make sure the binary conversion wasn't doing anything silly... but we can remove them.

sdatkinson · 2026-02-12T01:22:59Z

tools/test/test_namb.cpp

+    assert(namb_size < nam_size);
+
+    // Should be at least 50% reduction (typically ~85%)
+    assert(reduction > 50.0);


...and this

sdatkinson · 2026-02-12T01:41:33Z

A few high-level questions to ponder @jfsantos:

Will this be forward-compatible? I think so (magic number, file version) but give it a thought and lmk if you can think of any gotchas.
It'd be nice to discern file type and route to the appropriate methods...but as I sit and think about this I think we're on the path to doing that optionally in a future PR, right?
Any reasons you can think of why this shouldn't be in the main repo? Like I'm thinking that maybe this shouldn't be "the" binary format but one example / make some pattern for others to make their own? Or even better: consider how I routinely extend this library to make custom models. What would the experience doing so look like? Would it be necessary to change this library? The big example of this in the past was "registering" new models to load. In the current version of the code, I can create new architectures and register their factories for get_dsp to recognize; does this "box out" from doing that? Would it be much work to change?
3.1. If I squint at this, it's kind of like we're defining new factories for models. Now, the current factory makes some kind of restrictive assumptions (e.g. that it's getting a JSON object etc as input), but is there some "gentle" nudge to that (ok if breaking) where it paves the way for these .namb loaders to be on equal footing to the existing factories?

Thoughts appreciated.

sdatkinson · 2026-02-12T01:47:18Z

@mikeoliphant the former--the current JSON-subset ".nam" format will still be the recommended "lingua franca" for models. I agree with you--it's very handy to be able to look at the files as a human :) and this feels like a nice "add-on".

In fact, this might be entirely separate from this "core" library. Arguments why / why not to do that are welcome.

mikeoliphant · 2026-02-12T02:11:04Z

I'm thinking about it from a support perspective for NAM implementations that don't use NAM Core (I've got mine, Dimehead has theirs, etc.)

If this new format is going to be used as a new distribution format (ie: on Tone3000), then we'll need to add support for it.

jfsantos · 2026-02-12T02:31:00Z

@mikeoliphant this is intended for low memory platforms where parsing JSON is not feasible because of the high memory costs. It will never be the main distribution system; that will still be JSON-based.

mikeoliphant · 2026-02-12T02:38:26Z

@jfsantos 👍

jfsantos · 2026-02-12T03:17:32Z

@sdatkinson thanks for your questions!

It is meant to be forward compatible, hence the numbers. I think the approach should be to have it be forward compatible with anything that comes after A2.
Yes, the reason I didn't change anything other than loadmodel to call the method for the correct file format is that I do not think core users in general will need to support this and will continue loading from .nam files. We could definitely do it to simplify code and have a simple get_dsp_from_file which routes to the right implementation.
I think the ideal design would be to have some sort of ModelConfig object derived from either a .nam or .namb file, and have a single factory method that takes that object and instantiates a model. I reimplemented the factories for the binary format because it would make no sense to reuse the previous factories and this was the safest path that didn't risk breaking anything that already works. I guess one simpler approach would be to refactor the path that expects a JSON object to instead expect something based on std::map?

Introduce typed config structs per architecture (LinearConfig, LSTMConfig, ConvNetConfig, WaveNetConfig) and a single create_dsp() function that both the JSON and .namb binary loaders feed into. This eliminates duplicated construction logic — adding a new architecture now only requires a format- specific parser, not separate factory code in each loader. - Add config structs and parse_config_json() to each architecture header/impl - Add NAM/model_config.h with ModelConfig variant, ModelMetadata, create_dsp() - Refactor get_dsp(dspData&) to use parse_model_config_json() → create_dsp() - Refactor get_dsp_namb.cpp load_*() to return typed configs → create_dsp() - Register missing Linear factory in FactoryRegistry - Silence test_namb.cpp output on success to match other tests

jfsantos · 2026-02-13T04:09:32Z

Closed in favour of #227

sdatkinson reviewed Feb 12, 2026

View reviewed changes

João Felipe Santos and others added 5 commits February 12, 2026 10:48

Merge branch 'main' into feature/binary_loader

2a275d5

Small fix to botched merge

d68ea16

Merge branch 'main' into feature/binary_loader

0d1607a

Fixing another merge issue

ff8cd16

jfsantos closed this Feb 13, 2026

Conversation

jfsantos commented Feb 11, 2026

Uh oh!

mikeoliphant commented Feb 12, 2026

Uh oh!

sdatkinson Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

jfsantos Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

sdatkinson Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

jfsantos Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

sdatkinson Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

sdatkinson commented Feb 12, 2026

Uh oh!

sdatkinson commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikeoliphant commented Feb 12, 2026

Uh oh!

jfsantos commented Feb 12, 2026

Uh oh!

mikeoliphant commented Feb 12, 2026

Uh oh!

jfsantos commented Feb 12, 2026

Uh oh!

jfsantos commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sdatkinson commented Feb 12, 2026 •

edited

Loading