[winscripter]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

Description

This is a work-in-progress PR whose goal is to introduce decoding and encoding of JPEG XL (*.jxl) images.

Reference software
I use libjxl as reference. See https://github.com/libjxl/libjxl.

Performance
I will begin by applying light optimizations as I implement parts of the JPEG XL codec. Once the codec seems complete enough to handle decoding and encoding of JPEG XL images, I will apply heavier optimizations. Examples include but are not limited to stack allocation, array pooling, and SIMD.

Implementations
The JPEG XL codec lives under src/ImageSharp/Formats/Jxl.

Testing
I will start adding tests whenever the codec is complete enough to handle decoding of JPEG XL images.

Additionally, JPEG XL reference software, libjxl, contains its own tests too, which I might also implement without modification.

Progress

• 🔴 - work didn't start yet
• 🟠 - exists, but incomplete
• 🟡 - either decoding side, encoding side, or tests are complete, but not the other
• 🟢 - complete

---

• 🟡 AC strategy

• 🟢 AC strategy image/row

• 🟢 AC context

• 🔴 AC strategy tests

• 🟢 Adaptive Quantization (encoder)

• 🟡 ANS Entropy

• 🟢 ANS Entropy: Common

• 🟢 ANS Entropy: Common (Tests)

• 🟢 ANS Entropy: Decoder

• 🟠 ANS Entropy: Encoder (SIMD bit-cost calculation only)

• 🔴 ANS Entropy: Tests

• 🟢 Alpha Blending

• 🟡 Bit I/O

• 🟢 Bit I/O: Bit reader

• 🟢 Bit I/O: Bit writer

• 🔴 Bit I/O: tests

• 🟢 Box Content Decoder

• 🟢 Box Content Decoder: Uncompressed boxes

• 🟢 Box Content Decoder: Brotli-compressed boxes

• 🟢 Butteraugli

• 🟢 Butteraugli: Shared methods

• 🟢 Butteraugli: Abstractions

• 🟢 Butteraugli: Default comparator

• 🟢 Butteraugli: Encoder comparator

• 🟡 Cache

• 🟠 Cache: Decoder

• 🔴 Cache: Encoder

• 🟢 Chroma From Luma

• 🟢 Chroma From Luma Abstractions

• 🟢 Chroma From Luma Encoder

• 🟢 Coefficient Order

• 🟢 Coefficient Order: Forward

• 🟢 Coefficient Order: Main

• 🟢 Coefficient Order: Encoder

• 🟢 Coefficient Order: Tests

• 🟢 Compressed DC

• 🟢 Context Map

• 🟢 Context Map: Abstractions

• 🟢 Context Map: Decoder

• 🟢 Context Map: Encoder

• 🟡 Convolution

• 🟢 Convolution: Symmetric

• 🟢 Convolution: Separable

• 🟢 Convolution: Slow

• 🟢 Convolution: SIMD

• 🟢 Convolution: Separable5 Encoder

• 🔴 Convolution: Tests

• 🟠 Decoder: Frame

• 🟢 Decoder: Group

• 🟢 Decoder: Group Border

• 🟠 Decoder: Main

• 🟢 Decoder: Main: Codestream parser

• 🟢 Decoder: Main: Container format parser

• 🟢 Discrete Cosine Transform

• 🟢 Discrete Cosine Transform: DCT scales

• 🟢 Discrete Cosine Transform: Block data wrapper

• 🟢 Discrete Cosine Transform: Block-based

• 🟢 Discrete Cosine Transform: Slow DCT for reference in tests

• 🟢 Discrete Cosine Transform: Tests

• 🟢 Encoder dot detection

• 🟢 Encoder dot dictionary

• 🟢 Entropy coding

• 🟢 Encoder entropy coding

• 🟢 Fields

• 🟢 Fields: Visitor abstractions

• 🟢 Fields: Parser

• 🟢 Fields: Writer

• 🔴 Frame Encoder

• 🟢 Gaborish

• 🟢 Gaborish Encoder

• 🟢 Gaborish Tests

• 🟢 Group Encoder

• 🟢 Heuristics Encoder

• 🟢 Image Bundle

• 🟢 Image Bundle: Decoder/Common

• 🟢 Image Bundle: Encoder

• 🟢 LZ77 compression

• 🟢 LZ77: Standard Encoder

• 🟢 Huffman compression

• 🟢 Huffman compression: Shared

• 🟢 Huffman compression: Decoder

• 🟢 Huffman compression: Encoder

• 🟡 Modular

• 🟢 Modular: Transforms

• 🟢 Modular: Transforms: Palette (Inverse)

• 🟢 Modular: Transforms: Palette (Forward)

• 🟢 Modular: Transforms: RCT (Inverse)

• 🟢 Modular: Transforms: RCT (Forward)

• 🟢 Modular: Transforms: Squeeze (Inverse)

• 🟢 Modular: Transforms: Squeeze (Forward)

• 🟢 Modular: Encoding

• 🟢 Modular: Encoding: Context Prediction

• 🟢 Modular: Encoding: MA decoder

• 🟢 Modular: Encoding: MA encoder

• 🟢 Modular: Encoding: Tree Samples

• 🟢 Modular: Encoding: Encoding decoder

• 🟢 Modular: Encoding: Encoding encoder

• 🔴 Modular: Decoder

• 🔴 Modular: Encoder

• 🟢 Modular: Encoder SIMD

• 🔴 Modular: Tests

• 🟢 Patch Dictionary: Decoder

• 🔴 Patch Dictionary: Encoder

• 🟠 Passes State: Decoder

• 🔴 Passes State: Encoder

• 🟢 Passes State: Shared

• 🔴 Encoder Main

• 🔴 Encoder Main

• 🔴 Encoder Internal

• 🔴 Encoder Tests

• 🟢 Encoder: Linear Algebra

• 🟢 Encoder: Linear Algebra Tests

• 🟢 Image Operations

• 🟢 Image Operations

• 🟢 Image Operations: Tests

• 🟢 Common I/O: Frame Header

• 🟢 Common I/O: Metadata

• 🟢 Common I/O: Container format

• 🟠 JPEG to JPEG XL lossless compression

• 🟢 JPEG to JPEG XL lossless compression: JPEG parser/writer

• 🟠 JPEG to JPEG XL lossless compression (decoder)

• 🔴 JPEG to JPEG XL lossless compression (encoder)

• 🟡 Splines

• 🟢 Splines

• 🔴 Splines Tests

• 🟢 Quantizer

• 🟢 Dequantizer matrices

• 🟢 Quantizer encoding

• 🟢 Quantizer weights

• 🟢 Noise

• 🟢 Noise: Shared

• 🟢 Noise: Decoder

• 🟢 Noise: Encoder

• 🟢 Noise: Simulation of Photon Noise

• 🟢 Noise: Simulation of Photon Noise (Tests)

• 🔴 JPEG XL Testing Tools

• 🟠 Render Pipeline

• 🔴 Render Pipeline: Main

• 🔴 Render Pipeline: Low Memory Render Pipeline

• 🟢 Render Pipeline: Stages Abstractions

• 🟢 Render Pipeline: Stages: Blending

• 🟢 Render Pipeline: Stages: Chroma Upsampling

• 🔴 Render Pipeline: Stages: CMS

• 🟢 Render Pipeline: Stages: EPF

• 🟢 Render Pipeline: Stages: From Linear

• 🟢 Render Pipeline: Stages: Gaborish

• 🟢 Render Pipeline: Stages: Noise

• 🟢 Render Pipeline: Stages: Patches

• 🟢 Render Pipeline: Stages: Splines

• 🟢 Render Pipeline: Stages: Spot color

• 🟢 Render Pipeline: Stages: To Linear

• 🟢 Render Pipeline: Stages: Tone mapping

• 🟢 Render Pipeline: Stages: Upsampling

• 🟢 Render Pipeline: Stages: Write to Output

• 🟢 Render Pipeline: Stages: XYB

• 🟢 Render Pipeline: Stages: Y'Cb'Cr -> RGB

• 🟡 Color Management System (CMS)

• 🟢 CMS: Transfer Functions

• 🟢 CMS: Abstractions/Color Encoding

• 🟢 CMS: Tone Mapping

• 🔴 CMS: Interface

Other completed things:

• 🟢 Frame Dimensions
• 🟢 Field Encodings
• 🟢 Gamma correction (tests, encoder)
• 🟢 Shared math functions
• 🟢 Lehmer Codes
• 🟢 Lehmer Codes: Tests
• 🟢 Luminance helper
• 🟢 Opsin parameters
• 🟢 Passes shared state
• 🟢 SIMD utilities
• 🟢 TOC
• 🟢 XorShift128Plus
• 🟢 XorShift128Plus Tests

[CLAassistant]

All committers have signed the CLA.

[winscripter]

While I'm working on this, I'd like to note something important.

Libjxl is licensed under the BSD 3-Clause license, and since I'm using libjxl code as reference, that means the license must be included.

I'm not really sure what would be the proper way to include the license. I might place the LICENSE.txt file in the Jxl folder or add a README linking to the libjxl repo.

[JimBobSquarePants]

Thanks @winscripter I really appreciate you attempting this.

Examples include but are not limited to stack allocation, array pooling, and SIMD.

On a performance note. I think it would be best if you use these immediately as that can affect design. Any new code should also reuse utility components already available in the library where possible.

In addition, I normally document internal type as well as public types. This stuff is tough and every little helps.

[winscripter]

@JimBobSquarePants The examples I mentioned were the aggressive performance optimizations that I'd apply when the implementation can encode and decode images properly. I'd prefer not to apply them just yet, because in that case, the optimizations may introduce subtle bugs that would later be difficult to find while the implementation isn't fully complete.

I do consider applying light optimizations right now, e.g. using spans, pooling arrays with Configuration.MemoryAllocator.Allocate, inlining attributes, moving checks to debug assertions, etc. These optimizations don't largely differ from the reference implementation, so if things go wrong, I can compare my implementation with the reference implementation and spot the issue easily.

I'd consider documenting the internal stuff, but it might be easier said than done, because not all things in libjxl source are documented.

[gfoidl]

Is this safe?
Vector<T>'s size is dependent on the hardware, so not (really) fixed size.
Should it rather be Vector128<float>?

[winscripter]

Yup, it should be fixed-size vector. Libjxl reference configures vector sizes to be equal to the sz parameter which I didn't spot.

The only issue is, what would be the right way to do this? The sz parameter can be equal to 1, 2, 4, 8, 16, 32, 64, or 128. In order to achieve good performance using SIMD for discrete cosine transform, I'd have to duplicate the code multiple times: scalar (1), Vector64 (2), Vector128 (4), Vector256/128 (8), Vector512/256/128 (16), and the rest with nested loops, which is what libjxl seems to do. It would result in a large source file. Is it fine?

[gfoidl]

Unfortunately duplication is at the moment the only way, as something like dotnet/runtime#76244 hasn't landed so far.

Or something like TensorPrimitives is used (I think all needed APIs are there).
But I'll let @JimBobSquarePants say a word about having such a dependency.

[JimBobSquarePants]

I’ve got an inline copy of the tensor primitives code for several methods in main already. (Tweaked with custom inlining rules for perf) and we should use them where applicable.

We should be trying to use existing code as a guide where possible. For example DCT is a function of JPEG. Can it be solved in the same way?

[JimBobSquarePants]

I strongly advise ensuring this branch is updated from main.

[winscripter]

I strongly advise ensuring this branch is updated from main.

Done.

We should be trying to use existing code as a guide where possible. For example DCT is a function of JPEG. Can it be solved in the same way?

Technically yes, but there's a problem. JPEG only supports 8x8 blocks, while JPEG XL supports 2x2 all the way up to 256x256. So using JPEG's DCT wouldn't be possible for JPEG XL's. The JPEG DCT implementation also seems to use the Arai, Agui and Nakajima variant of the DCT which is incredibly fast and relies on approximations, but there's only a 4x4 and 8x8 variant of these.

I’ve got an inline copy of the tensor primitives code for several methods in main already. (Tweaked with custom inlining rules for perf) and we should use them where applicable.

That's awesome, I've worked with the TensorPrimitives class before and I think I understand how it works. However, the copy unfortunately doesn't have a Subtract method, which is important for the discrete cosine transform.

[JimBobSquarePants]

Technically yes, but there's a problem. JPEG only supports 8x8 blocks, while JPEG XL supports 2x2 all the way up to 256x256. So using JPEG's DCT wouldn't be possible for JPEG XL's. The JPEG DCT implementation also seems to use the Arai, Agui and Nakajima variant of the DCT which is incredibly fast and relies on approximations, but there's only a 4x4 and 8x8 variant of these.

Very good to know!

TensorPrimitives class before and I think I understand how it works. However, the copy unfortunately doesn't have a Subtract method, which is important for the discrete cosine transform.

We can copy that easily enough... However, it's very likely i'll be introducing the real thing soon enough. I expect us to start moving to .NET 10 and starting on v5 in the next few weeks.

[winscripter]

I've implemented the JxlDecoderCore prototype, but in doing so I had to change a few things.

JPEG XL reference, libjxl, as its own ICC parsing code (icc_codec.cc, icc_codec.h, icc_codec_common.cc and icc_codec_common.h). Of course, it's best to reuse existing components rather than reinventing the wheel, and I noticed that ImageSharp contains an ICC implementation too.

However, after parsing the ICC profile, the codec needs to know how many bytes of ICC were actually read, which by default the ImageSharp ICC codec did not expose.

So I had to make the following changes.

https://github.com/winscripter/ImageSharp/blob/jxl-support/src/ImageSharp/Metadata/Profiles/ICC/DataReader/IccDataReader.cs#L30-L33

I had to add a public Index property here so we can know how many bytes were actually read.

Also:

https://github.com/winscripter/ImageSharp/blob/jxl-support/src/ImageSharp/Metadata/Profiles/ICC/IccReader.cs#L56

https://github.com/winscripter/ImageSharp/blob/jxl-support/src/ImageSharp/Metadata/Profiles/ICC/IccReader.cs#L82

I had to change the accessibility of these methods from private to internal.

Let me know if that's fine.

[gfoidl]

👍🏻

IMO this could also be done post this PR (but then an issue should be filed for this).

[winscripter]

I'm currently working on changing the JPEG XL input access so we don't have to copy the entire stream into an array to continue decoding from and just use the stream directly (the copy approach can use a lot of memory for larger files, especially animated JXLs), so later the ICC code here won't use spans and will just pass the stream directly.

Still, ICC only operates on byte arrays. However, the ICC decoding methods are quite abstract, so using Stream for ICC won't be as difficult as with spans (because a class needs to be a ref struct so we can store a Span inside).

[JimBobSquarePants]

Hey @winscripter

Could you please do me a favour and grant me full write access to your fork? I want to help with this, but Git LFS can cause problems on GitHub which block me being able to push.

I'll spend some time reading and documenting methods to start with.

[winscripter]

Hello @JimBobSquarePants I've added you as a collaborator to the repo.

I want to help with this

I appreciate that!

[gfoidl]

The methods here are quite repetitive. Can generic math be used to reduce the code size?

[gfoidl]

OK, in the next file I see it's generated 😉 (I didn't check the file name first 🙈).

Anyway w/ generic math -- if possible -- the tt-file could be removed altogether.

[winscripter]

I think the current approach can be really good for performance, I do believe using generics here can be possible with some unsafe magic, but then it'd most likely (and I assume) be much slower, primarily because we may have to reverse the endianness which can be much more difficult to implement in an optimized way if the size of the primitive is not immediately known.

[gfoidl]

I meant something like

    public static unsafe T ReadLittleEndian<T>(Stream stream) where T : IBinaryInteger<T>
    {
        Span<byte> data = stackalloc byte[sizeof(T)];
        stream.ReadExactly(data);
        return T.ReadLittleEndian(data, false);
    }

Here the unsafe is only needed for the sizeof(T), and the machine code should be the same / similar.

[winscripter]

Perfect. Just a small problem... there doesn't seem to be a write counterpart in IBinaryInteger.

[gfoidl]

TryWriteLittleEndian and others are there since .NET 7.

[gfoidl]

I meant something like

    public static unsafe T ReadLittleEndian<T>(Stream stream) where T : IBinaryInteger<T>
    {
        Span<byte> data = stackalloc byte[sizeof(T)];
        stream.ReadExactly(data);
        return T.ReadLittleEndian(data, false);
    }

Here the unsafe is only needed for the sizeof(T), and the machine code should be the same / similar.

[winscripter]

I'd need a bit of help regarding a question on accessing JPEG data.

So, one awesome feature with JPEG XL is that, one can take an existing JPEG file and directly convert it to JPEG XL with zero quality loss. This is different from converting JPEG to other formats like HEIF or WebP, because such formats are architecturally incompatible, and given that JPEG is already a lossy format, taking a JPEG and converting it to more compressible formats would significantly lower image quality because information is discarded twice. JPEG XL supports this because both JPEG and JPEG XL use very similar compression algorithms (Huffman, arithmetic coding, DCT, ...) so the JPEG XL codec can inherit some of these parameters from the JPEG file and introduce modern compression methods on top.

ImageSharp supports decoding and encoding of JPEG images, which is awesome. The JPEG XL needs access to data within some JPEG markers, such as APP or COM, and those related to Huffman and scans, so we can support lossless JPEG<->JPEG XL conversion. However, I'm not actually sure how we can parse and retrieve those markers from a JPEG file using the existing implementation. Any ideas?

Libjxl does have JPEG parsing code to do this, but of course, it's better not to reinvent the wheel.

See https://github.com/libjxl/libjxl/blob/main/lib/jxl/jpeg/jpeg_data.h#L185-L197

[JimBobSquarePants]

Could you pause further commits for now, please? I’m reworking the integration to use ImageSharp’s existing codec architecture and shared APIs. Several areas will change substantially, so I’d like to avoid us making conflicting changes.

Thank you for the work you’ve put into this. I’ll post an update here once that work is ready, and we can coordinate the next steps together.

[winscripter]

Yep, understood. Sorry about that!