diff --git a/README.md b/README.md index 9a81ac6..f81a412 100644 --- a/README.md +++ b/README.md @@ -60,13 +60,13 @@ The environment variables are used to create a `samconfig.toml` file that config Previously, some custom docker images were needed to build this layer. AWS now publishes managed SAM build images for current Lambda runtimes, including `public.ecr.aws/sam/build-nodejs24.x`. ### Testing -The [Test layer](.github/workflows/test-layer.yml) workflow builds the layer with `sam build --use-container` and mounts the result at `/opt` in the `public.ecr.aws/lambda/nodejs:24` runtime image. There it runs [test/layer.test.js](test/layer.test.js), which checks shared library resolution, the libvips and sharp versions, HEIC decoding, WebP resizing, and HEIF encoding with both HEVC and AV1. Pixel colours are verified against [test/fixtures/quadrants.heic](test/fixtures/quadrants.heic), a synthetic image encoded with macOS `sips` (regenerate with `node test/fixtures/make-quadrants-heic.js`). The layer is only rebuilt when `layer/`, `test/`, `examples/src/`, `template.yaml` or the workflow itself change; for other changes (e.g. documentation) the build job is skipped. It also invokes the [example function](examples/src/index.js) through the Lambda Runtime Interface Emulator. +The [Test layer](.github/workflows/test-layer.yml) workflow builds the layer with `sam build --use-container` and mounts the result at `/opt` in the `public.ecr.aws/lambda/nodejs:24` runtime image. There it runs [test/layer.test.js](test/layer.test.js), which checks shared library resolution, the libvips and sharp versions, HEIC decoding, WebP resizing, HEIF encoding with both HEVC and AV1, static and animated GIF output, and palette PNG output. Pixel colours are verified against [test/fixtures/quadrants.heic](test/fixtures/quadrants.heic), a synthetic image encoded with macOS `sips` (regenerate with `node test/fixtures/make-quadrants-heic.js`). The layer is only rebuilt when `layer/`, `test/`, `examples/src/`, `template.yaml` or the workflow itself change; for other changes (e.g. documentation) the build job is skipped. It also invokes the [example function](examples/src/index.js) through the Lambda Runtime Interface Emulator. ## Background This repo exists as it is rather painful to compile all libraries required to get sharp to work with HEIC/HEIF files in an AWS Lambda environment. The sharp repository has several [issues](https://github.com/lovell/sharp/issues) related to this. ### Layer contents -This lambda layer contains the node module [sharp](https://github.com/lovell/sharp). But unlike a normal installation via `npm i sharp` this layer does not use the prebuilt sharp and libvips binaries. This layer compiles libwebp, libde265, x265, libaom, libheif, and libvips from source, then explicitly runs `sharp`'s build script against that global libvips installation in order to provide HEIC/HEIF (and WebP) functionality in an AWS Lambda environment. +This lambda layer contains the node module [sharp](https://github.com/lovell/sharp). But unlike a normal installation via `npm i sharp` this layer does not use the prebuilt sharp and libvips binaries. This layer compiles libwebp, libde265, x265, libaom, libheif, libimagequant, cgif, and libvips from source, then explicitly runs `sharp`'s build script against that global libvips installation in order to provide HEIC/HEIF (and WebP) functionality in an AWS Lambda environment. As of `sharp@0.35.1`, building from source is no longer triggered automatically during `npm install`, so the layer build now installs the package first and then runs `sharp`'s build script against the custom `libvips` installation. @@ -74,22 +74,26 @@ The native build is intentionally pinned end-to-end so the layer uses the versio ### Dependencies The following table lists the release version of this repo together with the version of each dependency. Patch versions are related to the build process or documentation and have the same dependencies as the minor version. -| release | sharp | libvips | libheif | libwebp | libde265 | x265 | libaom | nodejs | -|---------|--------|---------|---------|---------|-----------|--------|--------|--------| -| 1.2.0 | 0.28.2 | 8.10.6 | 1.12.0 | 1.2.0 | 1.0.8 | - | | 12 | -| 1.1.0 | 0.27.0 | 8.10.5 | 1.10.0 | 1.1.0 | 1.0.8 | - | | 12 | -| 2.0.0 | 0.29.1 | 8.11.3 | 1.12.0 | 1.2.1 | 1.0.8 | - | | 14 | -| 3.0.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.2.4 | 1.0.8 | - | | 16 | -| 3.1.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.2.4 | 1.0.8 | - | | 16 | -| 3.2.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.3.2 | 1.0.12 | - | | 16 | -| 4.1.0 | 0.33.3 | 8.15.2 | 1.17.6 | 1.4.0 | 1.0.15 | 3.6 | | 20 | -| 4.1.3 | 0.33.3 | 8.15.2 | 1.17.6 | 1.4.0 | 1.0.15 | 3.6 | | 20 | -| 4.2.0 | 0.33.5 | 8.15.3 | 1.18.2 | 1.4.0 | 1.0.15 | 3.6 | 3.9.1 | 20 | -| 5.0.0 | 0.34.3 | 8.17.1 | 1.20.1 | 1.6.0 | 1.0.16 | 4.1 | 3.12.1 | 22 | -| 5.1.0 | 0.34.4 | 8.17.2 | 1.20.2 | 1.6.0 | 1.0.16 | 4.1 | 3.13.1 | 22 | -| 6.0.0 | 0.34.5 | 8.17.3 | 1.21.2 | 1.6.0 | 1.0.16 | 4.1 | 3.13.1 | 24 | -| 6.1.0 | 0.35.1 | 8.18.3 | 1.23.0 | 1.6.0 | 1.0.18 | 4.1 | 3.14.1 | 24 | -| 6.2.0 | 0.35.5 | 8.18.7 | 1.23.6 | 1.6.0 | 1.1.3 | 4.2 | 3.15.1 | 24 | +| release | sharp | libvips | libheif | libwebp | libde265 | x265 | libaom | libimagequant | cgif | nodejs | +|---------|--------|---------|---------|---------|-----------|--------|--------|---------------|-------|--------| +| 1.2.0 | 0.28.2 | 8.10.6 | 1.12.0 | 1.2.0 | 1.0.8 | - | | - | - | 12 | +| 1.1.0 | 0.27.0 | 8.10.5 | 1.10.0 | 1.1.0 | 1.0.8 | - | | - | - | 12 | +| 2.0.0 | 0.29.1 | 8.11.3 | 1.12.0 | 1.2.1 | 1.0.8 | - | | - | - | 14 | +| 3.0.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.2.4 | 1.0.8 | - | | - | - | 16 | +| 3.1.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.2.4 | 1.0.8 | - | | - | - | 16 | +| 3.2.0 | 0.30.7 | 8.12.2 | 1.12.0 | 1.3.2 | 1.0.12 | - | | - | - | 16 | +| 4.1.0 | 0.33.3 | 8.15.2 | 1.17.6 | 1.4.0 | 1.0.15 | 3.6 | | - | - | 20 | +| 4.1.3 | 0.33.3 | 8.15.2 | 1.17.6 | 1.4.0 | 1.0.15 | 3.6 | | - | - | 20 | +| 4.2.0 | 0.33.5 | 8.15.3 | 1.18.2 | 1.4.0 | 1.0.15 | 3.6 | 3.9.1 | - | - | 20 | +| 5.0.0 | 0.34.3 | 8.17.1 | 1.20.1 | 1.6.0 | 1.0.16 | 4.1 | 3.12.1 | - | - | 22 | +| 5.1.0 | 0.34.4 | 8.17.2 | 1.20.2 | 1.6.0 | 1.0.16 | 4.1 | 3.13.1 | - | - | 22 | +| 6.0.0 | 0.34.5 | 8.17.3 | 1.21.2 | 1.6.0 | 1.0.16 | 4.1 | 3.13.1 | - | - | 24 | +| 6.1.0 | 0.35.1 | 8.18.3 | 1.23.0 | 1.6.0 | 1.0.18 | 4.1 | 3.14.1 | - | - | 24 | +| 6.2.0 | 0.35.5 | 8.18.7 | 1.23.6 | 1.6.0 | 1.1.3 | 4.2 | 3.15.1 | - | - | 24 | +| 6.3.0 | 0.35.5 | 8.18.7 | 1.23.6 | 1.6.0 | 1.1.3 | 4.2 | 3.15.1 | 2.4.1 | 0.5.4 | 24 | + +### Note regarding GIF and palette PNG output +libvips needs [cgif](https://github.com/dloebl/cgif) to write GIF files and an image quantiser to write GIF and palette PNG (`png({ palette: true })`) files. The layer builds cgif 0.5.4 and the BSD-licensed [libimagequant 2.4.1 fork](https://github.com/lovell/libimagequant) also used by sharp's prebuilt binaries. Without them, saving a GIF fails with `VipsOperation: class "gifsave_buffer" not found` ([#13](https://github.com/zoellner/sharp-heic-lambda-layer/issues/13)), and `palette: true` is ignored. ### Note regarding HEIF security limits `libvips` 8.18.3 and earlier cap `libheif`'s `max_items` at 16 when loading HEIF/HEIC files. Many files written by recent phone cameras exceed this (for example `iinf`, `iref` and `ipma` boxes with 20 to 50 entries) and fail to load with `Security limit exceeded`. `libvips` 8.18.4 raised these limits, and `sharp` 0.35.5 requires `libvips` 8.18.7 or later, so the layer no longer needs any patching to load these files. @@ -113,6 +117,8 @@ You can also use the Sponsor button on the right if you'd like. - libwebp is Copyright Google Inc. See https://github.com/webmproject/libwebp/blob/master/COPYING for details. - sharp is licensed under the Apache License, Version 2.0. Copyright Lovell Fuller and contributors. See https://github.com/lovell/sharp/blob/master/LICENSE for details. - libvips is licensed under the LGPL 2.1+. See https://github.com/libvips/libvips/blob/master/COPYING for details. +- cgif is licensed under the MIT License. Copyright Daniel Löbl. See https://github.com/dloebl/cgif/blob/main/LICENSE for details. +- libimagequant 2.4.x is licensed under the BSD 2-Clause License. Copyright Greg Roelofs and Kornel Lesiński. See https://github.com/lovell/libimagequant/blob/main/COPYRIGHT for details. - libaom is subject to the terms of the BSD 2 Clause License and the Alliance for Open Media Patent License 1.0. See https://aomedia.googlesource.com/aom/#license-header - The remainder of the code in this repository is licensed under the MIT License. See [LICENSE](LICENSE) for details. @@ -125,4 +131,6 @@ Visit [sharp.pixelplumbing.com](https://sharp.pixelplumbing.com/) for complete i - [libde265](https://github.com/strukturag/libde265) - [libwebp](https://github.com/webmproject/libwebp) - [libvips](https://github.com/libvips/libvips) +- [cgif](https://github.com/dloebl/cgif) +- [libimagequant](https://github.com/lovell/libimagequant) - [libaom](https://aomedia.googlesource.com/aom/) diff --git a/layer/Makefile b/layer/Makefile index 7377726..cdea735 100644 --- a/layer/Makefile +++ b/layer/Makefile @@ -4,6 +4,8 @@ LIBDE265_VERSION=1.1.3 LIBAOM_VERSION=3.15.1 X265_VERSION=4.2 LIBHEIF_VERSION=1.23.6 +LIBIMAGEQUANT_VERSION=2.4.1 +CGIF_VERSION=0.5.4 VIPS_VERSION=8.18.7 SHARP_VERSION=0.35.5 NODE_ADDON_API_VERSION=8.9.2 @@ -15,6 +17,8 @@ LIBDE265_SHA256=554228bd17788c99a7e63b37ab5634722190e6e2bf60c1dcb01cef328e133905 LIBAOM_SHA256=8ca0c52746174603500f0adb6f2a215d69c9ca2aab2acb3caa06fb791d8d01bf X265_SHA256=40b1ea0453e0309f0eba934e0ddf533f8f6295966679e8894e8f1c1c8d5e1210 LIBHEIF_SHA256=4484346dc5995319dbc11e3a1c35d0a2ec46511ce370900869337fd2c7033125 +LIBIMAGEQUANT_SHA256=47d2a84b7b1052975c9d50a3d4e3cacbf57b43d84a4c3131210848ead9964dfb +CGIF_SHA256=83a70a15bc2da41f081a44ebc58ee48e2e1d524a6d3fdb4a24064afa08d5ad4d VIPS_SHA256=5baaead3b0bb20ffdb9e9ff09aa9fda08620923df77b63b436654cb5e0b3bf94 # CodeBuild medium instances can OOM when libaom compiles with full parallelism. @@ -38,7 +42,7 @@ echo "$(3) $(1)" | sha256sum -c - endef ARTIFACTS_NODEJS_DIR=$(ARTIFACTS_DIR)/nodejs ARTIFACTS_LIB_DIR=$(ARTIFACTS_DIR)/lib -.PHONY: build-SharpHEICLayer clean installLib system-libs verifySystemLibs codec-stack verifyCodecStack verifyLibvips libwebp libde265 x265 libaom libheif libvips +.PHONY: build-SharpHEICLayer clean installLib system-libs verifySystemLibs codec-stack verifyCodecStack verifyLibvips libwebp libimagequant cgif libde265 x265 libaom libheif libvips build-SharpHEICLayer: libvips @printf '\n==> %s\n' "Preparing layer artifacts" @@ -46,7 +50,7 @@ build-SharpHEICLayer: libvips mkdir -p "$(ARTIFACTS_NODEJS_DIR)" "$(ARTIFACTS_LIB_DIR)" @printf '\n==> %s\n' "Using dependency versions" - printf ' sharp=%s\n libvips=%s\n libheif=%s\n libwebp=%s\n libde265=%s\n x265=%s\n libaom=%s\n' "$(SHARP_VERSION)" "$(VIPS_VERSION)" "$(LIBHEIF_VERSION)" "$(WEBP_VERSION)" "$(LIBDE265_VERSION)" "$(X265_VERSION)" "$(LIBAOM_VERSION)" + printf ' sharp=%s\n libvips=%s\n libheif=%s\n libwebp=%s\n libde265=%s\n x265=%s\n libaom=%s\n libimagequant=%s\n cgif=%s\n' "$(SHARP_VERSION)" "$(VIPS_VERSION)" "$(LIBHEIF_VERSION)" "$(WEBP_VERSION)" "$(LIBDE265_VERSION)" "$(X265_VERSION)" "$(LIBAOM_VERSION)" "$(LIBIMAGEQUANT_VERSION)" "$(CGIF_VERSION)" @printf '\n==> %s\n' "Installing sharp build prerequisites" npm install --save --prefix "$(ARTIFACTS_NODEJS_DIR)/" node-addon-api@$(NODE_ADDON_API_VERSION) node-gyp@$(NODE_GYP_VERSION) @@ -85,6 +89,22 @@ libwebp: tar zxf "libwebp-$(WEBP_VERSION).tar.gz" cd libwebp-$(WEBP_VERSION) && ./autogen.sh && ./configure --enable-libwebpmux --prefix=$(PREFIX_PATH) && make V=0 -j$(BUILD_JOBS) && make install +# libvips needs an image quantiser for GIF output and palette PNG output. libimagequant 2.4.x is the +# BSD-licensed fork used by sharp's prebuilt binaries (upstream 4.x is GPL-3.0). +libimagequant: + @printf '\n==> %s\n' "Building libimagequant $(LIBIMAGEQUANT_VERSION)" + rm -rf "libimagequant-$(LIBIMAGEQUANT_VERSION)" + $(call fetch_verify,libimagequant-$(LIBIMAGEQUANT_VERSION).tar.gz,https://github.com/lovell/libimagequant/archive/v$(LIBIMAGEQUANT_VERSION).tar.gz,$(LIBIMAGEQUANT_SHA256)) + tar zxf "libimagequant-$(LIBIMAGEQUANT_VERSION).tar.gz" + cd libimagequant-$(LIBIMAGEQUANT_VERSION) && meson setup build --prefix ${PREFIX_PATH} --libdir ${LIB_PATH} --buildtype release && meson compile -C build -j $(BUILD_JOBS) && meson install -C build + +cgif: + @printf '\n==> %s\n' "Building cgif $(CGIF_VERSION)" + rm -rf "cgif-$(CGIF_VERSION)" + $(call fetch_verify,cgif-$(CGIF_VERSION).tar.gz,https://github.com/dloebl/cgif/archive/v$(CGIF_VERSION).tar.gz,$(CGIF_SHA256)) + tar zxf "cgif-$(CGIF_VERSION).tar.gz" + cd cgif-$(CGIF_VERSION) && meson setup build --prefix ${PREFIX_PATH} --libdir ${LIB_PATH} --buildtype release -Dexamples=false -Dtests=false && meson compile -C build -j $(BUILD_JOBS) && meson install -C build + codec-stack: libheif verifyCodecStack verifyCodecStack: @@ -139,18 +159,19 @@ libheif: x265 libde265 libaom tar zxf "libheif-$(LIBHEIF_VERSION).tar.gz" cd libheif-$(LIBHEIF_VERSION) && mkdir -p build && cd build && cmake -DCMAKE_INSTALL_LIBDIR=$(LIB_PATH) -DWITH_EXAMPLES=OFF --preset=release-noplugins .. && make V=0 -j$(BUILD_JOBS) && make install -libvips: system-libs codec-stack libwebp +libvips: system-libs codec-stack libwebp libimagequant cgif @printf '\n==> %s\n' "Building libvips $(VIPS_VERSION)" rm -rf "vips-$(VIPS_VERSION)" $(call fetch_verify,vips-$(VIPS_VERSION).tar.xz,https://github.com/libvips/libvips/releases/download/v$(VIPS_VERSION)/vips-$(VIPS_VERSION).tar.xz,$(VIPS_SHA256)) tar xJf "vips-$(VIPS_VERSION).tar.xz" - cd vips-$(VIPS_VERSION) && meson setup build --prefix ${PREFIX_PATH} --libdir ${LIB_PATH} -Dheif=enabled -Dheif-module=disabled -Dlcms=enabled --buildtype release && meson compile -C build -j $(BUILD_JOBS) && meson install -C build + cd vips-$(VIPS_VERSION) && meson setup build --prefix ${PREFIX_PATH} --libdir ${LIB_PATH} -Dheif=enabled -Dheif-module=disabled -Dlcms=enabled -Dimagequant=enabled -Dcgif=enabled --buildtype release && meson compile -C build -j $(BUILD_JOBS) && meson install -C build $(MAKE) verifyLibvips verifyLibvips: @printf '\n==> %s\n' "Verifying libvips build" pkg-config --modversion vips-cpp "$(PREFIX_PATH)/bin/vips" -l | awk '/Heif/ { found = 1 } END { exit(found ? 0 : 1) }' + "$(PREFIX_PATH)/bin/vips" -l | awk '/gifsave_buffer/ { found = 1 } END { exit(found ? 0 : 1) }' # libheif: libde265 # # libheif @@ -188,6 +209,8 @@ clean: @printf '\n==> %s\n' "Removing downloaded source directories" rm -rf \ "libwebp-$(WEBP_VERSION)" "libwebp-$(WEBP_VERSION).tar.gz" \ + "libimagequant-$(LIBIMAGEQUANT_VERSION)" "libimagequant-$(LIBIMAGEQUANT_VERSION).tar.gz" \ + "cgif-$(CGIF_VERSION)" "cgif-$(CGIF_VERSION).tar.gz" \ "libde265-$(LIBDE265_VERSION)" "libde265-$(LIBDE265_VERSION).tar.gz" \ "x265_$(X265_VERSION)" "x265_$(X265_VERSION).tar.gz" \ "libaom-$(LIBAOM_VERSION)" "libaom-$(LIBAOM_VERSION).tar.gz" \ diff --git a/test/layer.test.js b/test/layer.test.js index c4f2ea5..38a1ad2 100644 --- a/test/layer.test.js +++ b/test/layer.test.js @@ -65,9 +65,9 @@ test('sharp uses the libvips built for the layer', () => { assert.deepEqual(prebuilt, [], 'prebuilt @img/sharp-* packages must not be bundled'); }); -test('sharp reports HEIF and WebP support', () => { +test('sharp reports HEIF, WebP and GIF support', () => { const { format } = require('sharp'); - for (const name of ['heif', 'webp', 'jpeg', 'png']) { + for (const name of ['heif', 'webp', 'jpeg', 'png', 'gif']) { assert.ok(format[name].input.buffer, `${name} input`); assert.ok(format[name].output.buffer, `${name} output`); } @@ -101,6 +101,51 @@ test('resizes synthetic HEIC to WebP with correct colours', async () => { await assertQuadrantColours(data, 12); }); +// GIF output needs cgif and an image quantiser (libimagequant) in libvips, see #13. +test('resizes synthetic HEIC to GIF with correct colours', async () => { + const sharp = require('sharp'); + const { data, info } = await sharp(QUADRANTS_IMAGE).resize({ width: 240 }).gif().toBuffer({ resolveWithObject: true }); + assert.equal(info.format, 'gif'); + assert.equal(info.width, 240); + assert.equal(info.height, 160); + assert.equal(data.toString('ascii', 0, 6), 'GIF89a'); + await assertQuadrantColours(data, 12); +}); + +test('encodes and resizes animated GIF', async () => { + const sharp = require('sharp'); + const frames = await Promise.all([ + sharp(QUADRANTS_IMAGE).png().toBuffer(), + sharp(QUADRANTS_IMAGE).flop().png().toBuffer(), + ]); + const animated = await sharp(frames, { join: { animated: true } }).gif({ delay: 100, loop: 0 }).toBuffer(); + // Same pipeline as #13: load all frames, resize, keep the GIF format. + const resized = await sharp(animated, { animated: true }).resize({ width: 240 }).gif().toBuffer(); + const metadata = await sharp(resized).metadata(); + assert.equal(metadata.format, 'gif'); + assert.equal(metadata.pages, 2); + assert.equal(metadata.width, 240); + assert.equal(metadata.height, 160); + await assertQuadrantColours(resized, 12); + // The second frame is mirrored, flopping it back must restore the original layout. + await assertQuadrantColours(await sharp(resized, { page: 1 }).flop().png().toBuffer(), 12); +}); + +// Without a quantiser libvips only warns and silently writes a truecolour PNG, so check the +// colour type in the IHDR chunk (byte 25, 3 = indexed colour) rather than just the call succeeding. +test('encodes palette PNG', async () => { + const sharp = require('sharp'); + const encoded = await sharp(QUADRANTS_IMAGE).png({ palette: true }).toBuffer(); + assert.equal(encoded.toString('ascii', 12, 16), 'IHDR'); + assert.equal(encoded[25], 3, 'PNG colour type should be indexed'); + const metadata = await sharp(encoded).metadata(); + assert.equal(metadata.format, 'png'); + assert.equal(metadata.isPalette, true); + assert.equal(metadata.width, QUADRANTS_LAYOUT.WIDTH); + assert.equal(metadata.height, QUADRANTS_LAYOUT.HEIGHT); + await assertQuadrantColours(encoded, 12); +}); + test('resizes HEIC to WebP', async () => { const sharp = require('sharp'); const { info } = await sharp(TEST_IMAGE).resize({ width: 200 }).webp().toBuffer({ resolveWithObject: true });