Skip to content

callouts: build + bake m-stdlib YDB call-out .so's into the engine image#4

Open
rafael5 wants to merge 1 commit into
mainfrom
stdlib-callouts
Open

callouts: build + bake m-stdlib YDB call-out .so's into the engine image#4
rafael5 wants to merge 1 commit into
mainfrom
stdlib-callouts

Conversation

@rafael5
Copy link
Copy Markdown
Contributor

@rafael5 rafael5 commented May 31, 2026

m-stdlib's three optional modules (STDCOMPRESS / STDCRYPTO / STDHTTP) reach
native code on YottaDB via $&pkg.fn -> a .so registered through a .xc
descriptor. The running container had no such libraries, so the optional
suites couldn't run — and with a .xc descriptor present but its .so missing,
STDCOMPRESSTST hung the engine. This bakes the libraries in and wires them.

How:

  • New multi-stage Dockerfile callout-builder stage: installs gcc + the
    -dev headers (zlib1g-dev, libzstd-dev, libssl-dev, libcurl4-openssl-dev),
    compiles m-stdlib's src/callouts/*.c via tools/build-callouts.sh. The
    toolchain lives ONLY in this stage; the final image carries just the .so's
    and the already-present runtime libs (libz 1.3 / libzstd 1.5.5 / libcrypto 3
    / libcurl 4.8.0) — staying minimal.
  • Final stage: COPY the compiled .so's -> /opt/stdlib/lib, the .xc descriptors
    -> /opt/stdlib/xc, and export STDLIB_LIB + ydb_xc_stdcompress / _stdcrypto /
    _stdhttp via /etc/profile.d/stdlib-callouts.sh (sourced by bash -lc, the
    m-cli DockerEngine transport). A build-time assert fails the image if any of
    the three .so's is absent — enforcing the hang-guard invariant (never a
    descriptor without its library).
  • Makefile: callouts-stage copies the C sources + .xc from m-stdlib into the
    build context (gitignored docker/_callouts/ — single-sourced in m-stdlib,
    never vendored here); up depends on it. New test-optional runs the four
    callout-backed suites in byte mode.
  • dist/: lifecycle.json gains the callout env vars + a stdlib_callouts block +
    the test-optional target; verified_on bumped to 2026-05-30 (re-verified with
    the new image).

Verification (via m test, no hand docker exec):

  • make smoke: healthy (mumps ok, OCI labels, healthcheck, mte status).
  • make test-optional (YDB byte mode, --chset m): 4 suites, 151 assertions,
    0 failed, no hang — STDCRYPTOTST 23, STDCRYPTODOCTST 1, STDCOMPRESSTST 59,
    STDHTTPTST 68. The deployed .so's resolve the Stage-A STDCOMPRESS hang
    (which was the missing-.so case).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

m-stdlib's three optional modules (STDCOMPRESS / STDCRYPTO / STDHTTP) reach
native code on YottaDB via $&pkg.fn -> a .so registered through a .xc
descriptor. The running container had no such libraries, so the optional
suites couldn't run — and with a .xc descriptor present but its .so missing,
STDCOMPRESSTST hung the engine. This bakes the libraries in and wires them.

How:
- New multi-stage Dockerfile `callout-builder` stage: installs gcc + the
  -dev headers (zlib1g-dev, libzstd-dev, libssl-dev, libcurl4-openssl-dev),
  compiles m-stdlib's src/callouts/*.c via tools/build-callouts.sh. The
  toolchain lives ONLY in this stage; the final image carries just the .so's
  and the already-present runtime libs (libz 1.3 / libzstd 1.5.5 / libcrypto 3
  / libcurl 4.8.0) — staying minimal.
- Final stage: COPY the compiled .so's -> /opt/stdlib/lib, the .xc descriptors
  -> /opt/stdlib/xc, and export STDLIB_LIB + ydb_xc_stdcompress / _stdcrypto /
  _stdhttp via /etc/profile.d/stdlib-callouts.sh (sourced by `bash -lc`, the
  m-cli DockerEngine transport). A build-time assert fails the image if any of
  the three .so's is absent — enforcing the hang-guard invariant (never a
  descriptor without its library).
- Makefile: `callouts-stage` copies the C sources + .xc from m-stdlib into the
  build context (gitignored docker/_callouts/ — single-sourced in m-stdlib,
  never vendored here); `up` depends on it. New `test-optional` runs the four
  callout-backed suites in byte mode.
- dist/: lifecycle.json gains the callout env vars + a stdlib_callouts block +
  the test-optional target; verified_on bumped to 2026-05-30 (re-verified with
  the new image).

Verification (via `m test`, no hand docker exec):
- make smoke: healthy (mumps ok, OCI labels, healthcheck, mte status).
- make test-optional (YDB byte mode, --chset m): 4 suites, 151 assertions,
  0 failed, no hang — STDCRYPTOTST 23, STDCRYPTODOCTST 1, STDCOMPRESSTST 59,
  STDHTTPTST 68. The deployed .so's resolve the Stage-A STDCOMPRESS hang
  (which was the missing-.so case).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant