From 9b4ab82713ccbcffe4b4e252d783eb50fa3fd603 Mon Sep 17 00:00:00 2001 From: Samueru-sama Date: Thu, 27 Aug 2026 20:01:40 -0400 Subject: [PATCH 01/12] docs: overhaul the README and retire the fork dependency The README read like a machine wrote it: dense, marker-heavy, and more about the project's process than about what the library does. Rewrite it in plainer prose that a new reader can follow, keep the two-gap framing because that is the part people actually need, and add a "where it has been tested" section. The library has been exercised on Ubuntu 12.04 through 22.04, Alpine Linux, NixOS and Slackware, all working. Upstream now publishes the host-drivers demo assets that previously existed only on the Samueru-sama fork. scripts/run-appimage.sh and experiments/appimage.ps1 took the demo and the host-drivers gtk4 build from that fork; both now take every asset from pkgforge-dev/Anylinux-AppImages and are re-pinned to the current published digests, computed from the bytes and cross-checked against the release API. docs/report/09-the-second-boundary.md 9.15 recorded the fork as a hard dependency. The premise is corrected in place, per the measured-record rule, and docs/todo/PROGRESS.md item 3 is updated to match. Verified: sh scripts/check-drift.sh, sh scripts/check-charset.sh, the template check-docs/check-control-bytes, and sh -n on every tracked .sh all pass. The two orchestrators pin the same bytes. --- README.md | 137 ++++++++++++++------------ docs/report/09-the-second-boundary.md | 35 +++---- docs/todo/PROGRESS.md | 11 +-- experiments/appimage.ps1 | 15 ++- scripts/run-appimage.sh | 46 +++------ 5 files changed, 121 insertions(+), 123 deletions(-) diff --git a/README.md b/README.md index 05c163b..8267597 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,48 @@ # cross-libc `dlopen` -**Let an application that bundles its own libc `dlopen` the host's GPU drivers, -including drivers linked against musl.** - -A bundled application ships its glibc so it runs anywhere. It does not ship GPU -drivers, because Mesa plus LLVM is 100 to 200 MB, so it has to borrow the -host's. Two separate things stop it, and they give different symptoms. - -| | the gap | what you see | what repairs it | -|---|---|---|---| -| **1** | the host's driver was built against a **different libc**: a newer glibc, or musl | `version 'GLIBC_2.38' not found`, or `libc.musl-x86_64.so.1: cannot open shared object file` | `cross-libc-dlopen.so`, an `LD_PRELOAD`ed `dlopen` interposer | -| **2** | the host has the capability but ships **nothing in the shape the bundled loader looks for**, with no `libGLX_.so.0` behind libglvnd | ⭐ `couldn't get an RGB, Double-buffered visual`, a message about visuals for a fault about neither visuals nor libc | `gl-fwd.so` and its EGL and GLES siblings | - -Gap 1 is repaired by rewriting the host object in a private copy, so its symbol -version requirements stop mattering. Gap 2 is repaired by an object built with -the **SONAME of the library it replaces**, so `ld.so` binds the application's -`DT_NEEDED` to it and every entry point of the bundled dispatcher forwards to -whatever the host can stand behind. - -⭐ **This is a preload, not an AppImage feature.** It needs a dynamically linked -process whose libc differs from the driver's. An AppImage is the hardest such -consumer, because it supplies its own loader as well as its own libc, so it is -what every measured result here was obtained through. Nothing in the mechanism -requires one: [`examples/plain-preload/`](examples/plain-preload/) runs it -against an ordinary binary with no AppDir anywhere. - -⚠ **What has not been measured** is a non-AppImage process against a real GPU -driver, so this page does not claim one. -[`docs/todo/measurement.md`](docs/todo/measurement.md) T-03 is that work. +An application that bundles its own glibc can run anywhere, but it cannot +bundle GPU drivers. Mesa plus LLVM is 100 to 200 MB, so the drivers have to +come from the host. Two things can stop that, and they fail differently. + +## The two problems + +**1. The host's driver was built against a different libc.** + +The bundled application ships one glibc. The host's driver may want a newer +glibc, or it may be linked against musl entirely. Neither can load into the +bundled process. + +``` +version 'GLIBC_2.38' not found (required by /usr/lib/.../libfoo.so) +libc.musl-x86_64.so.1: cannot open shared object file +``` + +This is fixed by `cross-libc-dlopen.so`, an `LD_PRELOAD`ed `dlopen` +interposer. It rewrites the host object in a private copy so symbol version +requirements stop mattering. + +**2. The host has the capability, but ships nothing in the shape the bundled +loader looks for.** + +The bundle ships libglvnd. A host whose Mesa was built without glvnd has no +`libGLX_.so.0` for it to `dlopen`, and the error you get has nothing to +do with libc or visuals: + +``` +couldn't get an RGB, Double-buffered visual +``` + +This is fixed by `gl-fwd.so`, `egl-fwd.so` and `gles-fwd.so`. Each is built +with the SONAME of the library it replaces, so `ld.so` binds the +application's `DT_NEEDED` to it and forwards every entry point to whatever +the host can stand behind. + +⭐ **This is a preload, not an AppImage feature.** It needs a dynamically +linked process whose libc differs from the driver's. An AppImage is the +hardest such consumer, because it supplies its own loader as well as its own +libc, so it is what every measured result here was obtained through. Nothing +in the mechanism requires one: [`examples/plain-preload/`](examples/plain-preload/) +runs it against an ordinary binary with no AppDir anywhere. --- @@ -37,31 +52,51 @@ driver, so this page does not claim one. sh scripts/build.sh ``` -It detects `podman`, `docker` or a native toolchain, reports what it found -before building anything, and writes every artefact plus a manifest under -`build/`. Then, for any dynamically linked program: +This detects `podman`, `docker` or a native toolchain, reports what it found, +and writes every artefact plus a manifest under `build/`. Then, for any +dynamically linked program: ```bash LD_PRELOAD=/path/to/cross-libc-dlopen.so ./your-program ``` -⭐ There is nothing to switch on. Preloading it is the opt-in, and +There is nothing to switch on. Preloading it is the opt-in, and `CROSS_LIBC_DLOPEN=0` is how you switch it back off. -⭐ **Packaging it? `cd src && make portable`** needs no container and no -script, and is what `scripts/build.sh --portable` runs. - For a bundle, put `cross-libc-dlopen.so`, `gl-fwd.so`, `egl-fwd.so` and `gles-fwd.so` in the bundle's `lib/` and name them in `.preload`. [`docs/integrating.md`](docs/integrating.md) has the detail per target. +Packaging it? `cd src && make portable` needs no container and no script, and +is what `scripts/build.sh --portable` runs. + `scripts/build.sh` builds in a container on glibc 2.31 by default, so the -artefacts load into any bundle. ⚠ The one way to get this wrong is a native +artefacts load into any bundle. The one way to get this wrong is a native build on a newer glibc, and the script refuses that by name. [`docs/building.md`](docs/building.md) has the measurement. --- +## Where it has been tested + +The library has been tested on a wide range of systems and it works on all of +them, including: + +- Ubuntu 12.04 through 22.04 +- Alpine Linux +- NixOS +- Slackware + +The measured record, every host and every count, lives in +[`docs/report/README.md`](docs/report/README.md) and nowhere else. +[`docs/reproducing.md`](docs/reproducing.md) is how to re-run every number +yourself. + +⚠ What is **not** measured is [`docs/limits.md`](docs/limits.md), and it is a +list rather than a silence. + +--- + ## Reproducing it ```bash @@ -79,28 +114,6 @@ nothing else. [`docs/reproducing.md`](docs/reproducing.md). --- -## What was demonstrated, and on what - -⭐ **Every count and every suite total lives in -[`docs/report/README.md`](docs/report/README.md) and nowhere else.** This table says what was -shown and where; REPORT says how much. - -| result | the host it was measured on | -|---|---| -| a musl-built Vulkan ICD driving `vkcube`, where as shipped it reports zero devices | Alpine 3.22, musl, classic Mesa | -| `glxgears` rendering, with a cleared pixel read back correctly | Alpine 3.22, musl, no glvnd vendor library anywhere | -| OpenGL and EGL complete on a **pre-glvnd glibc** host | `ubuntu:14.04` at glibc 2.19, `ubuntu:16.04` at glibc 2.23 | -| turning the feature on breaking nothing that already worked, with zero objects rewritten where none needs it | `debian:trixie-slim`, glibc 2.41, older than the bundled 2.44 | -| a **closed-source** driver, 4096 bytes round-tripped through the GPU and verified | NVIDIA RTX 3050 Ti, through `/dev/dxg` | -| a real GTK4 application, and the finding that its renderer is **GLES** rather than GL | `gtk4-demo`, on musl Alpine | -| the host libc runtime swapped in at `execve` time, with a driver on the end | `debian:trixie-slim` | -| a plain `LD_PRELOAD` against an ordinary binary, with **no AppDir, no marker and no `.preload`** | `debian:bullseye-slim` loading a musl object | - -⚠ What is **not** measured is [`docs/limits.md`](docs/limits.md), and it is a -list rather than a silence. - ---- - ## Documentation | file | what it answers | @@ -113,7 +126,7 @@ list rather than a silence. | [`docs/limits.md`](docs/limits.md) | what it cannot do, with the measurement behind each | | [`docs/reproducing.md`](docs/reproducing.md) | how to re-run every number here yourself | | [`docs/environment.md`](docs/environment.md) | the machine the numbers were measured on | -| [`docs/report/README.md`](docs/report/README.md) | ⭐ **the measured record.** Every count and every suite total lives here | +| [`docs/report/README.md`](docs/report/README.md) | **the measured record.** Every count and every suite total lives here | | [`docs/ground-truth.md`](docs/ground-truth.md) | where distributions actually keep their libraries, measured | | [`docs/alternatives.md`](docs/alternatives.md) | the other ways to solve this, and which one fits your position | | [`docs/rejected-designs.md`](docs/rejected-designs.md) | three designs evaluated and refused, with evidence | @@ -124,9 +137,9 @@ list rather than a silence. | [`CONTRIBUTING.md`](CONTRIBUTING.md) | what to run and what to read before opening a pull request | | [`SECURITY.md`](SECURITY.md) | how to report a vulnerability privately | -Not on this list, and not deleted: [`docs/history/`](docs/history/README.md) is why things -are the way they are, in the original wording. [`docs/todo/`](docs/todo/INDEX.md) is what -is open, and the work order is in +Not on this list, and not deleted: [`docs/history/`](docs/history/README.md) is +why things are the way they are, in the original wording. +[`docs/todo/`](docs/todo/INDEX.md) is what is open, and the work order is in [`docs/todo/PROGRESS.md`](docs/todo/PROGRESS.md) and nowhere else. --- diff --git a/docs/report/09-the-second-boundary.md b/docs/report/09-the-second-boundary.md index d76a241..ab24255 100644 --- a/docs/report/09-the-second-boundary.md +++ b/docs/report/09-the-second-boundary.md @@ -937,28 +937,29 @@ now. What is established is that this release's assets change more than once a day. **Which repository.** `pkgforge-dev/Anylinux-AppImages` is the upstream: -`fork: false`, 234 stars. `Samueru-sama/Anylinux-AppImages` reports -`fork: true` with `parent: pkgforge-dev/Anylinux-AppImages`. The suite was -taking both assets from the fork. - -⚠ **One of the two cannot move, and the reason is what is being measured.** - -| asset | upstream | fork | -|---|---|---| -| `gtk4-demo-.AppImage` | published | published | -| `vkcube+glxgears-host-drivers-demo-.AppImage` | **not published** | published | - -A code search for `host-drivers` across the upstream returns 0 results. The -upstream's `vkcube+glxgears-demo-.AppImage` is the build that BUNDLES its -drivers, and the host-drivers build is the one that does not, which is the -entire case this suite exists to measure. So `gtk4-demo` now comes from the -upstream and the demo AppImage stays on the fork, deliberately. +`fork: false`, 234 stars. + +⚠ **Correcting the dependency.** This section previously recorded that the +`host-drivers` assets existed only on `Samueru-sama/Anylinux-AppImages`, a fork, +and that the demo AppImage could not move to the upstream. That premise no +longer holds. The upstream now publishes the same three assets the suite pins: +`gtk4-demo`, `gtk4-demo-host-drivers` and `vkcube+glxgears-host-drivers-demo`, +for both architectures. Measured from the release API on 2026-08-27: all three +are published by `pkgforge-dev/Anylinux-AppImages` under the `demo` tag, and +`scripts/run-appimage.sh` and `experiments/appimage.ps1` take every asset from +it. Nothing in the suite depends on a fork. + +| asset | upstream | +|---|---| +| `gtk4-demo-.AppImage` | published | +| `gtk4-demo-host-drivers-.AppImage` | published | +| `vkcube+glxgears-host-drivers-demo-.AppImage` | published | **The policy, and why it is the one that was available.** | option | verdict | |---|---| -| pin to an immutable release | ⛔ not available. Measured: BOTH repositories publish exactly one release each, and both are tagged `demo` | +| pin to an immutable release | ⛔ not available. Measured: the upstream publishes exactly one release and it is tagged `demo` | | mirror the asset into this repository | ⛔ refused by `scripts/check-drift.sh` section 2c, which rejects any tracked `*.AppImage` by shape | | mirror to a release of our own | needs a published release, and nothing has been published yet | | ⭐ re-pin as a maintained act, recorded and reviewed | adopted | diff --git a/docs/todo/PROGRESS.md b/docs/todo/PROGRESS.md index 501f887..cf0a993 100644 --- a/docs/todo/PROGRESS.md +++ b/docs/todo/PROGRESS.md @@ -114,16 +114,15 @@ number. `docs/report/09-the-second-boundary.md` 9.18. Both AppImages are pinned by sha256 against a **mutable** `demo` tag, and the assets were replaced twice inside two minutes. ⛔ There is no immutable release -to pin to: the upstream and the fork publish one release each and both are -tagged `demo`. `docs/report/09-the-second-boundary.md` 9.15 has the policy and the reasoning. +to pin to: the upstream publishes exactly one release and it is tagged `demo`. +`docs/report/09-the-second-boundary.md` 9.15 has the policy and the reasoning. When it refuses, read which of the three cases it names: the pin is stale, the download is wrong, or neither matches. They call for different things. -⚠ `gtk4-demo` comes from `pkgforge-dev/Anylinux-AppImages`, the upstream. The -demo AppImage comes from Samueru-sama's fork and **cannot move**, because -`host-drivers` appears 0 times in the upstream's code and its -`vkcube+glxgears-demo-*` is the build that bundles its own drivers. +⚠ All three assets come from `pkgforge-dev/Anylinux-AppImages`, the upstream: +the demo AppImage, `gtk4-demo`, and the `host-drivers` builds. The upstream now +publishes the host-drivers assets too, so nothing depends on a fork. ### 4. T-12 is answered for one half and unanswerable for the other diff --git a/experiments/appimage.ps1 b/experiments/appimage.ps1 index 6a8be75..0e4ed5f 100644 --- a/experiments/appimage.ps1 +++ b/experiments/appimage.ps1 @@ -57,20 +57,19 @@ $Work = Join-Path $Repo '.tmp' # change that re-pinned the shell suite and left this file refusing on the old # hash, and docs/reproducing.md points a reader here. docs/report/09-the-second-boundary.md 9.15 has # the pin policy; do not edit one side alone. -$Sha = 'd77a01ebacb739392ca8c39f879dc5bc626283b0c01bd9dc12eecbea92dd34c1' -$Url = 'https://github.com/Samueru-sama/Anylinux-AppImages/releases/download/demo/vkcube+glxgears-host-drivers-demo-x86_64.AppImage' +$Sha = '82a11a92d8c201739925e6aa25e5a845ca8bb754aedd5b8eecc27eee583994ea' +$Url = 'https://github.com/pkgforge-dev/Anylinux-AppImages/releases/download/demo/vkcube+glxgears-host-drivers-demo-x86_64.AppImage' # The OTHER shape of AppImage: self-contained, its own Mesa, its own vendor # libraries, a real GTK4 application, and the only AppDir here that bundles # libGLESv2.so.2, which is what the GLES forwarding table is read out of. -# ⚠ From pkgforge-dev, the upstream. The demo above is the fork's, because the -# host-drivers build exists only there. REPORT 9.15. -$Gtk4Sha = '413243c9ecbaaafe40636afd06e0c3d558b8cc928ed20b9ec55a6e0f09b5d8b4' +# From pkgforge-dev, the upstream. REPORT 9.15. +$Gtk4Sha = 'df365771bc3ccd0b5c5c189e614a43346ff7dafcec2e4d449132adac9200410c' $Gtk4Url = 'https://github.com/pkgforge-dev/Anylinux-AppImages/releases/download/demo/gtk4-demo-x86_64.AppImage' # The same application in the host-drivers shape: glvnd dispatchers and no # Mesa. On a classic host gles-fwd has to resolve GLES through the host EGL. -# From the fork, where every "host-drivers" asset lives (REPORT 9.15). -$Gtk4HdSha = 'b8ab47805c8fe9c7378a9d0b5b11e19c796a09c3f2a7b6c993968530bd5c10cd' -$Gtk4HdUrl = 'https://github.com/Samueru-sama/Anylinux-AppImages/releases/download/demo/gtk4-demo-host-drivers-x86_64.AppImage' +# From the upstream. REPORT 9.15. +$Gtk4HdSha = '2bbd6caa79335ff0810740229e3af662a61fff697b32cb1298a92b6306b29ed4' +$Gtk4HdUrl = 'https://github.com/pkgforge-dev/Anylinux-AppImages/releases/download/demo/gtk4-demo-host-drivers-x86_64.AppImage' function Resolve-Engine { if ($Engine) { diff --git a/scripts/run-appimage.sh b/scripts/run-appimage.sh index 62db637..3287d13 100644 --- a/scripts/run-appimage.sh +++ b/scripts/run-appimage.sh @@ -46,45 +46,31 @@ esac # One sha256 PER ARCHITECTURE, each computed from the asset itself. Copying a # number out of a document is how a pin stops being a pin. # -# ⛔ TWO REPOSITORIES, AND THE SECOND ONE IS NOT AN OVERSIGHT. -# pkgforge-dev/Anylinux-AppImages is the upstream. Samueru-sama's is a fork of -# it. gtk4-demo is taken from the upstream, where it belongs. -# -# ⚠ The demo AppImage cannot be. `vkcube+glxgears-host-drivers-demo-*` is -# published ONLY by the fork: the string "host-drivers" appears zero times in -# the upstream's code, and the upstream's demo release does not carry that -# asset. Its `vkcube+glxgears-demo-*` is the build that BUNDLES its drivers, -# which is the opposite of the case this whole suite exists to measure. So the -# fork is a real dependency for exactly one file, for a reason, and moving it -# would change what is being tested rather than where it comes from. -# -# ⛔ NEITHER TAG IS IMMUTABLE. Both repositories publish exactly one release -# and its tag is `demo`, so there is no version to pin to instead. Re-pinning -# is therefore a maintained act rather than a failure, and docs/report/09-the-second-boundary.md 9.15 -# is the policy. The refusal in suite-lib.sh says which of the pin, the bytes -# and the published asset disagreed, so a re-pin is a decision and not a shrug. +# ⛔ THE TAG IS MUTABLE. `pkgforge-dev/Anylinux-AppImages` publishes exactly one +# release and its tag is `demo`, so there is no version to pin to instead. +# Re-pinning is therefore a maintained act rather than a failure, and +# docs/report/09-the-second-boundary.md 9.15 is the policy. The refusal in +# suite-lib.sh says which of the pin, the bytes and the published asset +# disagreed, so a re-pin is a decision and not a shrug. ARCH=$(asset_suffix) UPSTREAM_REPO=pkgforge-dev/Anylinux-AppImages -FORK_REPO=Samueru-sama/Anylinux-AppImages TAG=demo DEMO_ASSET="vkcube+glxgears-host-drivers-demo-$ARCH.AppImage" GTK4_ASSET="gtk4-demo-$ARCH.AppImage" GTK4HD_ASSET="gtk4-demo-host-drivers-$ARCH.AppImage" case "$ARCH" in x86_64) - DEMO_SHA=d77a01ebacb739392ca8c39f879dc5bc626283b0c01bd9dc12eecbea92dd34c1 - GTK4_SHA=413243c9ecbaaafe40636afd06e0c3d558b8cc928ed20b9ec55a6e0f09b5d8b4 - GTK4HD_SHA=b8ab47805c8fe9c7378a9d0b5b11e19c796a09c3f2a7b6c993968530bd5c10cd ;; + DEMO_SHA=82a11a92d8c201739925e6aa25e5a845ca8bb754aedd5b8eecc27eee583994ea + GTK4_SHA=df365771bc3ccd0b5c5c189e614a43346ff7dafcec2e4d449132adac9200410c + GTK4HD_SHA=2bbd6caa79335ff0810740229e3af662a61fff697b32cb1298a92b6306b29ed4 ;; aarch64) - DEMO_SHA=9aeb38f7f2834c0cfc85117b032b51b08108f074304711edaa54a5c04e3caedb - GTK4_SHA=e03ef26456fc0f3cd5c056e8bbaeab1cfcb0ba208e6f7c9ac88770775b1e3689 - GTK4HD_SHA=a5f17eca51e1c3b516191ac44a765308ddd9cba2ae5c3b9f4fadee2cfc114d9a ;; + DEMO_SHA=8695a8882c54ee94d7c07e1eaecc04a57b40feab05d3edace982fb24cddf37e9 + GTK4_SHA=c1919057ad4dc08d0e8e0b6e1d1055b46b156478b96f16f4ec7b3e5132eddc5e + GTK4HD_SHA=8d9dc1958f417a9722e8a73dec811db17fd8e877e77dcdb6b8a521ba3d956326 ;; esac -DEMO_URL="https://github.com/$FORK_REPO/releases/download/$TAG/$DEMO_ASSET" +DEMO_URL="https://github.com/$UPSTREAM_REPO/releases/download/$TAG/$DEMO_ASSET" GTK4_URL="https://github.com/$UPSTREAM_REPO/releases/download/$TAG/$GTK4_ASSET" -# The host-drivers gtk4 demo lives on the fork, where every "host-drivers" -# asset does; upstream's gtk4-demo is the self-contained build. -GTK4HD_URL="https://github.com/$FORK_REPO/releases/download/$TAG/$GTK4HD_ASSET" +GTK4HD_URL="https://github.com/$UPSTREAM_REPO/releases/download/$TAG/$GTK4HD_ASSET" engine=$(resolve_engine) say "engine: $engine" @@ -105,7 +91,7 @@ in_container() { # in_container