Skip to content

fix(scripts): use gzip -dc instead of zcat for portability - #51

Draft
malteos wants to merge 1 commit into
mainfrom
fix/zcat-portability
Draft

fix(scripts): use gzip -dc instead of zcat for portability#51
malteos wants to merge 1 commit into
mainfrom
fix/zcat-portability

Conversation

@malteos

@malteos malteos commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Problem

On macOS (and other BSD systems) /usr/bin/zcat is not GNU zcat — it is the
compress-era tool, which unconditionally appends .Z to its argument and cannot
read gzip files at all:

$ zcat cc-main-2026-jun-jul-aug-host-vertices.paths.gz
zcat: can't stat: cc-main-2026-jun-jul-aug-host-vertices.paths.gz
      (cc-main-2026-jun-jul-aug-host-vertices.paths.gz.Z): No such file or directory

Every zcat in src/script/ is fed .gz input, so the graph exploration and
processing workflows do not run on macOS.

Why the failure is hard to diagnose

In graph_explore_download_webgraph.sh the call is the left-hand side of a pipe
into a while loop:

zcat "$NAME-vertices".paths.gz \
    | while read path; do ... done

The pipeline's exit status is the while loop's, which succeeds trivially on
empty input, so set -e never fires. The script continues, downloads zero vertex
part files, and only fails afterwards at the list_webgraph_files check —
reporting Missing vertices.*.txt.gz () rather than the decompression error that
caused the files to be missing. The reported symptom is several steps removed
from the cause.

The same shape recurs in graph_explore_build_vertex_map.sh, where the iepm
build silently falls through to the mph/fcl path.

Fix

Replace zcat with gzip -dc at all 31 call sites across 6 scripts.

gzip -dc accepts the same inputs as GNU zcat — including .Z, which GNU
zcat also handles — and behaves identically on GNU, BSD and busybox. There is
no behavioural change on Linux; this is purely a portability fix.

The two occurrences in host2domaingraph.sh comments are included so the
documented example commands remain runnable as written.

Alternatives considered

  • A zcat() shell function wrapper. Much smaller diff, but it does not
    survive a process boundary: process_webgraph.sh invokes bash -c "zcat ...",
    which would silently fall back to the system zcat unless the function were
    also export -f'd. It also shadows a real command name.
  • A $ZCAT variable. Avoids shadowing, but touches the same call sites as
    the literal replacement while adding a definition to keep in sync.

Given equal churn, the direct replacement is the option with nothing to explain
at the call site.

Verification

  • bash -n clean on all six changed scripts.
  • The previously failing graph_explore_download_webgraph.sh vertex-path
    expansion verified against the live cc-main-2026-jun-jul-aug host graph:
    48 vertex part paths decoded and fetched.
  • The graph_explore_build_vertex_map.sh CAT_VERTICES block (both the
    single-file and the host-level $VERTICES/*.txt.gz directory form) verified
    against real host vertex data under set -eo pipefail.

Tested on macOS 26.5 (arm64), GNU bash 5.3.

Note on a pre-existing issue (not addressed here)

Independent of zcat: in graph_explore_build_vertex_map.sh, the iepm path
branches on [ -d $VERTICES ] to handle the host-level multi-file layout, but
the mph/fcl/smph fallback below it (lines 129 and 138) does not — it passes
$VERTICES as though it were a single file. That fallback only runs when the
iepm build fails, so it is not reachable in the normal path. Left alone to keep
this PR to one concern; happy to open a separate issue.

On macOS (and other BSD systems) /usr/bin/zcat is the compress-era tool:
it unconditionally appends `.Z` to its argument and cannot read gzip
files at all. Every script here feeds it `.gz` input, so the graph
exploration and processing workflows fail on macOS:

    $ zcat cc-main-2026-jun-jul-aug-host-vertices.paths.gz
    zcat: can't stat: ...paths.gz (...paths.gz.Z): No such file or
    directory

The failure is easy to miss. In graph_explore_download_webgraph.sh the
call is the left-hand side of a pipe into a `while` loop, so the
pipeline's exit status is the loop's, `set -e` does not fire, and the
script continues with an empty vertex list. It only fails later, at the
`list_webgraph_files` check, reporting missing vertex files rather than
the decompression error that caused them to be missing.

`gzip -dc` is equivalent to GNU `zcat` for both `.gz` and `.Z` input and
behaves identically on GNU, BSD and busybox, so this is a no-op on Linux.

Replaces all 31 call sites, including the two in host2domaingraph.sh
comments so the documented examples stay runnable as written.

Claude-Session: https://claude.ai/code/session_01UUzQo1VaEMKBQdt9aactiC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant