fix(scripts): use gzip -dc instead of zcat for portability - #51
Draft
malteos wants to merge 1 commit into
Draft
Conversation
On macOS (and other BSD systems) /usr/bin/zcat is the compress-era tool:
it unconditionally appends `.Z` to its argument and cannot read gzip
files at all. Every script here feeds it `.gz` input, so the graph
exploration and processing workflows fail on macOS:
$ zcat cc-main-2026-jun-jul-aug-host-vertices.paths.gz
zcat: can't stat: ...paths.gz (...paths.gz.Z): No such file or
directory
The failure is easy to miss. In graph_explore_download_webgraph.sh the
call is the left-hand side of a pipe into a `while` loop, so the
pipeline's exit status is the loop's, `set -e` does not fire, and the
script continues with an empty vertex list. It only fails later, at the
`list_webgraph_files` check, reporting missing vertex files rather than
the decompression error that caused them to be missing.
`gzip -dc` is equivalent to GNU `zcat` for both `.gz` and `.Z` input and
behaves identically on GNU, BSD and busybox, so this is a no-op on Linux.
Replaces all 31 call sites, including the two in host2domaingraph.sh
comments so the documented examples stay runnable as written.
Claude-Session: https://claude.ai/code/session_01UUzQo1VaEMKBQdt9aactiC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On macOS (and other BSD systems)
/usr/bin/zcatis not GNUzcat— it is thecompress-era tool, which unconditionally appends
.Zto its argument and cannotread gzip files at all:
Every
zcatinsrc/script/is fed.gzinput, so the graph exploration andprocessing workflows do not run on macOS.
Why the failure is hard to diagnose
In
graph_explore_download_webgraph.shthe call is the left-hand side of a pipeinto a
whileloop:The pipeline's exit status is the
whileloop's, which succeeds trivially onempty input, so
set -enever fires. The script continues, downloads zero vertexpart files, and only fails afterwards at the
list_webgraph_filescheck —reporting
Missing vertices.*.txt.gz ()rather than the decompression error thatcaused the files to be missing. The reported symptom is several steps removed
from the cause.
The same shape recurs in
graph_explore_build_vertex_map.sh, where theiepmbuild silently falls through to the
mph/fclpath.Fix
Replace
zcatwithgzip -dcat all 31 call sites across 6 scripts.gzip -dcaccepts the same inputs as GNUzcat— including.Z, which GNUzcatalso handles — and behaves identically on GNU, BSD and busybox. There isno behavioural change on Linux; this is purely a portability fix.
The two occurrences in
host2domaingraph.shcomments are included so thedocumented example commands remain runnable as written.
Alternatives considered
zcat()shell function wrapper. Much smaller diff, but it does notsurvive a process boundary:
process_webgraph.shinvokesbash -c "zcat ...",which would silently fall back to the system
zcatunless the function werealso
export -f'd. It also shadows a real command name.$ZCATvariable. Avoids shadowing, but touches the same call sites asthe literal replacement while adding a definition to keep in sync.
Given equal churn, the direct replacement is the option with nothing to explain
at the call site.
Verification
bash -nclean on all six changed scripts.graph_explore_download_webgraph.shvertex-pathexpansion verified against the live
cc-main-2026-jun-jul-aughost graph:48 vertex part paths decoded and fetched.
graph_explore_build_vertex_map.shCAT_VERTICESblock (both thesingle-file and the host-level
$VERTICES/*.txt.gzdirectory form) verifiedagainst real host vertex data under
set -eo pipefail.Tested on macOS 26.5 (arm64), GNU bash 5.3.
Note on a pre-existing issue (not addressed here)
Independent of
zcat: ingraph_explore_build_vertex_map.sh, theiepmpathbranches on
[ -d $VERTICES ]to handle the host-level multi-file layout, butthe
mph/fcl/smphfallback below it (lines 129 and 138) does not — it passes$VERTICESas though it were a single file. That fallback only runs when theiepmbuild fails, so it is not reachable in the normal path. Left alone to keepthis PR to one concern; happy to open a separate issue.