Skip to content

fix: ast-grep-runner: use --slurpfile to avoid ARG_MAX on large payloads #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bborbe merged 2 commits into from
Aug 10, 2026
Merged

fix: ast-grep-runner: use --slurpfile to avoid ARG_MAX on large payloads #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9101a1
fix: ast-grep-runner: use --slurpfile to avoid ARG_MAX on large payloads
bborbe Aug 10, 2026
531e4fa
Merge remote-tracking branch 'origin/master' into fix/argmax-slurpfile
bborbe Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/).
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.

## Unreleased

- fix: ast-grep-runner: pass finding payloads to `jq` via `--slurpfile` instead of `--argjson`, fixing `Argument list too long` on large result sets. Both sites moved: the per-owner merge and the final assembly. Payloads were being read out of files into shell variables and pushed back through `argv`; `jq` now reads the files directly, so payload size is irrelevant. Scales with finding count and matched-text size, not file count — reproduced with 1,956 findings from 429 files, which previously produced an **empty output file and exit 0**.

## v0.42.1

- fix: code-review: skip generated sources when building the scanned file set — the existing path filter (`vendor`, `node_modules`, `dist`, `build`, `coverage`) cannot catch generated code living in ordinary package directories (`k8s/client/`, `mocks/`, `zz_generated.*`). Files whose first 25 lines carry `Code generated … DO NOT EDIT` are now dropped. Measured on `bborbe/backup`: 97 of 553 findings were generated-file noise, and 44+ of the 102 that reached adjudication were refuted solely on that basis.
Expand Down
17 changes: 8 additions & 9 deletions scripts/ast-grep-runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,20 +297,19 @@ for owner_file in "$FINDINGS_DIR"/*.jsonl; do
# but owner names use '-' not '/', so the original name is preserved)
basename_no_ext="$(basename "$owner_file" .jsonl)"
owner_name="$basename_no_ext"
# Collect all findings for this owner into a JSON array
findings_array=$(jq -s '.' "$owner_file")
# Merge this owner's findings via --slurpfile, NOT --argjson.
# --argjson passes the whole payload through argv and dies with
# "Argument list too long" past ARG_MAX; --slurpfile makes jq read the
# file itself, so payload size is irrelevant.
FINDINGS_JSON_TMP="$TMPDIR_WORK/fbo_tmp.json"
jq --arg o "$owner_name" --argjson arr "$findings_array" \
jq --arg o "$owner_name" --slurpfile arr "$owner_file" \
'. + {($o): $arr}' "$FINDINGS_JSON" > "$FINDINGS_JSON_TMP" && mv "$FINDINGS_JSON_TMP" "$FINDINGS_JSON"
done

ERRORS_JSON="$(cat "$ERRORS_FILE")"
FINDINGS_OBJ="$(cat "$FINDINGS_JSON")"

jq -n \
--argjson yamls_run "$YAMLS_RUN" \
--argjson findings_count "$FINDINGS_COUNT" \
--argjson elapsed_ms "$ELAPSED" \
--argjson findings_by_owner "$FINDINGS_OBJ" \
--argjson errors "$ERRORS_JSON" \
'{stats:{yamls_run:$yamls_run, findings_count:$findings_count, elapsed_ms:$elapsed_ms}, findings_by_owner:$findings_by_owner, errors:$errors}'
--slurpfile findings_by_owner_arr "$FINDINGS_JSON" \
--slurpfile errors_arr "$ERRORS_FILE" \
'{stats:{yamls_run:$yamls_run, findings_count:$findings_count, elapsed_ms:$elapsed_ms}, findings_by_owner:$findings_by_owner_arr[0], errors:$errors_arr[0]}'
Loading