Skip to content

test(evm): cap child_process.exec in lib.js to surface stalled commands#3526

Open
wen-coding wants to merge 1 commit into
mainfrom
wen/exec_command_timeout
Open

test(evm): cap child_process.exec in lib.js to surface stalled commands#3526
wen-coding wants to merge 1 commit into
mainfrom
wen/exec_command_timeout

Conversation

@wen-coding
Copy link
Copy Markdown
Contributor

Summary

execCommand in contracts/test/lib.js wraps child_process.exec with no timeout. The Autobahn EVM integration matrix has hit multiple 30‑minute job timeouts where the hardhat process went silent at a test boundary, with the orphan‑process listing at cleanup consistently showing a stalled docker exec ... seid ... child still alive. The job then consumed its entire wall‑clock budget instead of surfacing a useful error.

This caps execCommand with a configurable timeout (default 60s, SIGKILL). On expiry, the rejection includes the offending command so the next occurrence is a 60s actionable error instead of a 30‑minute mystery. Override via EXEC_TIMEOUT_MS for tests that legitimately need longer.

Observed hang pattern (7 occurrences in 3 days)

Date (UTC) Run Affected Autobahn matrix job Branch
05‑28 23:48 26608835541 EVM Interoperability chore branch
05‑29 00:06 26609440164 EVM Interoperability chore branch
05‑29 06:00 26620669539 EVM Interoperability chore branch
05‑29 14:19 26642258916 EVM Interoperability merge_queue #3506
05‑29 16:22 26648904362 EVM Interoperability main push
05‑29 16:29 26648905384 EVM Interoperability merge_queue #3511
05‑31 01:00 26699345153 EVM Module PR #3525

Each ran for exactly ~30 minutes (job timeout). Investigation of one of the artifacts (26648905384): validators were perfectly healthy (continuing to produce blocks) the whole time, no test‑sender activity reached them during the silence — i.e., the hardhat process was alive but never sent another RPC, consistent with a stalled child process here.

Trade-off

  • Healthy docker exec calls in the suite complete in ~100–200ms; 60s is comfortably above the worst observed normal case.
  • If a specific test legitimately needs longer (e.g. genesis bootstrap), EXEC_TIMEOUT_MS overrides per-invocation.

Test plan

  • CI: full integration matrix passes (60s is well above normal).
  • The next time this stall pattern hits, the failing job will report a 60s execCommand timed out … <command> error instead of a 30‑min cancel — that error itself becomes the diagnostic.

🤖 Generated with Claude Code

The Autobahn EVM integration matrix has hit multiple 30-minute job
timeouts where hardhat went silent between test files with no error.
The orphan-process listing at cleanup consistently shows a docker exec
(running `seid tx evm send` or similar) still alive — `child_process.exec`
has no timeout by default, so a stalled CLI invocation eats the entire
job budget instead of surfacing as a clear error.

Add a default 60s timeout to execCommand with a SIGKILL kill signal.
On expiry, throw an error containing the offending command so the next
occurrence is a 60s actionable error instead of a 30-minute mystery.
Override via EXEC_TIMEOUT_MS for tests that legitimately need longer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cursor
Copy link
Copy Markdown

cursor Bot commented May 31, 2026

PR Summary

Low Risk
Test-harness-only change with no production, auth, or on-chain behavior; the only operational risk is a false timeout if a legitimate test exceeds the default without setting EXEC_TIMEOUT_MS.

Overview
execCommand in contracts/test/lib.js now passes a timeout (default 60s via EXEC_TIMEOUT_MS) and killSignal: SIGKILL to child_process.exec, so hung docker exec / seid shells fail fast instead of blocking CI until the job limit.

On timeout, rejections use a dedicated message that includes the timeout and a truncated command string; other exec errors are unchanged.

This affects every path that shells out through execute / executeOnAllNodes (integration tests only).

Reviewed by Cursor Bugbot for commit c9247d6. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 31, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMay 31, 2026, 11:40 AM

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c9247d6. Configure here.

Comment thread contracts/test/lib.js
if (error.killed || error.signal) {
const summary = command.length > 200 ? command.slice(0, 197) + "..." : command
reject(new Error(`execCommand timed out after ${timeoutMs}ms: ${summary}`))
return
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overly broad signal check misattributes non-timeout kills

Low Severity

The condition error.killed || error.signal catches any signal-terminated process, not just timeout kills. error.killed is only true when Node.js itself killed the process (i.e., the timeout fired), but error.signal is set for any signal death — including external OOM kills, CI runner resource-limit kills, or cleanup scripts. In those cases the rejection message incorrectly says "execCommand timed out after ${timeoutMs}ms", sending engineers looking for timeout causes when the actual issue is something else entirely. Since this PR's whole purpose is improving diagnostics, the condition could be narrowed to just error.killed to avoid misattribution.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c9247d6. Configure here.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 31, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.30%. Comparing base (3d3df7e) to head (c9247d6).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3526      +/-   ##
==========================================
- Coverage   59.15%   58.30%   -0.85%     
==========================================
  Files        2213     2140      -73     
  Lines      182710   174341    -8369     
==========================================
- Hits       108077   101650    -6427     
+ Misses      64930    63667    -1263     
+ Partials     9703     9024     -679     
Flag Coverage Δ
sei-db 70.41% <ø> (-0.22%) ⬇️
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 74 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant