Repository navigation
microcks test exits with code 1 (contract failure) instead of code 11 (connection) when polling loop times out #573
Description
Describe the bug
While going through the changes in PR #540 and the new documentation/error-handling.md, I noticed a gap in how the test polling loop handles timeouts.
In cmd/testExecutor.go, runTestAndWait polls the server in a loop until the test finishes or the deadline passes:
for nowInMilliseconds() < future {
testResultSummary, err := mc.GetTestResult(testResultID)
if err != nil {
return false, "", fmt.Errorf("checking test result: %w", err)
}
success = testResultSummary.Success
inProgress := testResultSummary.InProgress
if !inProgress {
break
}
time.Sleep(2 * time.Second)
}
return success, testResultID, nil
If the server hasn't finished by the time the deadline runs out, the loop exits with inProgress still true, but the function returns nil for the error. No timeout is ever flagged.
Back in cmd/test.go, the caller sees err == nil and !success, so it returns errors.ErrTestFailed:
success, testResultID, err := runTestAndWait(mc, params)
if err != nil {
return err
}
if !success {
return errors.ErrTestFailed // exit code 1
}
And cmd/exit.go maps that to exit code 1:
case stderrors.Is(err, errors.ErrTestFailed):
// A clean run with a non-conforming result — pytest-style exit 1.
return 1
The problem is that a polling timeout is not a contract conformance failure, it's an infrastructure/connection issue. According to error-handling.md, exit code 1 is strictly for "a clean run whose result did not conform". A timeout means the CLI never got a final result at all.
Expected behavior
When the polling loop runs out of time with inProgress still true, it should return a proper timeout error wrapped as KindConnection, something like:
if inProgress {
return false, testResultID, errors.Wrap(errors.KindConnection,
fmt.Errorf("test polling timed out after %dms waiting for result %s",
params.waitForMillis+10000, testResultID))
}
This would cause ExitCodeFor to return 11 and Handle to print a clear error to stderr, so CI/CD pipelines can tell apart "API contract broken" (exit 1) from "server was too slow or unreachable" (exit 11).
Actual behavior
MicrocksClient got status for test "test-hanging-1" - success: false, inProgress: true
MicrocksTester waiting for 2 seconds before checking again or exiting.
MicrocksClient got status for test "test-hanging-1" - success: false, inProgress: true
MicrocksTester waiting for 2 seconds before checking again or exiting.
... repeats until deadline ...
Test test-hanging-1: FAILURE (0ms)
Full TestResult details are available here: http://127.0.0.1:8888/#/tests/test-hanging-1
exit status 1
$ echo $?
1
The CLI exits with code 1 and falsely reports a contract test failure (FAILURE 0ms), with no error indicating that the run actually timed out.
How to Reproduce?
- Start a mock server simulating a test that remains in progress:
python3 -c '
from http.server import HTTPServer, BaseHTTPRequestHandler
class MockServer(BaseHTTPRequestHandler):
def do_POST(self):
self.send_response(201)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(b"{\"id\":\"test-hanging-1\"}")
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(b"{\"id\":\"test-hanging-1\",\"success\":false,\"inProgress\":true}")
def log_message(self, *args): pass
HTTPServer(("127.0.0.1", 8888), MockServer).serve_forever()
'
-
Run the test command with a short wait deadline:
go run main.go test "MyAPI:1.0" "http://localhost:8080" HTTP --microcksURL=http://127.0.0.1:8888 --waitFor=100milli -
Check the exit status: It returns 1 instead of 11.
Root cause: in cmd/testExecutor.go (lines 76–103), the loop ends when nowInMilliseconds() >= future, but falls through to return success, testResultID, nil without verifying if inProgress was still true.
Microcks version or git rev
main
Install method (docker-compose, helm chart, operator, docker-desktop extension,...)
CLI
Additional information
The fix is small and self-contained, only cmd/testExecutor.go needs to change (add a post-loop inProgress check). All three callers of runTestAndWait (cmd/test.go, and two places in cmd/testDryRun.go) already have if err != nil { return err } guards, so they'll handle the new error correctly without any changes.
Activity
ish-g09 commentedon Oct 9, 2026
Kindly give your take on this @Harsh4902 @Vaishnav88sk, would love to open a pr for this.
yada commentedon Oct 14, 2026
Reproduced on master (f9550f5) against a real Microcks server. I ran
microcks test … --waitFor 6secagainst an endpoint that never answers. The CLI polled 8 times, each time seeinginProgress: true, then gave up after about 17 s, printed the partial result and exited 1 (contract failure). No timeout error was reported, and the issue expects exit code 11.Cause: the polling loop in
cmd/testExecutor.goexits when the deadline passes and returnssuccess, id, nil, even if the last status was still in progress.cmd/test.gothen treats!successasErrTestFailed.Suggested fix: after the loop, if the last status was still
inProgress, return a classified timeout error instead ofnil.ish-g09 commentedon Oct 15, 2026
Thanks for verifying @yada ! I'll open a PR with the fix and test coverage shortly.
yada commentedon Oct 17, 2026
Update: PR #585 ("return KindConnection error on polling timeout") targets this, and its CI checks pass. I built the head and its
cmdtests pass. I could not confirm the behavior at runtime this time: my probe test failed instantly (FAILURE 0ms) on both master and the PR, so the polling-timeout path was never reached. It still needs a runtime check with an endpoint that really never answers.Disclosure: this comment was drafted with AI assistance, and the results were run locally.
ish-g09 commentedon Oct 17, 2026
Thanks for testing and verifying the CI/unit tests @yada!
I ran into that same instant
FAILURE (0ms)earlier when experimenting. What happens is if the target endpoint immediately resets or refuses the connection (like a dead port or 404), the Microcks server marks the test as completed right away withinProgress: false, so the CLI exits on the very first poll and never enters the timeout path.To trigger the polling timeout, the test needs to stay
inProgress: trueuntil the deadline passes. You can reproduce it in two quick ways:Use an endpoint that delays its response (e.g.
https://httpbin.org/delay/30) with a short wait flag:Because the endpoint takes 30s to respond, the server keeps
inProgress: trueuntil the 15s CLI timeout triggers, cleanly printing the timeout error and exiting with11.And in Terminal 2:
Let me know if that works or if you'd like me to change anything on the PR!