test(cli): TestRecommendationCompletenessCommand/savingsplans-database/api-error is flaky in CI #2140

Activity

1. cristim

   added

   triagedItem has been triagedItem has been triaged

   priority/p2Backlog-worthyBacklog-worthy

   severity/mediumModerate harmModerate harm

   urgency/this-quarterWithin the quarterWithin the quarter

   impact/internalTeam-internal onlyTeam-internal only

   effort/sHoursHours

   type/bugDefectDefect

   on Oct 7, 2026
2. cristim

   self-assigned this

   on Oct 16, 2026
3. cristim commented on Oct 16, 2026

   

   cristim

   on Oct 16, 2026

   MemberAuthor

   More actions

   Claiming: working on this [claude-cli-impl]
4. cristim commented on Oct 16, 2026

   

   cristim

   on Oct 16, 2026

   MemberAuthor

   More actions

   Investigation by [claude-cli-impl] on origin/main 817ab4b. I'm releasing the claim with no code change, because the failure does not reproduce and a fix without a root cause would be speculative.

   What failed in run 37624743342 (fc23e47, Go 1.26.6, -race, Linux)

   • The assertion was require.Contains(logs, "fixture denied recommendations"), now at cmd/recommendation_completeness_test.go:78.
   • The child process got GetSavingsPlansPurchaseRecommendation ... StatusCode: 400 ... deserialization failed, failed to copy error response body, read tcp 127.0.0.1:...: use of closed network connection.
   • The fixture's 400 was correct and was counted once by the proxy. The client's own socket was closed locally while it read the error body, so the fixture JSON was never parsed.
   • This is the only occurrence in the 10 most recent failed CI runs.

   Reproduction attempts (about 5,000 runs, 0 failures)

   • The single subtest at -count=100, under -race.
   • 8, 24 and 60 parallel processes, up to about 6x CPU oversubscription, with -cpu 1,2.
   • The child process with GOMAXPROCS=1 and 2.
   • The fixture writing its response slowly, 3ms per write.
   • The full TestRecommendationCompletenessCommand: 12 processes × 6 runs, about 2,500 subtests.

   Ruled out

   • Map ordering: the SP path is sequential (collection_sp.go:121-146).
   • Context cancellation: a cancel would surface canceledErr, not the raw net error.
   • Production CloseIdleConnections: there are no callers. The only one is the child test's cleanup, which runs after Execute.
   • The fixture closing the connection: that would show EOF or reset, and the 40s deadline is not reached.

   Remaining hypotheses (unproven)

   1. A rare net/http persistConn close on a loaded 2-core runner. Candidates are closeConnIfStillIdle, the writeLoop close and the readLoop peek-fail close.
   2. A Go 1.26.6 transport issue. CI was on 1.26.6, and go.mod is now 1.26.9.
   3. A race in the child's http.DefaultTransport swap. This is unlikely, because the SDK uses its own client.

   Next step if it recurs: capture the failing run ID and Go version. To instrument it, build with a -overlay copy of persistConn.closeLocked that prints a stack when the connection closes mid-body, and run that on Linux with limited CPU.

   I'm adding status/needs-info until a second occurrence gives more data.
5. cristim

   added

   status/needs-infoAwaiting clarification from the askerAwaiting clarification from the asker

   on Oct 16, 2026
6. cristim

   removed their assignment

   on Oct 16, 2026