e2e: retry event POST to broker in subscription tests#3929
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gauron99 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
TestMetadata_Subscriptions_Raw fails intermittently (and consistently on current CI runners) with "Broker rejected event: code: 500". The test waits for the Deployment to be Available and the Trigger to be Ready, but neither guarantees that the subscriber Service has ready endpoints programmed on the node yet. The delivery chain (broker ingress -> IMC -> filter -> raw Service) is fully synchronous with no retry anywhere — the trigger has no delivery spec and the test POSTs exactly once — so a sub-second endpoint-programming window turns into a hard failure: the filter gets "connection refused" dialing the Service VIP and the 500 propagates back to the one-shot producer. The non-raw variant does not hit this because its trigger targets a Knative Service, where the activator buffers and retries until the backend is up; a raw Kubernetes Service has no such shield. Fix on the producer side, per CloudEvents producer semantics: extract the duplicated broker-POST block from both subscription tests into a postEventToBroker helper that retries on transport errors and non-202 responses every 2s until accepted or a 60s deadline. Reusing the same event ID across retries is safe — a rejected POST means the synchronous fanout did not deliver, and the recorder matches by unique ID.
683b683 to
1c537a2
Compare
There was a problem hiding this comment.
Pull request overview
This PR reduces flakiness in the subscription e2e tests by retrying the CloudEvent POST to the Knative broker, addressing a race where Deployments/Triggers can be ready before Service endpoints are fully programmed.
Changes:
- Replace inline broker POST logic in two tests with a shared helper.
- Add
postEventToBrokerhelper that retries POSTing until the broker accepts the event (202) or a deadline expires.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3929 +/- ##
==========================================
+ Coverage 54.00% 54.02% +0.01%
==========================================
Files 200 200
Lines 23687 23687
==========================================
+ Hits 12792 12796 +4
+ Misses 9660 9658 -2
+ Partials 1235 1233 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add retry mechanism to the post of event to minimize flakiness/race condition
Analysis
We check
--for condition=Available deployment...which makes sure the pods are up and readiness check passes butthis does not wait for the
EndpointSlicescontroller to create/update slices from that Pod's "Ready" condition. AFAIR these ES are the source-of-truth for kube-proxy for some internal routing.Most likely what happened is that deployment was ready -> triggered the POST event (in miliseconds) before the EndpointSlices object was updated and therefore we hit a service without an endpoint.