fix(reporter):added idempotency gate to prevent API server flooding#263
fix(reporter):added idempotency gate to prevent API server flooding#263LightCreator1007 wants to merge 4 commits into
Conversation
✅ Deploy Preview for node-readiness-controller canceled.
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: LightCreator1007 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @LightCreator1007. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hey @LightCreator1007, PR looks good but the diff only touches Also, a comment noting that skipping the write stops refreshing the heartbeat (so it no longer signals the reporter is alive), would be good |
Yes! Thanks for the help!! |
|
I ran the reporter locally with a 10s check interval, to track how many writes actually went to the node. from my testing, it's one write, per tick (10s interval), per node. For a component that exists to periodically report node health, that's exactly what you would expect. So, in my opinion, it is not a problem to optimise away with 1 write/tick. Is there a specific scale scenario motivating this? |
Hi @Priyankasaggu11929 , thanks for testing this! While 1 update per 10 seconds is fine locally, the project targets a scale of say 5,000 nodes. At 5,000 nodes, a 10-second interval would create 500 requests per second hitting the server just for health reporting. Processing this constant stream of "no change" updates may quickly overwhelm the API server. To prevent this, the reporter should cache its state locally and only send an update to the API server when its health status actually changes. |
Description
Adds an idempotency gate to
updateNodeConditionin thereadiness-condition-reporter.UpdateStatusis now bypassed if theStatus,Reason, andMessageare unchanged, preventing unnecessary API server flooding andetcdwrite amplification on every tick.Related Issue
NONE
Type of Change
/kind bug
Testing
main_test.golocally to assert against thefake.Clientset's tracked actions, confirming that theUpdateStatuscall is definitively bypassed when the condition state is unchanged.Checklist
make testpassesmake lintpassesDoes this PR introduce a user-facing change?