fix: a failed login no longer silently removes a user from the run - #46
Merged
Conversation
A client whose login failed was counted as logged in and then generated
nothing for the rest of the run, so the test reported success while
applying less load than configured.
The chain: login() fails between `status = 'logging'` and
`status = 'logged'`; @suppressError swallows the exception so the status
stays 'logging' forever (the TODO in WebClient noted this);
getLoggedInClient then throws AlreadyLoggingError for that client on
every pick; profile.ts discards that error with no log and no metric.
Meanwhile getClients printed the requested count regardless, because
`await client.login()` never rejects and `results.filter(Boolean)` drops
nothing. A run where every login failed exited 0 and printed the full
count.
The suppression model is unchanged. Per-action failures are still logged,
still recorded as rc_actions{action,status="error"} and still do not abort
the run; @suppressError stays outside @action and the login catch
rethrows, so error reporting is identical. beforeLogin's own suppression
is also untouched.
- WebClient.login: wrap the body in try/catch and roll the status back to
'not-logged', so the client is retried the next time it is picked
instead of going silently dead. The rollback is conditional because the
DDP close handler sets 'error' asynchronously, and the try starts after
the guard clauses so a concurrent attempt cannot be reset.
- Client.handshakeComplete: new field, set at the end of beforeLogin.
beforeLogin suppresses its own errors, so login cannot observe a failed
handshake; this records it, and makes prom.connected.inc() idempotent
now that a retry can run beforeLogin twice. That gauge previously
reached 279 with a ceiling of 50 during retries.
- getClients: count clients that reached 'logged' rather than array
length, report "N of M", warn separately for failed and for degraded
clients, and throw when nothing logged in. The throw reaches the
existing emit('error') and process.exit(1) in BenchmarkRunner. All
clients are still returned so failed ones stay eligible for retry.
Verified against Rocket.Chat 8.7:
- unreachable host, 10 users: "0 of 10", abort, exit 1 (was "10", exit 0)
- healthy, limiters off: "10 of 10", no warnings, metrics unchanged
- DDP limiters on, 50 users: "50 of 50" plus 30 degraded, matching
beforeLogin error 30 and connected 20
- REST limiter on, 50 users: "10 of 50" plus warning, recovering to 39
logged in over four minutes as lazy retries landed
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sampaiodiego
force-pushed
the
fix/login-failure-not-silent
branch
from
September 8, 2026 14:24
b186073 to
63f7985
Compare
sampaiodiego
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A client whose login fails is counted as logged in, then silently generates nothing for the rest of the run. The test reports success while applying less load than configured, which is the one failure mode a load generator must not have.
The chain
login()fails somewhere betweenthis.status = 'logging'andthis.status = 'logged'.@suppressErrorswallows the exception, so the status stays'logging'forever.WebClient.tshas carried the note sinced50a3ae:// TODO if an error happens, we should rollback the status to not-logged.getLoggedInClientthrowsAlreadyLoggingErrorfor that client on every future pick.profile.tsdiscardsAlreadyLoggingErrorwith no log and no metric, which is correct for its intended case and hides this one.getClientsprintsLogged users total: <requested>regardless, becauseawait client.login()never rejects andresults.filter(Boolean)drops nothing.A run where every login fails exits 0 and prints the full count. Reproduced by pointing
HOST_URLat a closed port: 10 users requested,Logged users total: 10, exit 0, no traffic.What this does not change
The suppression model stays. Per-action failures being logged, recorded as
rc_actions{action,status="error"}and not aborting the run is deliberate and predates the decorators; one simulated user hitting a 429 is a data point, not a reason to kill a 500-user test.@suppressErrorsits outside@action, so the error metric is recorded before the exception is swallowed, and this PR keeps that ordering by rethrowing.beforeLogin's own suppression, added in2a8a029, is also untouched. Those clients still log in; they are now counted and reported rather than being invisible.Changes
WebClient.loginwraps the body intry/catchand rolls the status back, closing the TODO:Three deliberate details. The rollback is conditional, because the DDP close handler sets
'error'asynchronously and an unconditional reset would re-enable retries against a dead socket. Thetrystarts after the two guard clauses, so a concurrent attempt'sAlready logging inthrow cannot reset the in-flight attempt. Andthrow erroris required, not cosmetic, so@actionstill records the error before@suppressErroreats it.Rolling back to
'not-logged'rather than'error'gives a free lazy retry through the pathgetLoggedInClientalready has forDYNAMIC_LOGIN. Retries are self-throttling at the event rate times pick probability, far gentler than the originalLOGIN_BATCHburst, which suits the dominant real failure of a rate-limit storm during the login stampede. Setting'error'would make a transient failure permanent and print a stack trace on every later pick.Client.handshakeComplete, one new field, set at the end ofbeforeLogin. SincebeforeLoginsuppresses its own errors,logincannot observe a failed handshake; this records it. It also makesprom.connected.inc()idempotent, which matters now that a retry can runbeforeLogintwice.getClientscounts the state machine instead of the array length, reportsN of M, warns separately for failed and degraded clients, and throws when nothing logged in. That throw reachesemit('error')and the existingprocess.exit(1)inBenchmarkRunner, reusing the runner-level escape hatch rather than adding one. All clients are still returned, including failed ones, so they stay eligible for retry.profile.tsneeds no changes; its state handling is already correct once the state machine is honest.53 added and 2 removed lines of substance. The rest of the
WebClient.tsdiff is the re-indent from thetryblock, so?w=1is the useful view.Verified against Rocket.Chat 8.7
Logged users total: 10, exit 0Logged users total: 0 of 10, abort message, exit 11010 of 10, no warnings, metrics unchanged50, no signal50 of 50plus30 user(s) logged in with an incomplete handshake, matchingbeforeLogin error 30andconnected 2050, permanently short10 of 50plus warning, then recovery to 39 logged in over four minutes as retries landThe third row is the case that was previously invisible: those users are logged in but skipped the settings fetch and the stream subscriptions, so they apply less load than a real client.
The fourth row also caught a pre-existing bug that the retry makes reachable.
rc_load_connectedreached 279 against a ceiling of 50, because each retry re-ranbeforeLoginpast itsinc(). ThehandshakeCompleteguard fixes it; the same scenario now reports exactly 50 while still recovering.Known tradeoff
Retries are noisy against a rate-limited server: the recovery run above logged 686 failed login attempts in four minutes. With the rate limiters off, as a load test requires, logins succeed first time and no retry pressure exists. Capping attempts before promoting a client to
'error'would bound it and is a reasonable follow-up.This fixes login rejections, not hangs. A socket that accepts and never answers still parks a client in
'logging'. That needs a login timeout;WebClient.loginPromiseis declared and unused and is the natural home for it.Noticed while in here, not changed
profile.tsthrows a plainErrorfor'error'clients, logged in full on every pick forever, and nothing transitions a client out of'error'.Client.loginOrRegisterhas no callers, so the documentedLOG_INvariable has no effect.OmnichannelClient's eightif (this.status === 'logging')guards are dead, since baseClient.loginnever sets that status. The intent looks like!== 'logged'.rc_load_login,rc_load_register,rc_load_messages,rc_load_room_join,rc_load_room_subscribeandrc_load_role_addare declared inprom.tsand never observed.Based on #45 so the diff shows only this work. Happy to retarget to
mainonce that merges.