Skip to content

fix: a failed login no longer silently removes a user from the run - #46

Merged
sampaiodiego merged 1 commit into
mainfrom
fix/login-failure-not-silent
Sep 8, 2026
Merged

fix: a failed login no longer silently removes a user from the run#46
sampaiodiego merged 1 commit into
mainfrom
fix/login-failure-not-silent

Conversation

@Freedom101

Copy link
Copy Markdown
Collaborator

A client whose login fails is counted as logged in, then silently generates nothing for the rest of the run. The test reports success while applying less load than configured, which is the one failure mode a load generator must not have.

The chain

  1. login() fails somewhere between this.status = 'logging' and this.status = 'logged'.
  2. @suppressError swallows the exception, so the status stays 'logging' forever. WebClient.ts has carried the note since d50a3ae: // TODO if an error happens, we should rollback the status to not-logged.
  3. getLoggedInClient throws AlreadyLoggingError for that client on every future pick.
  4. profile.ts discards AlreadyLoggingError with no log and no metric, which is correct for its intended case and hides this one.
  5. getClients prints Logged users total: <requested> regardless, because await client.login() never rejects and results.filter(Boolean) drops nothing.

A run where every login fails exits 0 and prints the full count. Reproduced by pointing HOST_URL at a closed port: 10 users requested, Logged users total: 10, exit 0, no traffic.

What this does not change

The suppression model stays. Per-action failures being logged, recorded as rc_actions{action,status="error"} and not aborting the run is deliberate and predates the decorators; one simulated user hitting a 429 is a data point, not a reason to kill a 500-user test. @suppressError sits outside @action, so the error metric is recorded before the exception is swallowed, and this PR keeps that ordering by rethrowing.

beforeLogin's own suppression, added in 2a8a029, is also untouched. Those clients still log in; they are now counted and reported rather than being invisible.

Changes

WebClient.login wraps the body in try/catch and rolls the status back, closing the TODO:

if (this.status === 'logging') {
    this.status = 'not-logged';
}
throw error;

Three deliberate details. The rollback is conditional, because the DDP close handler sets 'error' asynchronously and an unconditional reset would re-enable retries against a dead socket. The try starts after the two guard clauses, so a concurrent attempt's Already logging in throw cannot reset the in-flight attempt. And throw error is required, not cosmetic, so @action still records the error before @suppressError eats it.

Rolling back to 'not-logged' rather than 'error' gives a free lazy retry through the path getLoggedInClient already has for DYNAMIC_LOGIN. Retries are self-throttling at the event rate times pick probability, far gentler than the original LOGIN_BATCH burst, which suits the dominant real failure of a rate-limit storm during the login stampede. Setting 'error' would make a transient failure permanent and print a stack trace on every later pick.

Client.handshakeComplete, one new field, set at the end of beforeLogin. Since beforeLogin suppresses its own errors, login cannot observe a failed handshake; this records it. It also makes prom.connected.inc() idempotent, which matters now that a retry can run beforeLogin twice.

getClients counts the state machine instead of the array length, reports N of M, warns separately for failed and degraded clients, and throws when nothing logged in. That throw reaches emit('error') and the existing process.exit(1) in BenchmarkRunner, reusing the runner-level escape hatch rather than adding one. All clients are still returned, including failed ones, so they stay eligible for retry.

profile.ts needs no changes; its state handling is already correct once the state machine is honest.

53 added and 2 removed lines of substance. The rest of the WebClient.ts diff is the re-indent from the try block, so ?w=1 is the useful view.

Verified against Rocket.Chat 8.7

Scenario Before After
Unreachable host, 10 users Logged users total: 10, exit 0 Logged users total: 0 of 10, abort message, exit 1
Healthy, limiters off, 10 users 10 10 of 10, no warnings, metrics unchanged
DDP limiters on, 50 users 50, no signal 50 of 50 plus 30 user(s) logged in with an incomplete handshake, matching beforeLogin error 30 and connected 20
REST limiter on, 50 users 50, permanently short 10 of 50 plus warning, then recovery to 39 logged in over four minutes as retries land

The third row is the case that was previously invisible: those users are logged in but skipped the settings fetch and the stream subscriptions, so they apply less load than a real client.

The fourth row also caught a pre-existing bug that the retry makes reachable. rc_load_connected reached 279 against a ceiling of 50, because each retry re-ran beforeLogin past its inc(). The handshakeComplete guard fixes it; the same scenario now reports exactly 50 while still recovering.

Known tradeoff

Retries are noisy against a rate-limited server: the recovery run above logged 686 failed login attempts in four minutes. With the rate limiters off, as a load test requires, logins succeed first time and no retry pressure exists. Capping attempts before promoting a client to 'error' would bound it and is a reasonable follow-up.

This fixes login rejections, not hangs. A socket that accepts and never answers still parks a client in 'logging'. That needs a login timeout; WebClient.loginPromise is declared and unused and is the natural home for it.

Noticed while in here, not changed

  • profile.ts throws a plain Error for 'error' clients, logged in full on every pick forever, and nothing transitions a client out of 'error'.
  • Client.loginOrRegister has no callers, so the documented LOG_IN variable has no effect.
  • OmnichannelClient's eight if (this.status === 'logging') guards are dead, since base Client.login never sets that status. The intent looks like !== 'logged'.
  • rc_load_login, rc_load_register, rc_load_messages, rc_load_room_join, rc_load_room_subscribe and rc_load_role_add are declared in prom.ts and never observed.

Based on #45 so the diff shows only this work. Happy to retarget to main once that merges.

Base automatically changed from fix/rocket-chat-8-compatibility to main September 8, 2026 14:22
A client whose login failed was counted as logged in and then generated
nothing for the rest of the run, so the test reported success while
applying less load than configured.

The chain: login() fails between `status = 'logging'` and
`status = 'logged'`; @suppressError swallows the exception so the status
stays 'logging' forever (the TODO in WebClient noted this);
getLoggedInClient then throws AlreadyLoggingError for that client on
every pick; profile.ts discards that error with no log and no metric.
Meanwhile getClients printed the requested count regardless, because
`await client.login()` never rejects and `results.filter(Boolean)` drops
nothing. A run where every login failed exited 0 and printed the full
count.

The suppression model is unchanged. Per-action failures are still logged,
still recorded as rc_actions{action,status="error"} and still do not abort
the run; @suppressError stays outside @action and the login catch
rethrows, so error reporting is identical. beforeLogin's own suppression
is also untouched.

- WebClient.login: wrap the body in try/catch and roll the status back to
  'not-logged', so the client is retried the next time it is picked
  instead of going silently dead. The rollback is conditional because the
  DDP close handler sets 'error' asynchronously, and the try starts after
  the guard clauses so a concurrent attempt cannot be reset.
- Client.handshakeComplete: new field, set at the end of beforeLogin.
  beforeLogin suppresses its own errors, so login cannot observe a failed
  handshake; this records it, and makes prom.connected.inc() idempotent
  now that a retry can run beforeLogin twice. That gauge previously
  reached 279 with a ceiling of 50 during retries.
- getClients: count clients that reached 'logged' rather than array
  length, report "N of M", warn separately for failed and for degraded
  clients, and throw when nothing logged in. The throw reaches the
  existing emit('error') and process.exit(1) in BenchmarkRunner. All
  clients are still returned so failed ones stay eligible for retry.

Verified against Rocket.Chat 8.7:
- unreachable host, 10 users: "0 of 10", abort, exit 1 (was "10", exit 0)
- healthy, limiters off: "10 of 10", no warnings, metrics unchanged
- DDP limiters on, 50 users: "50 of 50" plus 30 degraded, matching
  beforeLogin error 30 and connected 20
- REST limiter on, 50 users: "10 of 50" plus warning, recovering to 39
  logged in over four minutes as lazy retries landed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sampaiodiego
sampaiodiego force-pushed the fix/login-failure-not-silent branch from b186073 to 63f7985 Compare September 8, 2026 14:24
@sampaiodiego
sampaiodiego merged commit d1ed50d into main Sep 8, 2026
3 checks passed
@sampaiodiego
sampaiodiego deleted the fix/login-failure-not-silent branch September 8, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants