Skip to content

fix: stabilize bulk registration admission and polling#18

Open
AmPlace wants to merge 6 commits into
HM2899:mainfrom
AmPlace:agent/fix-registration-admission-stall
Open

fix: stabilize bulk registration admission and polling#18
AmPlace wants to merge 6 commits into
HM2899:mainfrom
AmPlace:agent/fix-registration-admission-stall

Conversation

@AmPlace

@AmPlace AmPlace commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • track process-wide registration admission usage and waiter counts
  • mark queued sessions as waiting_admission and refresh their durable heartbeat
  • stop the watchdog from reclaiming healthy admission waiters as orphaned sessions
  • fail an actually saturated admission wait with an explicit timeout instead of waiting forever
  • expose admission metrics in the registration health response
  • route only the full sessions listing through an isolated 30-second bulk client while keeping single-session polls fail-fast
  • allow large bulk-registration session lists without silently truncating JSON at 4 MiB
  • return an explicit size error if the bounded 32 MiB response ceiling is exceeded
  • refresh the xAI request fingerprint from the Cloudflare-blocked chrome131 profile to the maintained chrome profile
  • report Cloudflare 403, empty upstream responses, and missing JavaScript chunks with actionable diagnostics
  • avoid the misleading max_workers must be greater than 0 error when the live signup page contains no JavaScript chunks
  • document the timeout/heartbeat settings and run the new unit tests in CI

Root causes

  1. Batch workers changed a session to started before polling the global registration semaphore. While all slots were occupied, that polling loop had neither a timeout nor a session heartbeat. The watchdog could therefore classify a healthy queued worker as stale, set cancel_requested, and the worker would later report cancelled while waiting for admission.

  2. The full sessions listing used the same 750 ms client as small, high-frequency status polls. A real Docker Desktop list took 2.09 seconds to produce its first byte, so the start-registration preflight failed before POST /jobs was sent.

  3. The registration client silently read only the first 4 MiB of every sidecar response. The same real 1,000-session list was 5,486,793 bytes, so after extending the timeout its valid JSON was cut in the middle and surfaced as unexpected end of JSON input.

  4. The vendored registration client hard-coded curl_cffi's chrome131 impersonation profile. The current xAI/Cloudflare signup endpoint returned HTTP 403 for that profile, while the maintained chrome profile and a normal browser returned HTTP 200. Because the 403 page contained no expected Next.js chunks, the parser later produced the unrelated max_workers must be greater than 0 message instead of identifying the blocked fingerprint.

The bulk-list client is isolated so this fix does not make ordinary single-session polling wait for a slow sidecar.

Verification

  • python -m unittest discover -s tests -p "test_*.py" (20 consecutive runs for the admission/polling changes)
  • python -m py_compile grok2api/upstream/registration_admission.py grok2api/upstream/grok_build_adapter.py
  • go test ./internal/registration/client ./internal/server -count=1
  • 5 MiB sessions response regression test
  • short-vs-bulk HTTP client routing regression test
  • 7/7 offline signup-page/fingerprint regression tests passed
  • Python syntax checks passed for the updated xconsole client modules
  • real Docker sidecar registration verified end to end with the maintained chrome fingerprint:
    • signup page HTTP 200
    • Turnstile solved
    • email verification code received
    • account created
    • SSO/OAuth completed
    • account imported
    • post-import health probe: ok=1 fail=0
  • git diff --check

@AmPlace
AmPlace marked this pull request as ready for review July 21, 2026 17:42
@AmPlace AmPlace changed the title fix: prevent registration admission stalls fix: stabilize bulk registration admission and polling Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant