infra(api): single instance, non-blocking prewarm, OG renders off the loop

[MarkusNeusinger]

Summary

• anyplot-api runs as a single instance (min 1, max 1, concurrency 40). Every scale-out was a cold start that a live request paid for: between 2026-08-29 and 09-10 Cloud Run started 464 extra instances (16–54 a day) while the warm one sat at a concurrency of ~2, because a page load fans out 4–6 API calls in the same instant, and it pinned one call of each burst to the new instance for its full 9–13 s start — 8 to 37 browser requests a day (referer anyplot.ai, median 11.3 s) on /libraries, /languages, /stats, /specs. With one instance a burst queues on the warm one for milliseconds. Headroom: request p50 32 ms, CPU p95 6 %, more than 9 concurrent requests in ~2 minutes a day, no 429 in the window. Raising --concurrency alone would not have helped (the starts never came from the concurrency target); it is raised so the cap has headroom. If 429s ever appear, max 2 is the next step. The live service already carries this configuration (revision anyplot-api-00131-s6t, created 2026-09-10 21:41Z); the pipeline promotes it with this PR's deploy.
• The startup cache prewarm no longer blocks the port. uvicorn binds only after the lifespan returns, so the six awaited prewarm queries added a stable ~2.5 s to every cold start (2.0–3.6 s of an ~11 s start over 197 starts). It is now a background task that runs each key through get_or_set_cache — sharing the per-key lock with a request that arrives first instead of duplicating its query — and is cancelled on shutdown before the DB engine closes. init_db() stays awaited.
• OG image compositing runs in a worker thread, at most two at once. The collage and branded-image endpoints called PIL inline in the event loop, stalling every other request on the instance for the 1–3 s a render takes; a few concurrent collages produced the 647 MiB memory peak of 2026-08-26. With a single instance both would hit the whole service, so the renders go through asyncio.to_thread behind a two-slot semaphore.
• Small blocker fixed on the way (CLAUDE.md rule): a 0-byte MonoLisa cache file — what download_to_filename leaves behind when the fetch fails — was accepted as the cached font, so PIL failed on every render and the OG cards silently fell back to DejaVu for the life of the instance (and the swash test failed locally instead of skipping). Empty files now count as missing, a failed download deletes its leftover, and the failure is remembered in-process for ten minutes so a GCS outage costs one attempt per cooldown rather than one per render. Plus docs/reference/performance.md describes the live services again (frontend min 0 / 512Mi since 2026-08-29, Cloud SQL db-custom-1-3840 on a 3-year commitment).

Plan

N/A — the measurements behind this change (Cloud Monitoring, request logs, BigQuery billing export, twelve independent re-verifications) are recorded in the Todoist tasks "Nachprüfen: merkt jemand, dass anyplot-app auf null skaliert?" and "api/Dockerfile verschlanken" (both closed 2026-09-10 with the numbers).

Decisions taken here (routine, flagged for override)

• Concurrency 40, not higher: with max 1 the limit is the whole capacity, but the OG path is the one CPU-heavy request class and its own semaphore (2) is what bounds memory, not the request limit.
• The prewarm keeps its lightest-first order and swallow-and-continue semantics; only the awaiting moved.
• Font download cooldown 600 s, in-process (a dict keyed by cache filename). A per-instance negative cache, not a persisted one, so a fresh instance always tries once.

Test plan

• uv run ruff check . && uv run ruff format --check . — clean
• uv run --extra typecheck mypy api core --pretty — clean
• uv run pytest tests/unit tests/integration — 2008 passed, 1 skipped (MonoLisa italic not cached locally; the skip is the fixed behaviour, it used to fail on the 0-byte file)
• New tests: the lifespan returns while the prewarm is still running and shutdown cancels it before close_db; a key a request already computed is not recomputed by the prewarm; an empty cache file counts as missing; a failed download leaves no file and is not retried within the cooldown
• /verify-api against the branch code locally (shared prod DB, reads only): startup log shows Uvicorn running before the six Cache prewarm: … OK lines (prewarm finished 3.4 s after bind); /health, /libraries, /specs?limit=3, /stats, /plots/filter?limit=3 answer as before; /og/home.png 200 in 91 ms, /og/scatter-basic.png (collage) 200 in 1.23 s, /og/scatter-basic/python/matplotlib.png (branded) 200 in 0.73 s, /og/does-not-exist-xyz.png 404; three uncached collages requested concurrently finished in 1.0–1.4 s while /health answered in 1.5–2 ms throughout (the event loop stays free)
• api/cloudbuild.yaml parses; deploy step carries --min-instances=1 --max-instances=1 --concurrency=40; uv run python -m tools.changelog check --base origin/main passes
• After merge: Cloud Build deploy-api green and the promoted revision shows maxScale 1, containerConcurrency 40; then a week of run.googleapis.com/request_count by response_code — zero 429s expected — and the API's own request log for browser requests ≥ 5 s (referer anyplot.ai), which should drop from 8–37/day to the deploy/replacement starts only (~1–2/day)

🤖 Generated with Claude Code

https://claude.ai/code/session_01CELiZYpFBQc5bncjWXYrue

[Copilot]

🟡 Changes recommended

Concurrent render threads can race while downloading and deleting the same cached font file.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Pins the API to one warm instance, moves startup prewarming and image rendering off critical paths, and hardens font caching.

Changes:

• Configures single-instance Cloud Run deployment with concurrency 40.
• Runs cache prewarming asynchronously and bounds threaded OG rendering.
• Handles empty or failed font downloads and updates tests and documentation.

File summaries

File	Description
api/cloudbuild.yaml	Updates Cloud Run scaling and concurrency.
api/main.py	Makes cache prewarming non-blocking and cancellable.
api/routers/og_images.py	Offloads PIL rendering with bounded concurrency.
core/images.py	Validates cached fonts and adds download cooldowns.
tests/unit/api/test_main.py	Tests prewarming and lifespan behavior.
tests/unit/core/test_images.py	Tests invalid cache files and failed downloads.
docs/reference/performance.md	Documents current production configuration.
changelog.d/api-single-instance.md	Records the operational changes and fixes.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 95.16129% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
core/images.py	91.89%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!