From 49aac8735b4db3262b9a5ba5d0a818f665b9e5a2 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 11 Sep 2026 21:42:03 +0200 Subject: [PATCH 1/3] infra(app): one nginx instance, concurrency 80 anyplot-app goes from max 3 / concurrency 15 to max 1 / concurrency 80; min stays 0. A static nginx never needed a second instance for capacity, yet every crawler burst filled the 15 slots and started one or two more instances (50 AUTOSCALING starts in the ten days to 2026-09-10), because nginx holds a proxied bot request open for as long as the API takes to answer. With the API on one warm instance those waits are milliseconds, and one nginx with 80 slots absorbs the burst without a cold start. Owner decision 2026-09-11: one instance per service in both projects. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01CELiZYpFBQc5bncjWXYrue --- app/cloudbuild.yaml | 18 ++++++++++++++++-- changelog.d/app-single-instance.md | 11 +++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) create mode 100644 changelog.d/app-single-instance.md diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index 4e820ffbfb1..0f05cb1ed6d 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -83,15 +83,29 @@ steps: # Measurements behind this: #10812. - '--min-instances' - '0' + # max=1, with concurrency 80 below: one nginx is the whole capacity, and + # that is plenty — static files answer in milliseconds, memory sits at + # 15 MiB p99 of 512Mi, and the only requests that ever held a slot for + # long were bot requests proxied to the API while IT cold-started (~11 s; + # api/cloudbuild.yaml explains why that is gone). Under the old 15-slot + # limit one forged-UA burst on 2026-09-09 filled the instance and started + # two more, each a 0.26 s cold start nobody needed (50 AUTOSCALING starts + # in the ten days to 2026-09-10). A second instance was never for + # capacity; with max=1 a burst queues here for milliseconds instead of + # spreading over instances. Owner decision 2026-09-11: one instance per + # service in both projects. - '--max-instances' - - '3' + - '1' - '--port' - '8080' - '--execution-environment' - 'gen2' - '--cpu-throttling' + # 80, not 15: nginx serves static files and proxies the rest, and the API + # behind it caps its own work at 40 in flight. 15 was a default that + # turned every crawler burst into a scale-out (see --max-instances above). - '--concurrency' - - '15' + - '80' # Deploy WITHOUT routing traffic: the smoke step probes this revision on # its tag URL first, and only the promote step below shifts traffic — so # a broken nginx.conf never serves visitors or crawlers, not even between diff --git a/changelog.d/app-single-instance.md b/changelog.d/app-single-instance.md new file mode 100644 index 00000000000..16bda90bc29 --- /dev/null +++ b/changelog.d/app-single-instance.md @@ -0,0 +1,11 @@ +### Changed + +- **`anyplot-app` runs as a single instance with concurrency 80 (was max 3, + concurrency 15).** A static nginx never needed a second instance for capacity — + its files answer in milliseconds and it sits at 15 MiB of memory — yet every + crawler burst filled the 15 slots and started one or two more instances (50 + AUTOSCALING starts in the ten days to 2026-09-10), because nginx holds a + proxied bot request open for as long as the API takes to answer. With the API + on one warm instance those waits are milliseconds, and one nginx with 80 slots + absorbs the burst without a cold start. `min-instances` stays 0; the API behind + it caps its own work at 40 in flight. From 7393d4a4993d96ece7796266c4adfd610d9c4e40 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 11 Sep 2026 21:43:29 +0200 Subject: [PATCH 2/3] chore(changelog): reference #11829 in the fragment --- changelog.d/app-single-instance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/app-single-instance.md b/changelog.d/app-single-instance.md index 16bda90bc29..b2c204b41eb 100644 --- a/changelog.d/app-single-instance.md +++ b/changelog.d/app-single-instance.md @@ -8,4 +8,4 @@ proxied bot request open for as long as the API takes to answer. With the API on one warm instance those waits are milliseconds, and one nginx with 80 slots absorbs the burst without a cold start. `min-instances` stays 0; the API behind - it caps its own work at 40 in flight. + it caps its own work at 40 in flight. (#11829) From 054190c08a49dc959c589e98b38129ed1b6ffa3b Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 11 Sep 2026 21:47:25 +0200 Subject: [PATCH 3/3] chore(app): reword a comment (Copilot review) --- app/cloudbuild.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index 0f05cb1ed6d..0814b2f307f 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -86,7 +86,7 @@ steps: # max=1, with concurrency 80 below: one nginx is the whole capacity, and # that is plenty — static files answer in milliseconds, memory sits at # 15 MiB p99 of 512Mi, and the only requests that ever held a slot for - # long were bot requests proxied to the API while IT cold-started (~11 s; + # long were bot requests proxied to the API while the API cold-started (~11 s; # api/cloudbuild.yaml explains why that is gone). Under the old 15-slot # limit one forged-UA burst on 2026-09-09 filled the instance and started # two more, each a 0.26 s cold start nobody needed (50 AUTOSCALING starts