Skip to content

feat(stovepipe): reschedule process when concurrency gate is closed#338

Merged
mnoah1 merged 1 commit into
mainfrom
mnoah1/stovepipe-process-gate-defer
Jul 20, 2026
Merged

feat(stovepipe): reschedule process when concurrency gate is closed#338
mnoah1 merged 1 commit into
mainfrom
mnoah1/stovepipe-process-gate-defer

Conversation

@mnoah1

@mnoah1 mnoah1 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What?

When the gate is closed (on first check or after losing a slot-claim race), the controller now re-enqueues the same ProcessRequest to the process topic via PublishAfter(GateWaitDelayMs) and acks the current delivery.

  • The rescheduled message gets a fresh id ({request_id}/reschedule/{unix_ms}) rather than reusing the in-flight delivery's id: the message store dedupes on (topic, partition_key, id) and the original row is still present until the ack, so a same-id republish would be silently swallowed and then deleted by the ack — permanently stranding the queue head.
  • GateWaitDelayMs must be configured positive; a zero/negative value returns an error (with a config_errors counter) rather than silently publishing with no delay. Publish failures emit publish_errors and surface the raw error so the consumer's classifier decides retryability. The controller now takes the TopicRegistry for publishing, wired through in the stovepipe server main.

Why?

Implementing proposed PublishAfter behavior from RFC: https://github.com/roychying/submitqueue/blob/chenghan.ying/stovepipe-build/buildsignal-doc/doc/rfc/stovepipe/steps/process.md#option-2-ack-and-publishafter

This allows us to reschedule the delivery for later processing, rather than having publish pause the goroutine to wait for the gate to open.

Test Plan

  • make local-stovepipe-start
  • First ingest: grpcurl -plaintext -d '{"queue":"monorepo/main"}' localhost:32784 uber.submitqueue.stovepipe.Stovepipe/Ingest
  • Manually clear from DB, but leave the build slot claimed: DELETE FROM request_uri WHERE queue='monorepo/main';
  • Ingest again: grpcurl -plaintext -d '{"queue":"monorepo/main"}' localhost:32784 uber.submitqueue.stovepipe.Stovepipe/Ingest
  • Log: "rescheduled latest head awaiting build slot" request_id=request/monorepo/main/2 in_flight_count=1 delay_ms=5000 - appears every 5 seconds

@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-process-admit branch from 2dc04df to 75abca6 Compare July 14, 2026 14:14
@behinddwalls

Copy link
Copy Markdown
Collaborator

⚠️ Automatic stack rebase failed

This PR could not be automatically rebased after its base PR was merged. The rebase hit conflicts that need manual resolution.

To fix manually:

git fetch origin
git checkout mnoah1/stovepipe-process-gate-defer
git rebase --onto origin/main 75abca64a764251e8cd57316ebd6731ca24390b1 mnoah1/stovepipe-process-gate-defer
# resolve conflicts, then:
git push --force-with-lease

Then update this PR's base branch:

gh pr edit 338 --base main

Base automatically changed from mnoah1/stovepipe-process-admit to main July 14, 2026 15:25
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-process-gate-defer branch 2 times, most recently from e738ad3 to 5b651c5 Compare July 14, 2026 16:07
@mnoah1 mnoah1 changed the title [DRAFT/WIP] feat(stovepipe): reschedule process when concurrency gate is closed feat(stovepipe): reschedule process when concurrency gate is closed Jul 14, 2026
Comment thread stovepipe/controller/process/process.go
@mnoah1
mnoah1 marked this pull request as ready for review July 14, 2026 18:26
@mnoah1
mnoah1 requested review from a team, behinddwalls and sbalabanov as code owners July 14, 2026 18:26
@mnoah1
mnoah1 enabled auto-merge July 14, 2026 18:26
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-process-gate-defer branch from f7b7d43 to b0d9072 Compare July 20, 2026 16:59
Ack the current delivery and PublishAfter the same ProcessRequest when
the latest head cannot claim a build slot, so gate waits do not burn
MaxAttempts or block the partition.
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-process-gate-defer branch from b0d9072 to 81d950d Compare July 20, 2026 17:34
@mnoah1
mnoah1 added this pull request to the merge queue Jul 20, 2026
Merged via the queue into main with commit b9615ee Jul 20, 2026
15 checks passed
@behinddwalls
behinddwalls deleted the mnoah1/stovepipe-process-gate-defer branch July 20, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants