Skip to content

Add limit option for bounded prefetch, and drain for graceful shutdown - #9

Draft
himynameisjonas wants to merge 1 commit into
socketry:mainfrom
himynameisjonas:add-concurrency-limit-and-drain
Draft

Add limit option for bounded prefetch, and drain for graceful shutdown#9
himynameisjonas wants to merge 1 commit into
socketry:mainfrom
himynameisjonas:add-concurrency-limit-and-drain

Conversation

@himynameisjonas

Copy link
Copy Markdown

We run this processor in production for LLM jobs (long-running, IO-bound, ~25k jobs/day) and hit two operational issues that this PR addresses.

Bounded prefetch (limit:). The dequeue loop currently fetches jobs as fast as they arrive, so any backlog accumulates as spawned tasks inside the server process rather than in Redis. In our deployment this had three effects: the ready list read zero while ~1,100 claimed jobs were parked in-process (we were watching queue depth to detect backlog, and it lied), memory grew with the backlog since each parked job holds its deserialized payload, and every shutdown abandoned the whole pile at once, to be requeued and re-run by the recovery mechanism. With limit: n the server acquires a semaphore slot before BRPOPLPUSH, so at most n jobs are claimed at a time and backlog stays in the ready list where it is visible and cheap. Without the option, behavior is unchanged.

Graceful drain (#drain). Stopping the server cancels running jobs mid-flight; the abandoned-job recovery then requeues and re-runs them. For at-least-once processing that is correct but expensive when jobs are costly to repeat (in our case, one LLM call per job, plus duplicated side effects, on every deploy). drain(timeout:) stops the fetch loop and waits for running jobs to finish, up to the timeout. Jobs that do not finish in time are cancelled and recovered exactly as before. To support this, job tasks now run on their own barrier hosted outside the fetch loop's task tree, so stopping the loop does not cancel them; #stop stops both, preserving current semantics.

Both features are in production for us (as a subclass carrying these exact changes): deploys went from cancelling and re-running 60-920 jobs each to finishing the running jobs in ~10s and leaving the backlog in the ready list.

Tests included for both. Happy to split this into two PRs if you prefer, or rework the approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant