AsyncDataloader: don't reuse another dataloader's run when nested - #5703
Open
drhops wants to merge 1 commit into
Open
AsyncDataloader: don't reuse another dataloader's run when nested#5703drhops wants to merge 1 commit into
drhops wants to merge 1 commit into
Conversation
A query executed inside another AsyncDataloader's job (e.g. from a resolver or a subscription trigger) gets its own dataloader but shares the outer task tree. `run_isolated` treated the outer task's run as its own `previous_run` and cleared `@pending_run`, so the inner `run` fell back to the outer run and re-entered `run_queue` on it: `new_queues` replaced the outer run's channel/condition, its root task waited on the old activity condition forever, and the outer request hung. Only treat the current task's run as ours when it belongs to this dataloader.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An
AsyncDataloaderrun hangs forever when a second dataloader is used inside one of its jobs and that second dataloader callsrun_isolatedbefore its firstrun.In our case, a mutation triggered a subscription from an
after_commithook, and the subscription document is executed inline (Schema.execute=> its own dataloader) from within the mutation's job fiber.Introduced with the queue-based rewrite in #5479 (2.6.4+). Not fixed by #5672/#5679, and independent of #5702 (that race is collateral of the same nesting in our case: on
masterthis repro also logsAsync::Queue::ClosedErrorfromasync_dataloader.rb:341, because the inner run closes the outer run's channel).Minimal repro
Production symptom (Puma,
graphql2.6.8,async2.36.0): every mutation that triggered an inline subscription sat idle after ~0.2s of work untilRack::Timeout(10s) fired, with the root task parked inRun#wait_for_activity, and no job tasks alive.Root cause
run_isolateddecides whether it is nested by looking at the current task's run:Inside the outer dataloader's job fiber, that run belongs to the outer dataloader, but
run_isolatedtreats it as its own nesting case and clears the inner dataloader's@pending_run. The inner dataloader's nextrun(andactive_run/append_job) then falls back the same way,@pending_run || Async::Task.current?&.graphql_async_dataloader_run, and picks up the outer run. It callsnew_queues/run_queueon it, replacing the outer run'stasks_channel/@activity, and closes them when it finishes. The outer root task is still blocked on the old@activitycondition, which nothing signals anymore, so the reactor sits inselectwith no timers or IO. Meanwhile the outer job's:finished_taskpush lands on a closed channel (ClosedError).Fix
Only treat the current task's run as ours if it belongs to this dataloader:
Used by
active_run,run_isolated(previous_run) andrun.Runalready holds@dataloader, but this exposes it. Same-dataloader nesting (run_isolatedfrom a resolver of the same query) is unchanged; a foreign run is now simply not a candidate, so the inner dataloader keeps its own@pending_runand runs its own queues.Tests
spec/graphql/dataloader/async_dataloader_spec.rb: "when another dataloader runs inside a job, it doesn't take over the outer dataloader's run".