From f6efa5603eaf8419234e53f261f25fc81c07b57f Mon Sep 17 00:00:00 2001 From: Andrew Cheng Date: Tue, 7 Jul 2026 19:05:41 +0000 Subject: [PATCH 1/2] Optimize q filter to use pk__in subqueries instead of set operations The `q` complex filter composed its NOT/AND/OR operands with QuerySet .difference()/.intersection()/.union(), which compile to SQL EXCEPT/INTERSECT/UNION. For NOT in particular, the left operand is the full unfiltered queryset, so evaluating an expression like `repository_version=X AND NOT repository_version=Y` seq-scans the entire content table and sorts/hashes every leg, spilling to disk on large repositories. Rewrite the three operand evaluators to compose with `pk__in` subqueries: - _NotAction: qs.exclude(pk__in=expr.evaluate(qs).values("pk")) - _AndAction: chained qs.filter(pk__in=...) semi-joins - _OrAction: OR of Q(pk__in=...) subqueries This collapses the plan to indexed anti-/semi-joins with no full-table scan, no SetOp, and no disk-spilling sorts. Results are unchanged (each operand still filters the same base queryset on a unique pk, so the implicit de-duplication of the set operations is preserved); the existing q filter functional tests pass without modification. Benchmarked on a 200k-unit repository computing a 1000-unit diff between two repository versions (steady-state, work_mem=4MB): Version sizes Before (set-ops) After (pk__in) --------------- ------------------ ---------------- 199k units ~1400 ms ~680 ms 50k units ~450 ms ~150 ms Planning time also drops (e.g. ~1.4 ms -> ~0.5 ms) as the large inlined operand lists are replaced by parameterized subqueries. Closes #7831 Assisted by: Claude Opus 4.8 --- CHANGES/7831.bugfix | 3 +++ pulpcore/filters.py | 16 +++++++++------- 2 files changed, 12 insertions(+), 7 deletions(-) create mode 100644 CHANGES/7831.bugfix diff --git a/CHANGES/7831.bugfix b/CHANGES/7831.bugfix new file mode 100644 index 00000000000..8fa0b649419 --- /dev/null +++ b/CHANGES/7831.bugfix @@ -0,0 +1,3 @@ +Optimized the ``q`` complex filter to compose ``NOT``/``AND``/``OR`` operands using ``pk__in`` +subqueries instead of the ``EXCEPT``/``INTERSECT``/``UNION`` set operations, avoiding full-table +scans and disk-spilling sorts on large content sets. \ No newline at end of file diff --git a/pulpcore/filters.py b/pulpcore/filters.py index e225769a5c0..bcb99b5cd4f 100644 --- a/pulpcore/filters.py +++ b/pulpcore/filters.py @@ -215,7 +215,7 @@ def __init__(self, tokens): self.complexity = self.expr.complexity + 1 def evaluate(self, qs): - return qs.difference(self.expr.evaluate(qs)) + return qs.exclude(pk__in=self.expr.evaluate(qs).values("pk")) class _AndAction: def __init__(self, tokens): @@ -223,11 +223,10 @@ def __init__(self, tokens): self.complexity = sum((expr.complexity for expr in self.exprs)) + 1 def evaluate(self, qs): - return ( - self.exprs[0] - .evaluate(qs) - .intersection(*[expr.evaluate(qs) for expr in self.exprs[1:]]) - ) + result = qs + for expr in self.exprs: + result = result.filter(pk__in=expr.evaluate(qs).values("pk")) + return result class _OrAction: def __init__(self, tokens): @@ -235,7 +234,10 @@ def __init__(self, tokens): self.complexity = sum((expr.complexity for expr in self.exprs)) + 1 def evaluate(self, qs): - return self.exprs[0].evaluate(qs).union(*[expr.evaluate(qs) for expr in self.exprs[1:]]) + query = models.Q() + for expr in self.exprs: + query |= models.Q(pk__in=expr.evaluate(qs).values("pk")) + return qs.filter(query) def __init__(self, *args, **kwargs): self.filterset = kwargs.pop("filter").parent From 1e20976d39433c16558b3ad268c6e283d97d1555 Mon Sep 17 00:00:00 2001 From: Andrew Cheng Date: Wed, 8 Jul 2026 17:24:15 +0000 Subject: [PATCH 2/2] Flatten q filter operands into a single WHERE where possible Builds on the prior pk__in subquery rewrite. That change removed the set-operations, but still wrapped every operand of a NOT/AND/OR in its own pk__in subquery, so a q expression compiled to one subquery per operand and the SQL grew a nesting level for every AND/OR/NOT -- up to 8 stacked subqueries at the complexity cap. This is the "deep nesting" cost the reviewer asked about. Introduce a hybrid Q-composition. Each operand can now report an `as_q()`: - A leaf is Q-reducible only when it is a standard django-filter Filter (no custom .filter() override, no method=, no distinct=). It returns a plain `Q(field__lookup=value)` (negated for exclude filters). Opaque operands -- label filters, repository_version, href/prn/id resolvers, method filters, and ChoiceFilter (which overrides .filter() for null handling) -- return None. - NOT/AND/OR combine their children's Q objects with ~, &, | when all children are reducible. `evaluate()` then folds every reducible operand into a single flat Q and applies only the genuinely opaque operands as pk__in subqueries. An all-reducible expression -- however deeply nested -- collapses to one flat SELECT; mixed expressions keep one subquery per opaque operand. Opaque operands are always evaluated against the small base queryset, so they never re-embed an accumulated filter chain. Results are unchanged and the existing q filter functional tests (test_q_filter_valid / test_q_filter_invalid, 21 cases) pass without modification. Benchmarks (100k rows, steady-state, count(), best of 4), old set-ops vs. this change; SELECTs = number of SQL SELECT statements emitted: Repositories (all-reducible name/number fields) Expression set-ops this change ---------------------------------- -------------- -------------- name AND name 42 ms / 3 15 ms / 1 (A AND B) OR (C AND D) 79 ms / 7 20 ms / 1 (A AND (B OR C)) AND NOT D (cx 8) 144 ms / 8 19 ms / 1 name AND retain>=x AND retain