fix: Optimize TaskQueueDB matching with composite indices and EXISTS

[chrisburr]

The __generateTQMatchSQL matching query uses COUNT subqueries to check whether multi-value tables (tq_TQTo*) have rows for a given TQId. This is inefficient because COUNT scans all matching rows even when we only need to know if any row exists, and the single-column TaskIndex on TQId requires a separate lookup to fetch the Value column.

This PR makes two changes:

1. Composite (TQId, Value) indices on the multi-value tables, replacing the single-column TaskIndex on TQId. This makes the index a covering index for all subqueries that filter on both columns. The existing per-field {Field}Index on Value alone is kept for reverse lookups (e.g. BannedSites antijoin).

2. EXISTS/NOT EXISTS instead of COUNT in __generateTQMatchSQL, __generateTagSQLSubCond, and __generateTQFindSQL. EXISTS short-circuits on the first matching row rather than counting all of them.

Together these drop __generateTQMatchSQL query time from ~30 ms to ~3 ms on a production-sized DB (confirmed via EXPLAIN ANALYZE).

Note: For existing deployments the schema change only takes effect on newly created tables (__initializeDB skips tables that already exist). Production DBs need a one-off ALTER TABLE to add the composite index.

BEGINRELEASENOTES
*WorkloadManagement
FIX: Optimize TaskQueueDB matching queries by adding composite (TQId, Value) indices and replacing COUNT subqueries with EXISTS/NOT EXISTS

*Deployment
CHANGE: ALTER TABLE tq_TQToSites ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;
CHANGE: ALTER TABLE tq_TQToPlatforms ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;
CHANGE: ALTER TABLE tq_TQToJobTypes ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;
CHANGE: ALTER TABLE tq_TQToBannedSites ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;
CHANGE: ALTER TABLE tq_TQToGridCEs ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;
CHANGE: ALTER TABLE tq_TQToTags ADD INDEX idx_tqid_value(TQId, Value), DROP INDEX TaskIndex;

ENDRELEASENOTES

[chrisburr]

@fstagni At the dops it's probably worth recommending people do this:

ALTER TABLE tq_TQToSites ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToPlatforms ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToJobTypes ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToBannedSites ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToGridCEs ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToTags ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;

Even on lbwms this ran in a few hundred milliseconds so it's pretty safe to do even in a production system.

[aldbr]

Not an expert of that part of the code but, given the description of the PR, the changes look good to me

[fstagni]

My favorite llm is telling me that:

When tagMatchList is empty, the new code accepts ANY task (with zero values), but the old code required tasks to ONLY contain empty string values.

Old logic:
(SELECT COUNT(all) WHERE id=tq.TQId) = (SELECT COUNT(...) AND value='')

New logic: NOT EXISTS (... WHERE value != '')
→ Accepts tasks with 0 values OR tasks where all values = ''

This changes semantics - might incorrectly accept tasks that should be rejected.

I would say this is OK. Any other ideas?

[chrisburr]

I think you need a smarter favorite LLM, I think that comment is claiming that 0 != 0

[fstagni]

@fstagni At the dops it's probably worth recommending people do this:

ALTER TABLE tq_TQToSites ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToPlatforms ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToJobTypes ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToBannedSites ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToGridCEs ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;
ALTER TABLE tq_TQToTags ADD INDEX idx_tqid_value (TQId, Value), DROP INDEX TaskIndex;

Even on lbwms this ran in a few hundred milliseconds so it's pretty safe to do even in a production system.

I will do, but just for the sake of organization, I would:

• mark these as CHANGES in the release notes. I know they are "recommendations" and not mandatory changes, but still...
• create a 9.1.0 release at the next tagging

Like this:

• they will get into the release notes
• installing a minor release instead of a patch one will provide a decent reminder to check for changes in the release notes (something that I will anyway advertise)

[atsareg]

The TQId,Value pair is the PRIMARY KEY in this table. So, the corresponding index is anyway created. Why do you want to add it once again explicitly ?
It is probably just better to drop the "TaskIndex": ["TQId"]

[chrisburr]

It appears the defintion of the table here is inconsistent with what is actually present for LHCb so maybe this is an LHCb specific problem?

We should probably run:

  ALTER TABLE tq_TQToBannedSites ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToGridCEs ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToGridMiddlewares ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToJobTypes ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToLHCbPlatforms ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToPilotTypes ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToPlatforms ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToSites ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;
  ALTER TABLE tq_TQToTags ADD PRIMARY KEY (TQId, Value), DROP INDEX idx_tqid_value;