feat: use temporary table instead of large IN query in PilotAgentsDB

[chaen]

BEGINRELEASENOTES

*WMS

CHANGE: use temporary table instead of large IN statement in PilotAgentsDB

ENDRELEASENOTES

[aldbr]

I am not fully sure to understand what you are trying to solve exactly here, can you provide further details please?

I assume it's because of:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/PilotStatusAgent.py

Lines 85 to 98 in 42c75a1

	result = self.pilotDB.selectPilots(
	{"Status": PilotStatus.PILOT_TRANSIENT_STATES}, older=timeLimitToConsider, timeStamp="LastUpdateTime"
	)
	if not result["OK"]:
	self.log.error("Failed to get the Pilot Agents")
	return result
	if not result["Value"]:
	return S_OK()
	refList = result["Value"]
	result = self.pilotDB.getPilotInfo(refList)
	if not result["OK"]:
	self.log.error("Failed to get Info for Pilot Agents")
	return result

If it's the case, I am wondering why we don't just simplify by having a getPilotInfoForSelection method (merging selectPilots and getPilotInfo into a single SQL request). Or even having selectPilots returning any pilot information? (that would make a breaking change in the API here)

[chaen]

the problem is the query itself.

select * from Table where column in (1,2,3,4,...,126456464)

is not efficient.

On the other hand, this is very fast

create temporary table tmp_table(column INT PRIMARY KEY);
insert into tmp_table (column) values (1),(2),(3),(4),...,(126456464);
select * from table t join temporary table tt on t.column = tt.column;

Of course, it's for large number of values.

[aldbr]

the problem is the query itself.

select * from Table where column in (1,2,3,4,...,126456464)

is not efficient.

This I understand, but it looks like the query itself could be avoided by returning the pilot info instead of just the pilot reference from selectPilots(), unless I am missing some details