ITS: staggering

[f3sch]

This is to run the CI and possibly to run tests at P2.

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[alibuild]

Error while checking build/O2/fullCI_slc9 for bd394b8 at 2026-03-25 08:56:

## sw/BUILD/O2Physics-latest/log
c++: fatal error: Killed signal terminated program cc1plus
c++: fatal error: Killed signal terminated program cc1plus
ninja: build stopped: subcommand failed.

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 2ef458e at 2026-03-24 22:19:

## sw/BUILD/O2-full-system-test-latest/log
command /sw/slc9_x86-64/O2/15188-slc9_x86-64-local2/prodtests/full-system-test/dpl-workflow.sh had nonzero exit code 1
[10000:BadMapCalibSpec]: [22:19:04][ERROR] Insufficient statistics: 1 entries in lowE histo, do nothing
[10408:qc-task-ITS-ITSTrackTask]: [22:19:36][ERROR] Exception while running: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312. Rethrowing.
[10408:qc-task-ITS-ITSTrackTask]: [22:19:36][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312
[ERROR] Workflow crashed - PID 10408 (qc-task-ITS-ITSTrackTask) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device qc-task-ITS-ITSTrackTask: pid 10408 (exit 128)
[INFO]    - First error: [22:19:36][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312
[ERROR] SEVERE: Device qc-task-ITS-ITSTrackTask (10408) had at least one message above severity 7: Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/a1363ec459b97a7b813e711198a4757bd0352af1/slc9_x86-64/o2checkcode/1.0-local29/etc/modulefiles
++ cat
--

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 5453589 at 2026-03-25 17:30:

## sw/BUILD/O2-full-system-test-latest/log
command /sw/slc9_x86-64/O2/15188-slc9_x86-64-local1/prodtests/full-system-test/dpl-workflow.sh had nonzero exit code 1
[10361:BadMapCalibSpec]: [17:30:09][ERROR] Insufficient statistics: 0 entries in lowE histo, do nothing
[10457:qc-task-ITS-ITSTrackTask]: [17:30:30][ERROR] Exception while running: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 208. Rethrowing.
[10457:qc-task-ITS-ITSTrackTask]: [17:30:30][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 208
[ERROR] Workflow crashed - PID 10457 (qc-task-ITS-ITSTrackTask) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device qc-task-ITS-ITSTrackTask: pid 10457 (exit 128)
[INFO]    - First error: [17:30:30][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 208
[ERROR] SEVERE: Device qc-task-ITS-ITSTrackTask (10457) had at least one message above severity 7: Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 208


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/6f43f17e7871dc59a89d8f566999d06adeb4a933/slc9_x86-64/o2checkcode/1.0-local630/etc/modulefiles
++ cat
--

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 7f7c1bd at 2026-03-26 03:45:

## sw/BUILD/O2-full-system-test-latest/log
command /sw/slc9_x86-64/O2/15188-slc9_x86-64-local2/prodtests/full-system-test/dpl-workflow.sh had nonzero exit code 1
[9943:BadMapCalibSpec]: [03:45:10][ERROR] Insufficient statistics: 1 entries in lowE histo, do nothing
[10388:qc-task-ITS-ITSTrackTask]: [03:45:40][ERROR] Exception while running: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312. Rethrowing.
[10388:qc-task-ITS-ITSTrackTask]: [03:45:40][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312
[ERROR] Workflow crashed - PID 10388 (qc-task-ITS-ITSTrackTask) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device qc-task-ITS-ITSTrackTask: pid 10388 (exit 128)
[INFO]    - First error: [03:45:40][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312
[ERROR] SEVERE: Device qc-task-ITS-ITSTrackTask (10388) had at least one message above severity 7: Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 312


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/e1cb0f4095002bdd6f9111661b36c0016fa6b91c/slc9_x86-64/o2checkcode/1.0-local331/etc/modulefiles
++ cat
--

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 507223a at 2026-04-01 07:39:

## sw/BUILD/O2-full-system-test-latest/log
command /sw/slc9_x86-64/O2/slc9_x86-64-slc9_x86-64-local5/prodtests/full-system-test/dpl-workflow.sh had nonzero exit code 1
[8734:internal-dpl-ccdb-backend]: [07:23:37][ERROR] Failed to open file alien:///alice/data/CCDB/CTP/Calib/OrbitReset/01/35179/b07c53e0-b4c0-11ec-b66d-90ce809b250c?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:39][ERROR] Failed to open file alien:///alice/data/CCDB/EMC/Config/RecoParam/02/14620/d791f4c0-3ffb-11ed-a67e-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:41][ERROR] Failed to open file alien:///alice/data/CCDB/GLO/Config/GRPECS/04/02718/cf968f3f-0210-11ed-8000-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:44][ERROR] Failed to open file alien:///alice/data/CCDB/ITS/Config/ClustererParam/06/44918/b0c41285-f1ae-11ec-b9c0-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:46][ERROR] Failed to open file alien:///alice/data/CCDB/MFT/Config/ClustererParam/06/44388/8c661b10-f480-11ec-841e-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:48][ERROR] Failed to open file alien:///alice/data/CCDB/MFT/Calib/ClusterDictionary/13/20237/c9a0bd70-9fe3-11ec-975c-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:51][ERROR] Failed to open file alien:///alice/data/CCDB/CTP/Config/TriggerOffsets/05/51872/907486a2-ecaa-11ec-bea3-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:55][ERROR] Failed to open file alien:///alice/data/CCDB/CPV/Calib/Gains/13/50097/aa7cef8c-caea-11ec-8000-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:57][ERROR] Failed to open file alien:///alice/data/CCDB/CPV/Calib/Pedestals/06/01515/aa7d7bad-caea-11ec-8000-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:23:59][ERROR] Failed to open file alien:///alice/data/CCDB/CPV/Calib/BadChannelMap/12/29504/aa7c4363-caea-11ec-8000-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:01][ERROR] Failed to open file alien:///alice/data/CCDB/ZDC/Config/Module/12/06629/706c8778-60e8-11ed-9dc3-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:06][ERROR] Failed to open file alien:///alice/data/CCDB/TPC/Config/FEEPad/07/32310/3e279f1b-5e92-11ed-9dc3-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:09][ERROR] Failed to open file alien:///alice/data/CCDB/TPC/Calib/PadGainFull/10/00989/b929213c-a46e-11ec-982e-0aa2043e1b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:14][ERROR] Failed to open file alien:///alice/data/CCDB/ZDC/Calib/TDCCorr/01/30762/0d5c2790-cca6-11ec-b790-200114580d00?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:16][ERROR] Failed to open file alien:///alice/data/CCDB/ZDC/Calib/BaselineCalib/12/18437/28ea63c9-f95f-11ed-9692-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:19][ERROR] Failed to open file alien:///alice/data/CCDB/CPV/Config/CPVSimParams/02/18202/5ce9988b-d60c-11ec-8000-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:21][ERROR] Failed to open file alien:///alice/data/CCDB/PHS/Calib/L1phase/13/59044/633640d8-4b07-11ed-832f-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:24][ERROR] Failed to open file alien:///alice/data/CCDB/PHS/Calib/BadMap/10/40794/45929b61-dd0a-11ec-8000-0aa202a21b9a?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:29][ERROR] Failed to open file alien:///alice/data/CCDB/TRD/Calib/LocalGainFactor/15/61343/100756ad-4ec7-11ed-bee3-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:35][ERROR] Failed to open file alien:///alice/data/CCDB/GLO/Calib/MeanVertex/10/06884/9c31089f-2c77-11ed-ac0b-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:39][ERROR] Failed to open file alien:///alice/data/CCDB/GLO/Config/SVertexerParam/02/38730/8172363d-dfc5-11ee-82dc-200114580202?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:42][ERROR] Failed to open file alien:///alice/data/CCDB/EMC/Calib/BadChannelMap/13/27208/bd8faa80-3f56-11ed-ac0b-2a010e0a0b16?filetype=raw
[8734:internal-dpl-ccdb-backend]: [07:24:46][ERROR] Failed to open file alien:///alice/data/CCDB/EMC/Calib/TimeCalibParams/08/53931/ea66f883-3f56-11ed-ac0b-2a010e0a0b16?filetype=raw
[8757:BadMapCalibSpec]: [07:25:58][ERROR] Insufficient statistics: 0 entries in lowE histo, do nothing
[9156:qc-task-ITS-ITSTrackTask]: [07:39:39][ERROR] Exception while running: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 260. Rethrowing.
[9156:qc-task-ITS-ITSTrackTask]: [07:39:39][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 260
[ERROR] Workflow crashed - PID 9156 (qc-task-ITS-ITSTrackTask) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device qc-task-ITS-ITSTrackTask: pid 9156 (exit 128)
[INFO]    - First error: [07:39:39][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 260
[ERROR] SEVERE: Device qc-task-ITS-ITSTrackTask (9156) had at least one message above severity 7: Unhandled o2::framework::runtime_error reached the top of main of o2-qc, device shutting down. Reason: Inconsistent type and payload size at Vertices(DS/tracks1/0): type size 48  payload size 260


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/d0478a093af42beb2d89a6c601018984d03310e7/slc9_x86-64/o2checkcode/1.0-local37/etc/modulefiles
++ cat
--

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 92cd379 at 2026-04-06 00:20:

## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Common/DCAFitter/GPU/cuda/GPUInterface.cu:55:15: error: use '= default' to define a trivial destructor [modernize-use-equals-default]
++ [[ 0 == 0 ]]
++ exit 1
--

Full log here.

[f3sch]

@shahor02 from my side this is ready to merge. @mpuccio checked that for online everything should work, we only have to arrange with Barth when to do it.

[shahor02]

@f3sch fine for me.
Pinging to @Barthelemy with a question if it is ok for him to merge/tag AliceO2Group/QualityControl#2653 together with this PR (since our usual 2-step PR merging will not work in this case).
But I'll absent till April 12, so I will not be able to follow this in case of problems).

[alibuild]

Error while checking build/O2/fullCI_slc9 for 9995505 at 2026-04-07 03:20:

++ cp /sw/slc9_x86-64/O2/slc9_x86-64-slc9_x86-64-local11/compile_commands.json .
+++ python3 -c 'import json, os; print(os.path.commonpath([x["file"] for x in json.loads(open("compile_commands.json").read()) if "sw/BUILD" not in x["file"] and "G__" not in x["file"] and x["file"].endswith(".cxx")]))'
++ O2_SRC=/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0
++ [[ -e /sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/CMakeLists.txt ]]

Full log here.