You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While working on #4665, I discovered that several tests start failing randomly when running with to many jobs. And I also remember having rare CI fails which just went away on the next push.
This is not a big issue, you can just run the parallel tests with less jobs. But these races might hit also somewhere else.
This can be reproduced sometimes by running on a slow PC or in a VM with only 2 CPU's mapped. When the CPU is kept busy with stress-ng --cpu ncpu, the tests fail also in single threaded mode.
touchy UI tests, 100% reproducible with ncpu=16:
tests/ui-smoke/touchy-fit
UI_SMOKE_FAIL: window-fit: no window matching ("touchy" "Touchy") found
touchy UI tests, somewhat reproducible, better with ncpu=128. Looks like loading touchy_postgui.hal has a race.
tests/ui-smoke/touchy-quit
UI_SMOKE_QUIT_FAIL: GUI (pid 206) showed no window within 60s
...
touchy_postgui.hal:7: Pin 'touchy_test.mpg' does not exist
tests/ui-smoke/touchy
UI_SMOKE_FAIL: MODE_AUTO did not hold stable across 6 attempts
...
touchy_postgui.hal:7: Pin 'touchy_test.mpg' does not exist
SET_TERM_COND / SET_LINE appearing, somewhat reproducible, better with ncpu=128:
Found the first source of flakiness. Looks like SET_TERM_COND termCond=2, tolerance=0.001 should normally be shown, except you manage to be fast enough to not process this command. The following patch would fix it.
Before I fix many tests which have this part copy-paste: Is this the right way to wait for linuxcnc to be up and running? There are 57 tests having test-ui.py and at least 4 have this issue.
diff --git a/tests/motion-logger/basic/expected.builtin-startup.in b/tests/motion-logger/basic/expected.builtin-startup.in
index 2296018b31..fce76f310a 100644
--- a/tests/motion-logger/basic/expected.builtin-startup.in+++ b/tests/motion-logger/basic/expected.builtin-startup.in@@ -176,6 +176,7 @@ SPINDLE_OFF
DISABLE
JOINT_UNHOME joint=-2
FREE
+SET_TERM_COND termCond=2, tolerance=0.001
JOG_ABORT joint=0
JOG_ABORT joint=1
JOG_ABORT joint=2
diff --git a/tests/motion-logger/basic/test-ui.py b/tests/motion-logger/basic/test-ui.py
index 21da4c2bd4..8d85df15bc 100755
--- a/tests/motion-logger/basic/test-ui.py+++ b/tests/motion-logger/basic/test-ui.py@@ -45,6 +45,15 @@ c = linuxcnc.command()
s = linuxcnc.stat()
e = linuxcnc.error_channel()
+#+# Wait for interpreter to be ready+#++while True:+ s.poll()+ time.sleep(0.1)+ if s.interp_state == linuxcnc.INTERP_IDLE:+ break
#
# Come out of E-stop, turn the machine on, and switch to Auto mode.
An other sometimes failing tests, does not reproduce well except sometimes with parallel tests:
tests/ui-smoke/gmoccapy
...
WARN: homing did not complete on attempt 1, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 1, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 2, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 3, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 4, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 5, retrying
WARN: STATE_ESTOP_RESET not reached on attempt 6, retrying
UI_SMOKE_FAIL: STATE_ESTOP_RESET did not hold stable across 6 attempts
...
4. > This is not a big issue, you can just run the parallel tests with less jobs. But these races might hit also somewhere else.
All these are races that need fixing. Either the base code has a race or the test has a race. Both are equally bad.
Agreed. Now with the parallel tests, you can easily detect them without running tests for weeks... :-)
Meanwhile, I found another one:
tests/linuxcncrsh
--- expected-telnet-output 2026-10-11 14:14:10.328086275 +0200
+++ telnet-output 2026-10-12 20:24:54.139307950 +0200
@@ -136,10 +136,10 @@
MODE MDI
ERROR OK
SET PROBE ACK
-ERROR G38.2 move finished without making contact.
+ERROR OK
PROBE_TRIPPED NO
SET PROBE_CLEAR ACK
-ERROR OK
+ERROR G38.2 move finished without making contact.
MODE MDI
SET MODE ACK
MODE MANUAL
I can now run 32 full test runs in 7 min, 64x faster than before.
tests/linuxcncrsh fails quite often, log in last comment, it is not always the same
tests/multiclick fails from time to time, log below
Otherwise, it looks quite stable with tests, task: fix load-dependent test failures #4674, only a few sporadic failures. But this is to be expected if up to 256 linuxcnc instances run on a 16 core cpu. I've seen up to 2000 tasks running. Once in a while, a timeout will hit.
There 25 failures. Probably some due to just overload and timeouts but there might be other bugs hidden. You might find something worth investigating, so the report below.
Most failures are:
ui-smoke/gmoccapy 10x
ui-smoke/qtdragon-quit 5x
And then there are rare occurrences of:
interp/g28.2/home-all-sequence 2x
ui-smoke/gmoccapy-quit 2x
linuxcncrsh 1x
threads.0 1x
kins-switch-unsolved 1x
raster 1x
tests/m62-m68-range 1x
halmodule/stream 1x
Disabling copy back to test directory, it could have an influence when you are executing parallel runtests in parallel. But it looks like it did not change much. More statistics would be needed.
Each 128 runs. tests_takset_nocopy.txt tests_all_cpu_nocopy.txt
qtdragon-quit, gmoccapy-quit and the no-window failures look like fixed timeouts under that load. kins-switch-unsolved, threads.0 and raster I haven't looked at yet. For linuxcncrsh I'd need the telnet-output diff, which isn't in the report.
qtdragon-quit: both failures are qtvcp dying at its first PyQt import ("the sip module has already registered a module called PyQt5.QtCore"), before any LinuxCNC code runs. Which PyQt5, sip and qtpy versions do you have?
threads.0 has the "Connection reset by peer" on unload that #4630 fixed, merged after your run.
Starts getting complicated with that many branches... ;-)
So I reset the cherry-picks, rebased to master and merged in this order: #4674 #4666 #4680 #4672 #4684
There was one conflict but easy to solve.
@hdiethelm two questions on the rest of your earlier list:
Results will follow. I had this once in 544 test runs. So it might appear or not. I would have to run 2504 tests rounds = ~10h to have 99% certainty it is fixed. I will keep shorter test rounds for now but why not running over night once most issues are solved and merged.
qtdragon-quit: both failures are qtvcp dying at its first PyQt import ("the sip module has already registered a module called PyQt5.QtCore"), before any LinuxCNC code runs. Which PyQt5, sip and qtpy versions do you have?
I have debian trixie stock:
ii python3-pyqt5 5.15.11+dfsg-2 amd64 Python 3 bindings for Qt5
ii python3-pyqt5.sip 12.17.0-1+b1 amd64 runtime module for Python extensions using SIP
ii python3-qtpy 2.4.3-1 all Abstraction layer for PySide2/PySide6/PyQt5/PyQt6 (Python 3)
threads.0 has the "Connection reset by peer" on unload that #4630 fixed, merged after your run.
This should be included in the next test round by rebasing to master.
abort/stop-button-crazy-move:
There is now a new error: Homing not possible until current homing process is finished.
ui-smoke/qtdragon-quit
I see: USRMOT: ERROR: command 32 timeout (seq: 1) before RuntimeError: the sip module has already registered a module called PyQt5.QtCore, looks like the UI crashes due to a command timeout.
stale-motion-status/slow-controller: can't read motion status, just increasing the timeout was not fully doing it:
diff --git a/src/emc/task/emctaskmain.cc b/src/emc/task/emctaskmain.cc
index 72afc74b7b..bdae35b94e 100644
--- a/src/emc/task/emctaskmain.cc+++ b/src/emc/task/emctaskmain.cc@@ -3177,7 +3177,7 @@ static int emctask_startup()
std::chrono::seconds end;
int good;
-constexpr auto RETRY_TIME = 10s; // wait for subsystems to come up+constexpr auto RETRY_TIME = 60s; // wait for subsystems to come up
constexpr auto RETRY_INTERVAL = 1s; // between wait tries for a subsystem
// moved up so it can be exposed in taskmodule at init time
@@ -3321,6 +3321,7 @@ constexpr auto RETRY_TIME = 10s; // wait for subsystems to come up
exit(1);
}
} while (end > 0s);
+ log_error("emctaktest: Time: {}\n", std::chrono::duration<float>(end).count());
if (!good) {
log_error("can't read motion status\n");
return -1;
So I used the -o feature to gather all results and looks like some times is 60s is not enough while normally it is just 0s. Probably I am overdoing it a bit with the load, I think there is no need to increase the timeout here. But I leave the load for the moment, it shows some false positive but obviously, there where also real bugs.
I need to redo the test with a way higher timeout tomorrow, there could be also a real race stalling it fully.
Hmm, something bothered me and looking in the kernel messages, I found something:
[13031.599101] rtapi_app:T#0[257933]: segfault at 7f3d9ca811f5 ip 00007f3d9ca811f5 sp 00007f3d9b70ce00 error 14 likely on CPU 15 (core 0, socket 15)
[13031.599125] Code: Unable to access opcode bytes at 0x7f3d9ca811cb.
Now the issue is, that rtapi_app just restarts automatically and the test fails due to a funny reason or even passes. Also changing to exit(1) does not help. But there is a message:
fprintf(stderr, "WARNING: Deprecated: No active realtime environment found. Use \"realtime start\" to start one.\n"
By grepping for Deprecated: No active realtime environment found. it the output, crashes can be found:
testout/8/probe-queue-not-empty/stderr:WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
testout/8/probe-queue-not-empty/stderr:WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
testout/10/stale-motion-status/slow-controller/stderr:WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
testout/10/stale-motion-status/slow-controller/stderr:WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
testout/10/stale-motion-status/slow-controller/stderr:WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
Backtrace shows:
#0 0x00007f3a856941f5 in ?? ()
#1 0x00000d3e533f8cda in ?? ()
#2 0x00007f3a8586a5cc in thread_task (arg=0x7f3a849fd9e0) at hal/hal_lib.c:3318
#3 0x00007f3a85862570 in (anonymous namespace)::PosixApp::wrapper (arg=0x555db2004360) at rtapi/uspace_posix.cc:124
#4 0x00007f3a8529e8bb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f3a8531c538 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
Only a halrun / linuxcnc crash. After an rtapi_app crash, halrun will cleanup shm using realtime stop.
I have to figure out a way to fail a tests if rtapi_app crashed. Right now, it looks like there is no real error.
You are right, 0x00000064 (Emc motion key) is not removed in the realtime script. It removes only the other shm keys. However, the way it crashes in the case I am debugging just now, this key is removed due to rtapi_app crashed during unload. This is why I have more segfaults than SHMERR in the logs.
Based on the backtrace and the log, I have the suspicion that the thread function is called after the component has already been unloaded or is still running while being unloaded. Both tests segfaulting do a delay inside the component.
probe-queue-not-empty/stderr:
+ set -e
+ halcompile --install servo_delay.comp
+ linuxcnc -r test.ini
Note: realtime scheduling unavailable (sched_setscheduler SCHED_FIFO: Operation not permitted).
Process capabilities: cap_sys_nice=no cap_ipc_lock=no.
Falling back to POSIX non-realtime.
Fix: 'sudo make setcap' (preferred) or 'sudo make setuid' on rtapi_app.
Override (testing only): set LINUXCNC_FORCE_REALTIME=1.
Note: Using POSIX non-realtime
link (updating variable file): No such file or directory
probes done: 100
rtapi_app: recv_result 1 failed: Connection reset by peer
rtapi_app: failed to read from master
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
WARNING: Deprecated: No active realtime environment found. Use "realtime start" to start one.
A realtime environment is started automatically.
If this appears while using halcmd: Use halrun instead.
halcmd should only be used with an already running realtime environment.
halrun creates a realtime environment and tears it down at exit.
Note: realtime scheduling unavailable (sched_setscheduler SCHED_FIFO: Operation not permitted).
Process capabilities: cap_sys_nice=no cap_ipc_lock=no.
Falling back to POSIX non-realtime.
Fix: 'sudo make setcap' (preferred) or 'sudo make setuid' on rtapi_app.
Override (testing only): set LINUXCNC_FORCE_REALTIME=1.
Note: Using POSIX non-realtime
hypot: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
ddt: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
motmod: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
trivkins: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
homemod: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
tpmod: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
Indeed, the following patch fixes all segfaults, proving the theory. Doing it this way is a bad idea thought, works only with one thread and is not 100% race save. #4630 fixes most of it but left one race condition behind: hal_data->threads_running == 0 does not guarantee that the thread is not still processing a function call, so it can race with unloading the function while it is still running.
diff --git a/src/hal/hal_lib.c b/src/hal/hal_lib.c
index 227cf6ad62..56873c2baf 100644
--- a/src/hal/hal_lib.c+++ b/src/hal/hal_lib.c@@ -2776,6 +2776,9 @@ int hal_stop_threads(void)
}
hal_data->threads_running = 0;
+ while(hal_data->threads_running_ack){+ rtapi_delay(1000000);+ }
rtapi_print_msg(RTAPI_MSG_DBG, "HAL: threads stopped\n");
return 0;
}
@@ -3275,6 +3278,7 @@ static void thread_task(void *arg)
thread = arg;
while (1) {
+ hal_data->threads_running_ack = hal_data->threads_running;
if (hal_data->threads_running > 0 && !thread->init_done) {
/* special init cycle: run init_funct_list once with no
timing measurement, re-anchor the period so the long
diff --git a/src/hal/hal_priv.h b/src/hal/hal_priv.h
index 8692aae946..a487b9021e 100644
--- a/src/hal/hal_priv.h+++ b/src/hal/hal_priv.h@@ -286,7 +286,8 @@ typedef struct hal_data_t {
SHMFIELD(hal_funct_t) funct_list_ptr; // root of linked list of functions
SHMFIELD(hal_thread_t) thread_list_ptr; // root of linked list of threads
long base_period; // timer period for realtime tasks
- int threads_running; // non-zero if threads are started+ int threads_running; // non-zero if threads are started (set and reset from externally)+ int threads_running_ack; // non-zero if threads are running (set by and reset by thread)
SHMFIELD(hal_oldname_t) oldname_free_ptr; // list of free oldname structs
SHMFIELD(hal_comp_t) comp_free_ptr; // list of free component structs
SHMFIELD(hal_pin_t) pin_free_ptr; // list of free pin structs
Activity
hdiethelm commentedon Oct 11, 2026
Found the first source of flakiness. Looks like
SET_TERM_COND termCond=2, tolerance=0.001should normally be shown, except you manage to be fast enough to not process this command. The following patch would fix it.Before I fix many tests which have this part copy-paste: Is this the right way to wait for linuxcnc to be up and running? There are 57 tests having
test-ui.pyand at least 4 have this issue.hdiethelm commentedon Oct 11, 2026
An other sometimes failing tests, does not reproduce well except sometimes with parallel tests:
BsAtHome commentedon Oct 12, 2026
All these are races that need fixing. Either the base code has a race or the test has a race. Both are equally bad.
hdiethelm commentedon Oct 12, 2026
Agreed. Now with the parallel tests, you can easily detect them without running tests for weeks... :-)
Meanwhile, I found another one:
hdiethelm commentedon Oct 12, 2026
With the following commands together with #4648:
or
I can now run 32 full test runs in 7 min, 64x faster than before.
tests/hal-show: mask the id together with its padding
task: report a probe move done only once motion has judged it
tests/multiclick, tests/mux: load the whole input before starting
hal: a stream of depth N holds N samples
hdiethelm commentedon Oct 13, 2026
After cherry-picking all from #4674 and #4663 together, I run 544 tests runs, massive parallel with the commands above. The branch is here: https://github.com/hdiethelm/linuxcnc-fork/tree/tests_parallel_v2_combined_tests
There 25 failures. Probably some due to just overload and timeouts but there might be other bugs hidden. You might find something worth investigating, so the report below.
Most failures are:
ui-smoke/gmoccapy 10x
ui-smoke/qtdragon-quit 5x
And then there are rare occurrences of:
interp/g28.2/home-all-sequence 2x
ui-smoke/gmoccapy-quit 2x
linuxcncrsh 1x
threads.0 1x
kins-switch-unsolved 1x
raster 1x
tests/m62-m68-range 1x
halmodule/stream 1x
Here just the full reports:
With taskset:
32x
tests_takset_1.txt
128x
tests_takset_2.txt
With all CPU's:
128x
tests_all_cpu_1.txt
Disabling copy back to test directory, it could have an influence when you are executing parallel runtests in parallel. But it looks like it did not change much. More statistics would be needed.
Each 128 runs.
tests_takset_nocopy.txt
tests_all_cpu_nocopy.txt
tests/halmodule/stream: print the result before signalling ready
tests/ui-smoke: let a slow homing sequence finish instead of homing a…
grandixximo commentedon Oct 14, 2026
@hdiethelm thanks for the long runs. From your report:
touchy sim: wait for the test panel before the postgui file
tests/ui-smoke: measure the touchy window only once it is mapped
19 remaining items
grandixximo commentedon Oct 15, 2026
@hdiethelm two questions on the rest of your earlier list:
wait_complete()and its own idle check had passed. I could not reproduce that or explain it from the log. Does it come back with tests, task: fix load-dependent test failures #4674, task: count a G28.2 homing run that ends between two status reads #4680 and motion: fix the split read check on the shared status structs #4672 applied, and how often?threads.0 has the "Connection reset by peer" on unload that #4630 fixed, merged after your run.
hdiethelm commentedon Oct 15, 2026
Starts getting complicated with that many branches... ;-)
So I reset the cherry-picks, rebased to master and merged in this order:
#4674
#4666
#4680
#4672
#4684
There was one conflict but easy to solve.
Results will follow. I had this once in 544 test runs. So it might appear or not. I would have to run 2504 tests rounds = ~10h to have 99% certainty it is fixed. I will keep shorter test rounds for now but why not running over night once most issues are solved and merged.
I have debian trixie stock:
ii python3-pyqt5 5.15.11+dfsg-2 amd64 Python 3 bindings for Qt5
ii python3-pyqt5.sip 12.17.0-1+b1 amd64 runtime module for Python extensions using SIP
ii python3-qtpy 2.4.3-1 all Abstraction layer for PySide2/PySide6/PyQt5/PyQt6 (Python 3)
This should be included in the next test round by rebasing to master.
touchy sim: wait for the test panel before the postgui file
tests/ui-smoke: measure the touchy window only once it is mapped
tests/motion-logger: let task issue its startup queue before the UI acts
tests/hal-show: ignore the component id column
tests/ui-smoke: skip ESTOP_RESET when the homing retry finds the mach…
task: count a G28.2 homing run that ends between two status reads
raster: do not show a running raster as disabled mid-cycle
hdiethelm commentedon Oct 15, 2026
So, new reports:
abort/stop-button-crazy-move:
There is now a new error:
Homing not possible until current homing process is finished.ui-smoke/qtdragon-quit
I see:
USRMOT: ERROR: command 32 timeout (seq: 1)beforeRuntimeError: the sip module has already registered a module called PyQt5.QtCore, looks like the UI crashes due to a command timeout.stale-motion-status/slow-controller:
can't read motion status, just increasing the timeout was not fully doing it:So I used the
-ofeature to gather all results and looks like some times is 60s is not enough while normally it is just 0s. Probably I am overdoing it a bit with the load, I think there is no need to increase the timeout here. But I leave the load for the moment, it shows some false positive but obviously, there where also real bugs.I need to redo the test with a way higher timeout tomorrow, there could be also a real race stalling it fully.
Full log, few new errors popping up from time to time:
128x, no code change
tests_takset1.txt
128x, timeout 30s
tests_takset2.txt
128x, timeout 60s
tests_takset3.txt
hdiethelm commentedon Oct 15, 2026
Hmm, something bothered me and looking in the kernel messages, I found something:
Now the issue is, that rtapi_app just restarts automatically and the test fails due to a funny reason or even passes. Also changing to
exit(1)does not help. But there is a message:linuxcnc/src/rtapi/uspace_rtapi_main.cc
Line 1308 in b99baf8
By grepping for
Deprecated: No active realtime environment found.it the output, crashes can be found:Backtrace shows:
hal/hal_lib.c:3318->funct_entry->funct(funct_entry->arg, thread->period);This is probably the reason for the
stale-motion-statustimeout but theprobe-queue-not-emptytest just passes.BsAtHome commentedon Oct 15, 2026
This:
*** SHMERR: Shared memory segment 0x00000064 (Emc motion key) was not removed. Removing...indicates a crash.
hdiethelm commentedon Oct 15, 2026
Only a halrun / linuxcnc crash. After an rtapi_app crash, halrun will cleanup shm using realtime stop.
I have to figure out a way to fail a tests if rtapi_app crashed. Right now, it looks like there is no real error.
hdiethelm commentedon Oct 16, 2026
You are right,
0x00000064 (Emc motion key)is not removed in the realtime script. It removes only the other shm keys. However, the way it crashes in the case I am debugging just now, this key is removed due to rtapi_app crashed during unload. This is why I have more segfaults than SHMERR in the logs.Based on the backtrace and the log, I have the suspicion that the thread function is called after the component has already been unloaded or is still running while being unloaded. Both tests segfaulting do a delay inside the component.
probe-queue-not-empty/stderr:
hdiethelm commentedon Oct 16, 2026
Indeed, the following patch fixes all segfaults, proving the theory. Doing it this way is a bad idea thought, works only with one thread and is not 100% race save. #4630 fixes most of it but left one race condition behind:
hal_data->threads_running == 0does not guarantee that the thread is not still processing a function call, so it can race with unloading the function while it is still running.