You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Discussion: Real Time Cores (Preempt-RT, RTAI, Xenomai and others) #3878
After the long discussions in #3873 and #3875, I investigated the real time options a bit more in depth and created a summary.
LinuxCNC has quite a number of available real time cores. Some of them are broken, others work fine. What I found so far:
All userspace variants are configured by --with-realtime=uspace and built if the needed library is installed. During runtime, a variant is detected using the kernel name, version and other hints and selected automatically or also not if your kernel name doesn't match expectations.
Format: {Text shown at start of LinuxCNC}:{My short description}
User Space: --with-realtime=uspace
POSIX non-realtime: Any old kernel or you forgot make setuid
Works, but big latency
Basically what starts when you don't have any other realtime or it is not properly detected
POSIX realtime: PREEMPT_RT
Works
Probably the most used option
Debian stock RT kernel works fine
XENOMAI (posix-skin) realtime: Xenomai2: Might be / Xenomai3: Works / Xenomai4: Not compatible
Works: I managed to get it running with Xenomai3
Probably seldom used due to no up to date compiled kernel package is available
LXRT realtime: RTAI Userspace
Seams to be broken after 2.8.4. I had no success, 2.8.4 is buster where I don't have a VM. In Bookwork/Trixie I was not able to compile 2.8.4. 2.9.* and master builds but has an error starting.
Xenomai4 has a new API: EVL. I did not found a posix skin. I managed to get the kernel and tools built for anybody interested in creating a Xenomai4 module for LinuxCNC: https://github.com/hdiethelm/xenomai4-linuxcnc
If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.
BTW: Is it fine starting this discussion here or should I move it to the forum?
mce (machine check) gave interrupts on the isolated CPU's. However, the latency's did not change a lot with this change, with or without load.
The 140us spikes seam to come from the GPU and watching youtube is not really needed for CNC work... ;-) So I used stress-ng.
Results with latency-histogram, load is 2x glx gears and stress-ng --cpu 2 --iomix 2 --vm 2 --vm-bytes 128M --fork 2:
PREEMPT_RT: Idle: 1.7us / Load: 2.9us
Xenomai3: Idle: 1.1us / Load: 12us
Xenomai4: Idle: 1.2us / Load: 1.9us
LXRT: Idle: 28us / Load: 24us
RTAI: Idle: 6.9us / Load: 13us
So at least for my PC, PREEMPT_RT and Xenomai4 are the best, followed by Xenomai3.
RTAI/LXRT perform way worse. The new isolcpus syntax is not recognized. As soon as I use isolcpus=2,3 the system gets unstable and crashes often on load. So I forced the thread to CPU 3, so there is at leas no interrupt noise.
Loading hm2_eth, needed for my machine, i use a ethernet mesa card:
PREEMPT_RT: Works out of the box, latency ~25us
Xenomai3/Xenomai4: Works out of the box
However, for RTnet, additional work and an e1000e ethernet card is required
Without RTnet, the normal kernel processes the ethernet workload but the latency is still fine (~50us)
In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.
If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.
I had to try it anyway... ;-) It doesn't look that bad. configure, automake and that WithRoot() was the biggest issue so far. Now I have proper EVL threads:
linuxcnc:
Note: Using XENOMAI4 EVL realtime
linuxcnc-thread:88560
linuxcnc-thread:88561
evl ps -l
CPU PID SCHED PRIO ISW CTXSW SYS RWA STAT TIMEOUT %CPU CPUTIME WCHAN NAME
15 88560 fifo 98 1 1 1 0 X - 0.0 0:000.003 (null) linuxcnc-thread:88560
15 88561 fifo 97 1 1 1 0 X - 0.0 0:000.001 (null) linuxcnc-thread:88561
The harder part is finding anything which uses functions not suited for real time and port that to EVL. And I wont think about HW drivers so far... ;-)
In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.
RTAI and Ethernet is probably not impossible, it just wasn't implemented.
Xenomai? Who knows.
LXRT RTAI should work with Ethernet too, and does have slightly better latency on my test system than preempt-rt.
(But my test system is a freak, less than 1k latency on preempt-rt and on RTAI so who knows what the typical LXRT/RTAI latency is?)
@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master? For me it fails. Any tricks?
In Xenomai3, network runs out of the box, however, probably not as good as it could. My machine has a Ethernet Mesa card and just works with 2.9.8 / Xenomai3 kernel. However, I did not yet do extensive testing.
Xenomai3 and Xenomai4 are different and bot a bit special concerning syscalls / device drivers / network.
Both: As soon as you call a non-Xenomai part, it switches from the Xenomai to the normal kernel, does the work and switches back. Due to you can have a PREEMT_RT kernel with Xenomai, this should still be fast but not optimal. There is a way of tracing this, I have to look into it.
You should be able to use Xenomai 4 in a kernel thread and not worry about ethernet drivers, similar as to how it's used with RTAI and a PCI(e) parallel port card.
Is there any reason to run in kernel space? Xenomai 3 works well also in user space and I will start test with Xenomai 4 in a few days. So now I think I should have all configs running including RTAI, so I can compare all of them on my CNC PC in the exact same configuration except different kernels.
All information I found was pretty outdated and most just recommend to test your app. I did not find any reference yet that Xenomai3/4 still support kernel mode.
The whole point of the co-kernel approach, whether it's IPIPE or dovetail is that you isolate tasks at kernel level. People talk about isolcpus being all great and wonderful but on most systems I've seen it doesn't help latency because the spikes come from hardware interrupts, most frequently APIC, even when paired with nohz_full and rcu_nocbs, and in some cases I've actually seen latency higher than if the kernel parameters are left alone. isolcpus=cores herenohz_full=cores herercu_nocbs=cores here don't help with hardware interrupts if they're happening on the isolated core and there's really no way around that once you start digging into the backtraces of using rtla timerlat top and you can actually see where the spikes come from on PREEMPT_RT, instead of just looking at numbers.
IPIPE (RTAI's backbone) only goes up to 5.4 but there is a currently maintained kernel repository that's based on 4.19 which is why I'm asking, should we start looking into using the 4.19 cip branch since it's actively maintained, or is an older, completely EOL 5.4 kernel better in your opinion?
If implemented correctly, doing RT in kernel space using a dual kernel will always be superior, whether it be Xenomai 4 or RTAI. I personally would never use uspace with either of those though, because if I want uspace, I'd just use plain ole PREEMPT_RT.
I have RTAI for linux-cip-4.19.325-cip130 now. (just locally atm)
There may be additional issues. I discovered possible races between RT/non-RT while looking into the latency-plot/latency-histogram code. The data pulled out of RT (components timedelta and histobins) is not latched and values read may not be of one and the same cycle.
I've been working on a python replacement, but am still battling matplotlib/pyplot. However, my new components where data is latched may possibly be backported.
Additionally, using glxgears on modern machines may no longer have a real effect because drawing is offloaded almost entirely to the GPU on modern machines.
@NTULINUX As much as I understand it and what I read so far:
RTAI/LXRT/Xenomai: They have a co-kernel and an interrupt pipeline (IPIPE / Dovetail) to handle interrupts and forward them to the linux kernel.
If you are running in kernelspace, you have a bit less latency due to no user/kernelspace switch is needed.
If you are running in userspace, the co-kernel handels the thread, so you don't have any interference from interrupts.
PREEMPT_RT: The main disadvantage is that the kernel sometimes disables interrupts, so your thread won't get executed until they are enabled again. This is mitigated by the above interrupt pipeline.
Anyway, the 140us spikes on my PC seam to have an other reason. Tested with PREEMPT_RT Debian kernel: sudo cyclictest --mlockall -t 4 -a 0-3 --priority=80 --interval=50 --distance=0 --smi shows no SMI interrupts sudo rtla timerlat top -a 50 -Pf:98 -c 2-3 shows:
## CPU 2 hit stop tracing, analyzing it ##
IRQ handler delay: 0.46 us (0.34 %)
IRQ latency: 135.75 us
Blocking thread:
swapper/2:0
Blocking thread stack trace
-> timerlat_irq
-> __hrtimer_run_queues
-> hrtimer_interrupt
-> __sysvec_apic_timer_interrupt
-> sysvec_apic_timer_interrupt
-> asm_sysvec_apic_timer_interrupt
-> cpuidle_enter_state
-> cpuidle_enter
-> do_idle
-> cpu_startup_entry
-> start_secondary
-> common_startup_64
------------------------------------------------------------------------
IRQ latency: 135.75 us (100%)
Yes:
^Note, that is in a VM, so really bad timing. But it works the same on my CNC PC.
One thing in the doc is already outdated: Xenomai4 initial support got merged to master meanwhile: #3903 I will update that.
I always run the xenomai*-build.sh scripts line by line manually, I have to test if these scripts run also fully without intervention and don't mess anything up.
@NTULINUX I updated https://github.com/hdiethelm/xenomai4-linuxcnc/blob/lfs/README.md also with plain install that should work for non-debian os. That probably should work for you. I don't know how wide spread it is to install the headers/libs in non-standard paths but if this is often the case, it would probably make sense to add a --with-evl= option to configure.ac. I am new to automake, so still learning...
Thanks, I'll have to try this on hardware at some point! I can help with autotools btw. I'm no pro but I can get a lot done :D Re-doing the ENTIRE RTAI build system taught me a lot. Sweet work!
I discovered possible races between RT/non-RT while looking into the latency-plot/latency-histogram code. The data pulled out of RT (components timedelta and histobins) is not latched and values read may not be of one and the same cycle.
There is a new module hm2_eth_evl which uses out of band xenomai4 networking but is otherwise equal to hm2_eth.
I use: isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off evl.oobcpus=3
as the kernel command line. Having CPU 0/1 for the user, 2/3 isolated.
evl.oobcpus=3 should be set to the CPU linuxcnc normally runs on, so the network card tasks run on the same CPU. Otherwise, the CPU-CPU latency will create al lot of jitter.
Do not use nohz=on nohz_full=2-3, this will stall Xenomai4 threads. Why is still being investigated.
To test timings directly with my mill, i use:
loadrt timedelta count=2
addf timedelta.0 servo-thread #Start jitter
#all other functions
addf timedelta.1 servo-thread #End jitter, contains all ethernet communication
Results, not touching the mouse:
PREEMT_RT: Start 1.2 End 7.6 us
Xenomai4: Start 1.0 End 6.3 us
Results with re scaling the window and so on:
PREEMT_RT: Start 4.8 End 20 us
Xenomai4: Start 2.8 End 18 us
servo_thread.time both ~220 clocks = 70 ns
So not that much better but it may be a lot different on other PC's. Mine behaves quite well except as soon as the GPU has something to do, I get spikes, independent of the real time type.
With #4199 #4240
merged to master, there is now full Xenomai4 + Mesa card support with real-time Ethernet.
To activate this:
Add initf hm2_eth.realtime-init servo-thread to the hal file and add board_rtnet=evl to the hm2_eth config. It should look something like:
It looks like it depends on the hardware if you have more or less benefits from using Xenomai4. @zz912 has seen way less Ethernet read time hm2_7i96s.0.read.time. On my setup, the difference is less but still noticeable, so I will stay with Xenomai4 for the future.
Depending on the user's taste, there are now github releases and scripts to download and install the packages but also all tools that I use to build the packages from source.
I think there is one important piece missing here: the EtherCAT side.
LinuxCNC itself may be relatively straightforward to port from the Xenomai 3 POSIX skin to Xenomai 4/EVL, but if the goal is to use LinuxCNC with the IgH EtherCAT Master, IgH currently does not support Xenomai 4.
The IgH maintainers explicitly state there that adding EVL support is currently not planned on their side. The existing Xenomai integration relies on RTDM, whereas Xenomai 4 uses the EVL API instead.
So I would consider Xenomai 4 support in LinuxCNC only one part of the problem. For a LinuxCNC + IgH + Xenomai 4 setup, we would also need either an IgH EVL port or a different EtherCAT master/integration layer.
It would therefore be useful to clarify whether the proposed Xenomai 4 work is intended only for the LinuxCNC realtime/userspace layer, or also for an EtherCAT setup using IgH.
Does ethercat suport Xenomai3? I have also a quick&dirty Xenomai3 hm2_eth variant on a branch. If this is of use, it should not be to much effort to finalize it, now that hm2_eth backend switching is in place.
On the otherhand, I could also create a EVL backend for ethercat. Hovever, I have no hardware to test (yet).
Due to LinuxCNC is a hobby, all I do is in my free time. I don't really plan and I can only invest a limited amount of time.
I understand. I became more interested in EtherCAT and Xenomai 4 mainly because I have also invested quite a lot of time in testing your project, and I am curious to see where it can go from here.
Personally, though, I don't see much reason to go back to Xenomai 3 and maintain two different Xenomai versions. At the same time, I don't want to convince you to develop an EVL backend for EtherCAT when I don't actually need it myself. I'm mostly just curious to see which direction you choose to take. I think you should mainly do whatever you enjoy working on.
If I were in your position, I would probably wait until you have some EtherCAT hardware and then decide what to do next. But if you decide that EtherCAT and Xenomai 4 are something you really want to explore, I wouldn't hesitate to invest in some inexpensive EtherCAT hardware. Some EtherCAT modules are really quite affordable, and compared to your time, I think the cost of the EtherCAT hardware is almost negligible.
And if you find some inexpensive hardware that makes sense for testing, I would be happy to buy one as well. Having two units could actually be quite useful for development and testing.
My motivation for being a tester isn't just that I need it to work for me. I enjoy being part of this nice LinuxCNC community and having the opportunity to contribute to the development by actually testing things in practice.
One thing I was thinking about regarding what you could do next: I think it would be really useful to add an EVL mode checkbox to PNCconf.
Under the checkbox, you could add a link to your GitHub repository with the EVL installation instructions. From the user's perspective, this would make your work much more visible and much easier to discover and try.
I think it should be relatively little work at this point, but it could make a big difference in getting more people to use and test what you have developed. Instead of EVL support being something that people have to know about and find manually, PNCconf would basically point them in the right direction.
Just my thought — I think this might actually be a better next step than immediately starting another backend. It would give your current work a much better chance to reach more LinuxCNC users and, in turn, perhaps bring you more testers and feedback.
Get some more experience with Xenomai4 and iron out some issues with deployment
Add basic Xenomai3 support for hm2_eth but it will be less comfortable than with Xenomai4 due to Xenomai3 needs kernel modules and other configuration that can not be easily integrated into linuxcnc. However, it might help people with Ethercat and basic support already works.
I will look into PNCconf but this is a Python GUI where I don't have that much experience
Update the doc on realtime
Improve the realtime configuration and testing process
Might add Xenomai4 support for Raspberry's. As much as I read, there is a way but I have to test how well this works and see how much effort it is to get RT-Network running.
Might be add Xenomai4 support for Ethercat. Could be an interesting project.
Some time ago, I started a crash avoidance algorithm that is after the path planner in linuxcnc which estimates when an emergency stop is needed before a crash is imminent. The actual path planner just continues until it is to late to stop in specific cases.
Activity
hdiethelm commentedon Mar 25, 2026
Benchmarks so far on my CNC PC:
Hardware:
Results 1:
Kernel command line:
isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1Results with latency-histogram, load is 2x glx gears and watching video on youtube with sound:
Caveats:
Results 2:
Kernel command line:
isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=offmce (machine check) gave interrupts on the isolated CPU's. However, the latency's did not change a lot with this change, with or without load.
The 140us spikes seam to come from the GPU and watching youtube is not really needed for CNC work... ;-) So I used stress-ng.
Results with latency-histogram, load is 2x glx gears and
stress-ng --cpu 2 --iomix 2 --vm 2 --vm-bytes 128M --fork 2:So at least for my PC, PREEMPT_RT and Xenomai4 are the best, followed by Xenomai3.
RTAI/LXRT perform way worse. The new isolcpus syntax is not recognized. As soon as I use isolcpus=2,3 the system gets unstable and crashes often on load. So I forced the thread to CPU 3, so there is at leas no interrupt noise.
Loading hm2_eth, needed for my machine, i use a ethernet mesa card:
rene-dev commentedon Mar 25, 2026
In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.
NTULINUX commentedon Mar 25, 2026
Sweet! Good luck!
hdiethelm commentedon Mar 25, 2026
I had to try it anyway... ;-) It doesn't look that bad. configure, automake and that WithRoot() was the biggest issue so far. Now I have proper EVL threads:
linuxcnc:
evl ps -l
The harder part is finding anything which uses functions not suited for real time and port that to EVL. And I wont think about HW drivers so far... ;-)
andypugh commentedon Mar 26, 2026
RTAI and Ethernet is probably not impossible, it just wasn't implemented.
Xenomai? Who knows.
LXRT RTAI should work with Ethernet too, and does have slightly better latency on my test system than preempt-rt.
(But my test system is a freak, less than 1k latency on preempt-rt and on RTAI so who knows what the typical LXRT/RTAI latency is?)
hdiethelm commentedon Mar 26, 2026
@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master? For me it fails. Any tricks?
In Xenomai3, network runs out of the box, however, probably not as good as it could. My machine has a Ethernet Mesa card and just works with 2.9.8 / Xenomai3 kernel. However, I did not yet do extensive testing.
Xenomai3 and Xenomai4 are different and bot a bit special concerning syscalls / device drivers / network.
Both: As soon as you call a non-Xenomai part, it switches from the Xenomai to the normal kernel, does the work and switches back. Due to you can have a PREEMT_RT kernel with Xenomai, this should still be fast but not optimal. There is a way of tracing this, I have to look into it.
Xenomai3: It has a set of ported network drivers: https://v3.xenomai.org/rtnet/config/index.html
Xenomai4: A driver has to be "enabled" to work, what ever this is: https://v4.xenomai.org/core/net/index.html#evl-net-port
This presentation explains it: https://www.youtube.com/watch?v=WXKsLTfrOiE 23:40 for Xenomai3 / 27:33 for Xenomai4
hdiethelm commentedon Mar 26, 2026
Xenomai3 network drivers: https://gitlab.com/Xenomai/xenomai3/xenomai/-/tree/master/kernel/drivers/net/drivers?ref_type=heads
Xenomai4: https://v4.xenomai.org/core/drivers/net/index.html :-D Looking in Git and grep for NET_OOB probably only Intel e1000e and 82575/82576: https://gitlab.com/Xenomai/xenomai4/linux-evl/-/blob/v6.12.y-cip-evl-rebase/drivers/net/ethernet/intel/Kconfig?ref_type=heads
It seams that Xenomai4 is still under heavy construction.
andypugh commentedon Mar 26, 2026
No, I haven't tried LXRT RTAI since 2022.
However LXRT RTAI has run at least once on 2.9:
https://sourceforge.net/p/emc/mailman/message/37718519/
(note the link to the .png)
hdiethelm commentedon Mar 26, 2026
I found it: #3879 Seams no one used this for a pretty long time...
NTULINUX commentedon Mar 30, 2026
You should be able to use Xenomai 4 in a kernel thread and not worry about ethernet drivers, similar as to how it's used with RTAI and a PCI(e) parallel port card.
I found a 5.4 tree that is still active!
https://github.com/oracle/linux-uek/tree/v5.4.17-2136.354.4
edit: bit off more than I could chew trying to build that..
What if we went back to 4.19 but use the
cipbranch that is currently maintained?https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/log/?h=linux-4.19.y-cip
hdiethelm commentedon Mar 31, 2026
Is there any reason to run in kernel space? Xenomai 3 works well also in user space and I will start test with Xenomai 4 in a few days. So now I think I should have all configs running including RTAI, so I can compare all of them on my CNC PC in the exact same configuration except different kernels.
All information I found was pretty outdated and most just recommend to test your app. I did not find any reference yet that Xenomai3/4 still support kernel mode.
Does it really make sense to setup a new system based on that old kernels? I did my test on 6.12.67 both on
Xenomai3: https://gitlab.com/Xenomai/linux-dovetail/-/tree/v6.12.67-dovetail1?ref_type=tags
Xenomai4: https://gitlab.com/Xenomai/xenomai4/linux-evl/-/tree/v6.12.67-evl2-rebase?ref_type=tags
NTULINUX commentedon Mar 31, 2026
The whole point of the co-kernel approach, whether it's IPIPE or dovetail is that you isolate tasks at kernel level. People talk about isolcpus being all great and wonderful but on most systems I've seen it doesn't help latency because the spikes come from hardware interrupts, most frequently APIC, even when paired with nohz_full and rcu_nocbs, and in some cases I've actually seen latency higher than if the kernel parameters are left alone.
isolcpus=cores herenohz_full=cores herercu_nocbs=cores heredon't help with hardware interrupts if they're happening on the isolated core and there's really no way around that once you start digging into the backtraces of usingrtla timerlat topand you can actually see where the spikes come from on PREEMPT_RT, instead of just looking at numbers.IPIPE (RTAI's backbone) only goes up to 5.4 but there is a currently maintained kernel repository that's based on 4.19 which is why I'm asking, should we start looking into using the 4.19
cipbranch since it's actively maintained, or is an older, completely EOL 5.4 kernel better in your opinion?If implemented correctly, doing RT in kernel space using a dual kernel will always be superior, whether it be Xenomai 4 or RTAI. I personally would never use uspace with either of those though, because if I want uspace, I'd just use plain ole PREEMPT_RT.
I have RTAI for linux-cip-4.19.325-cip130 now. (just locally atm)
hdiethelm commentedon Apr 3, 2026
Finally, I got the first full testing round done. It took some patience to get all variants running. I tested with this branch where I back ported everything to 2.9.8: https://github.com/hdiethelm/linuxcnc-fork/tree/2.9.8-hdiethelm
The pure xenomai4 uspace code base master is here, but RTAI is not compiling: https://github.com/hdiethelm/linuxcnc-fork/tree/xenomai4
I updated this comment: #3878 (comment)
The results are not yet conclusive. I have to figure out what generates this ~140us spike.
Hacks needed:
sd_modto/etc/initramfs-tools/modulesBsAtHome commentedon Apr 3, 2026
There may be additional issues. I discovered possible races between RT/non-RT while looking into the latency-plot/latency-histogram code. The data pulled out of RT (components timedelta and histobins) is not latched and values read may not be of one and the same cycle.
I've been working on a python replacement, but am still battling matplotlib/pyplot. However, my new components where data is latched may possibly be backported.
Additionally, using glxgears on modern machines may no longer have a real effect because drawing is offloaded almost entirely to the GPU on modern machines.
hdiethelm commentedon Apr 3, 2026
@NTULINUX As much as I understand it and what I read so far:
RTAI/LXRT/Xenomai: They have a co-kernel and an interrupt pipeline (IPIPE / Dovetail) to handle interrupts and forward them to the linux kernel.
If you are running in kernelspace, you have a bit less latency due to no user/kernelspace switch is needed.
If you are running in userspace, the co-kernel handels the thread, so you don't have any interference from interrupts.
PREEMPT_RT: The main disadvantage is that the kernel sometimes disables interrupts, so your thread won't get executed until they are enabled again. This is mitigated by the above interrupt pipeline.
Then there are the SMI interrupts which break everything independent of the implementation. Xenomai3 has a workaround but at least on my PC, it fails: https://v3.xenomai.org/tips/x86/smi/index.html Xenomai4 says: https://v4.xenomai.org/core/caveat/
Anyway, the 140us spikes on my PC seam to have an other reason. Tested with PREEMPT_RT Debian kernel:
sudo cyclictest --mlockall -t 4 -a 0-3 --priority=80 --interval=50 --distance=0 --smishows no SMI interruptssudo rtla timerlat top -a 50 -Pf:98 -c 2-3shows:Any clue what this means?
15 remaining items
hdiethelm commentedon Apr 13, 2026
https://github.com/hdiethelm/xenomai3-linuxcnc/blob/lfs/README.md
https://github.com/hdiethelm/xenomai4-linuxcnc/blob/lfs/README.md
It's all early prototype but way better than starting from xenomai doc. It works on my system, debian trixie, but others I don't know.
PS: Xenomai4 gives a kernel warning about double registration but it schould be save to ignore. I know where it comes from but I have to report it upstream.
NTULINUX commentedon Apr 13, 2026
Does LinuxCNC's latency test work yet or are those docs still accurate after all your changes?
hdiethelm commentedon Apr 13, 2026
Yes:

^Note, that is in a VM, so really bad timing. But it works the same on my CNC PC.
One thing in the doc is already outdated: Xenomai4 initial support got merged to master meanwhile: #3903 I will update that.
I always run the xenomai*-build.sh scripts line by line manually, I have to test if these scripts run also fully without intervention and don't mess anything up.
hdiethelm commentedon Apr 13, 2026
@NTULINUX I updated https://github.com/hdiethelm/xenomai4-linuxcnc/blob/lfs/README.md also with plain install that should work for non-debian os. That probably should work for you. I don't know how wide spread it is to install the headers/libs in non-standard paths but if this is often the case, it would probably make sense to add a
--with-evl=option to configure.ac. I am new to automake, so still learning...NTULINUX commentedon Apr 13, 2026
Thanks, I'll have to try this on hardware at some point! I can help with autotools btw. I'm no pro but I can get a lot done :D Re-doing the ENTIRE RTAI build system taught me a lot. Sweet work!
grandixximo commentedon Apr 26, 2026
@BsAtHome did you confirm this?
BsAtHome commentedon Apr 26, 2026
see #3968
hdiethelm commentedon May 30, 2026
So, the first prototype with Xenomai4 out of band networking for mesa cards works nicely.
It took some time and a bit of bug fixing both on my side and also in Xenomai4 but the devs there helped a lot.
I run now an Intel I350 network card. You can get them for cheep from china.
LinuxCNC:
https://github.com/hdiethelm/linuxcnc-fork/tree/hm2_eth_oob_v4
You need to install the latest kernel / libevl from here:
https://github.com/hdiethelm/xenomai4-linuxcnc
There is a new module hm2_eth_evl which uses out of band xenomai4 networking but is otherwise equal to hm2_eth.
I use:
isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off evl.oobcpus=3as the kernel command line. Having CPU 0/1 for the user, 2/3 isolated.
evl.oobcpus=3should be set to the CPU linuxcnc normally runs on, so the network card tasks run on the same CPU. Otherwise, the CPU-CPU latency will create al lot of jitter.Do not use
nohz=on nohz_full=2-3, this will stall Xenomai4 threads. Why is still being investigated.To test timings directly with my mill, i use:
Results, not touching the mouse:
PREEMT_RT: Start 1.2 End 7.6 us
Xenomai4: Start 1.0 End 6.3 us
Results with re scaling the window and so on:
PREEMT_RT: Start 4.8 End 20 us
Xenomai4: Start 2.8 End 18 us
servo_thread.time both ~220 clocks = 70 ns
So not that much better but it may be a lot different on other PC's. Mine behaves quite well except as soon as the GPU has something to do, I get spikes, independent of the real time type.
Who wants to test it?
hdiethelm commentedon Aug 25, 2026
With
#4199
#4240
merged to master, there is now full Xenomai4 + Mesa card support with real-time Ethernet.
To activate this:
Add
initf hm2_eth.realtime-init servo-threadto the hal file and addboard_rtnet=evlto the hm2_eth config. It should look something like:It looks like it depends on the hardware if you have more or less benefits from using Xenomai4. @zz912 has seen way less Ethernet read time
hm2_7i96s.0.read.time. On my setup, the difference is less but still noticeable, so I will stay with Xenomai4 for the future.Here: https://github.com/hdiethelm/xenomai4-linuxcnc I provide a matching kernel, userspace library and also a back-port for LinuxCNC 2.9.10.
Depending on the user's taste, there are now github releases and scripts to download and install the packages but also all tools that I use to build the packages from source.
pcw-mesa commentedon Aug 26, 2026
Thanks for all your work on this!
This ( LinuxCNC on Xenomai ) should
benefit EtherCAT as well.
zz912 commentedon Aug 26, 2026
I think there is one important piece missing here: the EtherCAT side.
LinuxCNC itself may be relatively straightforward to port from the Xenomai 3 POSIX skin to Xenomai 4/EVL, but if the goal is to use LinuxCNC with the IgH EtherCAT Master, IgH currently does not support Xenomai 4.
There is an upstream IgH issue specifically about adding Xenomai 4 (EVL) support:
https://gitlab.com/etherlab.org/ethercat/-/issues/97
The IgH maintainers explicitly state there that adding EVL support is currently not planned on their side. The existing Xenomai integration relies on RTDM, whereas Xenomai 4 uses the EVL API instead.
So I would consider Xenomai 4 support in LinuxCNC only one part of the problem. For a LinuxCNC + IgH + Xenomai 4 setup, we would also need either an IgH EVL port or a different EtherCAT master/integration layer.
It would therefore be useful to clarify whether the proposed Xenomai 4 work is intended only for the LinuxCNC realtime/userspace layer, or also for an EtherCAT setup using IgH.
hdiethelm commentedon Aug 27, 2026
Does ethercat suport Xenomai3? I have also a quick&dirty Xenomai3 hm2_eth variant on a branch. If this is of use, it should not be to much effort to finalize it, now that hm2_eth backend switching is in place.
On the otherhand, I could also create a EVL backend for ethercat. Hovever, I have no hardware to test (yet).
Due to LinuxCNC is a hobby, all I do is in my free time. I don't really plan and I can only invest a limited amount of time.
zz912 commentedon Aug 27, 2026
I understand. I became more interested in EtherCAT and Xenomai 4 mainly because I have also invested quite a lot of time in testing your project, and I am curious to see where it can go from here.
Personally, though, I don't see much reason to go back to Xenomai 3 and maintain two different Xenomai versions. At the same time, I don't want to convince you to develop an EVL backend for EtherCAT when I don't actually need it myself. I'm mostly just curious to see which direction you choose to take. I think you should mainly do whatever you enjoy working on.
If I were in your position, I would probably wait until you have some EtherCAT hardware and then decide what to do next. But if you decide that EtherCAT and Xenomai 4 are something you really want to explore, I wouldn't hesitate to invest in some inexpensive EtherCAT hardware. Some EtherCAT modules are really quite affordable, and compared to your time, I think the cost of the EtherCAT hardware is almost negligible.
And if you find some inexpensive hardware that makes sense for testing, I would be happy to buy one as well. Having two units could actually be quite useful for development and testing.
My motivation for being a tester isn't just that I need it to work for me. I enjoy being part of this nice LinuxCNC community and having the opportunity to contribute to the development by actually testing things in practice.
zz912 commentedon Aug 27, 2026
One thing I was thinking about regarding what you could do next: I think it would be really useful to add an EVL mode checkbox to PNCconf.
Under the checkbox, you could add a link to your GitHub repository with the EVL installation instructions. From the user's perspective, this would make your work much more visible and much easier to discover and try.
I think it should be relatively little work at this point, but it could make a big difference in getting more people to use and test what you have developed. Instead of EVL support being something that people have to know about and find manually, PNCconf would basically point them in the right direction.
Just my thought — I think this might actually be a better next step than immediately starting another backend. It would give your current work a much better chance to reach more LinuxCNC users and, in turn, perhaps bring you more testers and feedback.
hdiethelm commentedon Sep 5, 2026
My plans are roughly: