[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945
Draft
lyakh wants to merge 101 commits into
Draft
[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945lyakh wants to merge 101 commits into
lyakh wants to merge 101 commits into
Conversation
Replace the k_thread_cpu_mask_clear() + k_thread_cpu_mask_enable()
sequence with a single k_thread_cpu_pin() call.
The clear-then-enable pattern momentarily leaves the thread with a
zero CPU mask, which is invalid under CONFIG_SCHED_CPU_MASK_PIN_ONLY
and triggers an assertion after Zephyr commit 7798570a031d ("kernel:
sched: enforce non-zero CPU mask invariant in PIN_ONLY mode").
k_thread_cpu_pin() atomically sets exactly one bit in the mask,
avoiding the transient zero-mask state.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
The real fix is to remove the locking around dai_get_properties() altogether, but this depends on fixes in Zephyr DAI drivers. To unblock user-space work, remove the calls to spinlocks for now. This opens up possibility to hit issues with concurrent playback and capture cases on multiple cores, so this commit remains a WIP until fixes in Zephyr drivers land. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify code to allocate DAI properties object on stack and use dai_get_properties_copy(). This is required when DAI code is run in user-space and a syscall is needed to talk to the DAI driver. It's not possible to return a pointer to kernel memory, so instead data needs to be copied to caller stack. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Turn the pdata->sem into a dynamic object in userspace LL builds, implemented with Zephyr k_sem. Add POSIX no-op stubs for sys_sem to maintain testbench build compatibility. Keep statically allocated semaphore for kernel LL builds. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function scheduler_get_data_for_core() to look up scheduler data for a particular type of scheduler. This variant allows to pass the core number as an argument, so it can be called from unprivileged code. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function user_ll_grant_access() to allow other threads to access the scheduler mutex. This is needed if work is submitted from other threads to the scheduler. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
When LL scheduler is run in user-space, use a different Zephyr thread name. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The COHERENT_CHECK_NONSHARED_CORES debug macros call cpu_get_id() which invokes arch_proc_id() - a privileged hardware register read that faults in user-space context. Disable the entire debug block at compile time when CONFIG_SOF_USERSPACE_LL is enabled. This also fixes the same latent issue in CORE_CHECK_STRUCT and CORE_CHECK_STRUCT_INIT. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify the checks in zephyr_ll_assert_core() to make them safe to call from user-space LL threads. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Needs more review, but makes the tests pass again. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Build fails when building with CONFIG_THREAD_NAME disabled. Fix the issue by conditional compilation of code using CONFIG_THREAD_MAX_NAME_LEN. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add support to run pipeline_schedule_triggered() in user-space. Use the user_ll_lock/unlock_sched() interface if building with CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
In user-space LL builds the low-latency scheduler runs its work in a dedicated privileged domain thread, created together with its timer and access grants by scheduler_init_context() (zephyr_ll_init_context() -> domain_thread_init()). This context is per-core and must exist on every core that runs LL tasks. So far it was only established for the primary core, so LL tasks could not be scheduled on secondary cores when CONFIG_SOF_USERSPACE_LL is enabled. Allocate an LL task in secondary_core_init() and run scheduler_init_context() on it, giving each secondary core its own LL domain thread. A dedicated sec_core_init UUID is registered for the task. The whole block is compiled in only for CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Copier set_chmap() blocks IRQs to atomically update the converters. This code is not safe to be moved to user-space, so replace the locks with calls to block LL scheduler execution. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The module_adapter prepare() blocks IRQs to atomically to connect the sink/source buffers. This code is not safe to be moved to user-space, so replace the locks with calls to block LL scheduler execution, when compiled for user-space. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Make comp_drivers_get() usable from user-space when SOF is built with CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add CONFIG_SOF_USERSPACE_LL support to chain DMA component: - Allocate comp_dev, private data, and DMA buffer from user heap - Use k_work_delayable for periodic task instead of LL timer scheduler to keep DMA operations in kernel context - Guard all changes with #ifdef CONFIG_SOF_USERSPACE_LL Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Use a module context instead of heap to allocate objects in chain-dma module. This is required to support vregion use. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Commit f515f3b kept chain DMA in kernel context for CONFIG_SOF_USERSPACE_LL by driving chain_task_run() from a k_work_delayable on the system workqueue, because chain DMA used the raw Zephyr DMA API (dma_get_status/dma_reload/dma_start/dma_stop) which is not callable from an unprivileged user thread. Running a ~1ms (500us on AMD) audio DMA cadence on the low-priority system workqueue, rescheduled completion-relative via k_work_reschedule(), is prone to preemption and drift and risks xruns. Port chain DMA to SOF's sof_dma_* syscall wrappers (as host and dai already do) so it can run unprivileged, and schedule it as a normal SOF_SCHEDULE_LL_TIMER task on the user LL thread. This removes the kernel-workqueue path entirely; both userspace and non-userspace builds now share the LL scheduler path with scheduler-managed task lifecycle. Channel handles are stored as integer indices instead of kernel-only struct dma_chan_data pointers. The user-heap allocations for the component, private data and DMA buffer are retained so the user thread can access them. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Currently building with userspace LL and the debug overlay generates non-functional images. This is due to two incompatibilities: (1) cpu_get_id() called in userspace in zephyr_ll_task_init() and (2) the heap_alloc boot-test currently causing an exception. Fix the former and disable the latter to re-enabld debug builds with the userspace enabled. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make scheduling LL thread and synchronisation objects per-core and forward IPCs and scheduling events accordingly. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
set_processing_mode() and get_processing_mode() methods of struct module_interface aren't unused, they are used by IADK. Fix respective comments. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The global driver list should never be accessed from the audio context. Add an assertion for that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_get_data_for_core() returns void pointer, no need to type-cast it to other pointer types. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
syscall-related code can be used when CONFIG_USERSPACE is undefined too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "core" variable only when it is used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "ipc" variable only for cases, when it's used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
ipc4_init_module_instance() should be called when CONFIG_SOF_USERSPACE_LL isn't selected but also when initializing a DP module. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
If LL runs in userspace, it needs access to loaded LLEXT modules, running in DP more too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_is_user() should return true for DP as well. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
schedule_free() is only called from testbench and stand-alone ztest. Remove it and all scheduler .scheduler_free() methods for all other builds. Also fix memory leaks in the Zephyr LL version. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
When running the LL scheduler in userspace, it can happen that no kernel-mode schedulers get registered on a running secondary core. To recognise such cases add a check for userspace schedulers to check_restore(). Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In kernel context cpu_is_me() should return a correct result for correct multicore support. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make ll_sch_is_current() available for builds with CONFIG_COLD_STORE_EXECUTE_DEBUG unselected. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Add two syscall functions to allocate and map, and to unmap vregion for userspace modules. For now only used for DP modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
vregion_get(), vregion_put() and vregion_set_interim() should also be callable from the userspace. Make them syscalls. Also remove redundant symbol exporting since the vregion API shouldn't be used directly by LLEXT modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick() is currently registered as a notifier callback, but it's always triggered deterministically, always with the same-core-only flag, which leads to it being called immediately. So the notifier only adds a layer of indirection and reduces clarity. Replace it with a direct function call. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick(() has to recalculate DP deadlines and reschedule DP threads. Make it a syscall to be able to call it from the userspace LL scheduler. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Both arch_schedulers_get() and arch_user_schedulers_get() should only be called from the privileged context. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make scheduler_dp_internal_free() a syscall in the "application" DP implementation. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In case of userspace LL scheduling the (also userspace) IPC thread needs access rights to DP assets like the thread itself and its stack and synchronisation primitives. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
zephyr_ll_domain() and scheduler_get_task_info_ll() are needed in non-userspace builds too, make them universally available. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The LL userspace thread has to interact with the DP one. Grant required rights. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
When LL runs in userspace, multiple DP functions are called in userspace mode too. They cannot use privileged instructions then. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
DP scheduler operations, instance data and DP module memory have to be accessible to the userspace LL scheduler. Allocate dynamic data on the userspace heap and place static data in the userspace accessible ELF section. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Remove duplicate ll_schedule_domain.h inclusion in ipc-common.c. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Switching to the userspace mode in DP and LL cases differs. Add a comment to explain that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The LL tick task's .priv_data pointer has to be NULL for the check in zephyr_ll_task_init() to pass. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The LLEXT manifest declared the module name as "PHASE_VOCODER", which rimage truncates to "PHASE_VO" (SOF_MAN_MOD_NAME_LEN is 8), while phase_vocoder.toml uses "PHASEVOC". The names do not match, so rimage fails to package modular builds with the error "Module PHASE_VO not found in TOML." on platforms that build the component as loadable module (e.g. PTL). Use "PHASEVOC" in the manifest to match the TOML entry. Fixes #e9bfc2f6b402 Signed-off-by: Adrian Bonislawski <adrian.bonislawski@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This includes #10558 and my patches on top to enable LLEXT and multicore. Current status: passes simple tests with nocodec with both core 0 and core 1 streaming. 2 streams simultaneously run into a problem when the first of them terminates. WiP.