Resolve native symbols for containerized python applications - #329
Resolve native symbols for containerized python applications#329pramodk wants to merge 3 commits into
Conversation
9558b20 to
695ff34
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #329 +/- ##
==========================================
- Coverage 77.93% 77.81% -0.13%
==========================================
Files 56 56
Lines 5988 6005 +17
Branches 631 637 +6
==========================================
+ Hits 4667 4673 +6
- Misses 1321 1332 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| if (ret < 0) { | ||
| ret = find_elf_through_proc_pid_root(userdata, modname, file_name); | ||
| } |
There was a problem hiding this comment.
This only falls back to checking through /proc/$pid/root if the lookup on the host filesystem failed, but that seems wrong to me - isn't it possible that we've found entirely the wrong file if we looked on the host filesystem first? I would think that we'd want to always go through /proc/$pid/root whenever the mount namespace doesn't match. Or always, really? I can't think of any reason we wouldn't want to always resolve relative to it...
There was a problem hiding this comment.
In fact, I wonder whether this is something we should be implementing in pystack at all. Perhaps libdwfl should automatically be using /proc/$pid/root whenever it resolves files from /proc/$pid/map...
There was a problem hiding this comment.
@godlygeek - Sorry for delay in response during last month.
I have pushed new commits to address some of these points:
- Use
/proc/$pid/rootfor module lookup. I have kept the old path as a fallback in case there are corner cases where this is required. - Integration test that exercises this behavior with separate namespaces.
Regarding libdwfl taking care of this, I have checked the libdwfl implementation and asked on the developer list about possible fixes to support this scenario. I will see if there is any response or concrete proposal.
Until that type of support is generally available, it would be helpful to have this fix in PyStack?
I have tested this in a fork here. So hopefully the tests will pass after CI is approved to run.
|
This will definitely need an automated integration test added to the test suite, also! |
2e21b43 to
32d6854
Compare
- Add a libdwfl fallback that opens absolute module paths through /proc/<pid>/root so remote Pyxis/Enroot targets can resolve libraries that are only visible inside the container filesystem. - Associate the analyzed PID with each DWFL module before attach so the ELF lookup callback can find the target process root during native unwinding. - Preserve the existing build-id and linux-proc lookup paths first; the process-root fallback is used only when normal host lookup fails. - This fixes the behavior seen with the ImageNet Pyxis NCCL run where PyStack 1.6 reported insufficient native information or lost libtorch/libtorch_cuda/NCCL frames from container targets. Signed-off-by: Pramod Kumbhar <prkumbhar@nvidia.com>
- Resolve mapped paths through /proc/<pid>/root before libdwfl's fallback to avoid opening a different host file. - Preserve plain module paths for identical files so same-namespace module and debuginfo behavior stays unchanged. - Handle deleted and missing module names safely, and match load points against reported, main, and debug paths. Signed-off-by: Pramod Kumbhar <prkumbhar@nvidia.com>
- Build target and host-decoy libraries at one mapped path to verify native symbols use the target filesystem view. - Require the mount-namespace test in coverage CI while allowing unsupported local environments to skip it. Signed-off-by: Pramod Kumbhar <prkumbhar@nvidia.com>
32d6854 to
bfeccb0
Compare
3 participants
Describe your changes
This change fixes native symbol lookup for Python processes running inside container runtimes such as Enroot.
When the libdwfl lookup cannot open an absolute module path from
/proc/<pid>/maps, this PR now retries that same path through the target process root:The fallback is only used after build-id lookup and normal host path lookup fail, so existing host behavior remains unchanged.
Fixes #327.
Testing performed
Additional context
The issue is specific to native C/C++ symbol lookup. Python frames can still be visible because PyStack can inspect the target process, but native library paths from
/proc/<pid>/mapsmay exist only inside the container filesystem. As containers are used extensively in certain domains like AI/ML/Data Science, it would be really helpful to fix this.