pyxis: allow unprivileged user namespaces so enroot works on Ubuntu 23.10+

[100milliongold]

Problem

Ubuntu 23.10 and later set kernel.apparmor_restrict_unprivileged_userns=1 by default. Under that default, enroot-nsenter fails as soon as pyxis starts a container:

enroot-nsenter: failed to create user namespace: Permission denied

The nvidia.enroot galaxy role does not handle this, and DeepOps lists Ubuntu 24.04 LTS as a supported OS, so container jobs do not work out of the box on a supported platform.

Observed on DGX OS 7.5.0 (Ubuntu-based), Slurm 26.05.1, pyxis 0.11.1.

Fix

Add a sysctl task in roles/pyxis for compute nodes. The task is gated on the presence of /proc/sys/kernel/apparmor_restrict_unprivileged_userns rather than on ansible_distribution_version, so it is a no-op on kernels built without AppArmor userns restrictions and does not need updating for future releases.

Note on hardening

This relaxes a system-wide hardening default: unprivileged user namespaces are what AppArmor is restricting here. A narrower alternative is an AppArmor profile scoped to enroot-nsenter (as packaged by some distributions for other userns consumers). If maintainers prefer that approach, I am happy to rework the patch.

The sysctl is written to /etc/sysctl.d/60-enroot-userns.conf so it is visible and revertible, rather than being applied only at runtime.

[dholt]

Please replace the unconditional host-wide sysctl change with a narrowly scoped AppArmor policy for enroot-nsenter and provide evidence from a real Pyxis/Enroot container launch on an affected supported OS. If a global sysctl fallback is still needed for unusual installations, make it an explicit, documented, default-off administrator choice rather than changing the security default whenever the kernel knob exists.

---

Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).

[100milliongold]

Agreed on all three points — clearing kernel.apparmor_restrict_unprivileged_userns host-wide removes exactly the attack surface the Ubuntu default was added to reduce, and the original patch did it unconditionally.

What changed (commit 86ecc83):

• The role now installs an AppArmor profile that grants the userns capability to /usr/bin/enroot-nsenter alone and leaves the host default in place.
• The host-wide sysctl remains available but is now an explicit, documented, default-off choice: pyxis_userns_allow_globally, false by default. If the profile fails to load, the play stops with the parser error and points at that variable rather than silently falling back to the weaker setting.
• Both paths stay a no-op where /proc/sys/kernel/apparmor_restrict_unprivileged_userns does not exist, and the profile path additionally requires apparmor_parser.

Why the profile only covers enroot-nsenter:

$ grep -l CLONE_NEWUSER /usr/bin/enroot-*
/usr/bin/enroot-nsenter

Checked against enroot 3.2.0-1, so enroot-mount, enroot-switchroot, enroot-mksquashovlfs and enroot-aufs2ovlfs do not need it.

Evidence so far, on a DGX B300 running DGX OS 7.5 (Ubuntu 24.04.4, AppArmor 4.0 ABI available):

$ apparmor_parser -Q -T -v enroot-nsenter
Addition succeeded for "enroot-nsenter".

What is still missing, and I would rather say so than leave it implied: I do not have a container launch on a host where the restriction is active. This node was already worked around out-of-band before I picked up the issue — /etc/sysctl.d/91-enroot.conf, not owned by any package, sets the knob to 0 — so it is not in the affected state. Restoring the restriction to reproduce the failure would break the Pyxis jobs currently running on it (an MLPerf Inference run occupying all 8 GPUs).

I will add the before/after launch output once the node is free. Please hold the merge until then if that evidence is required — I would rather this sit than be merged on a parser check alone.

[100milliongold]

The node is free now, so here is the container launch evidence I said I would add.

To produce it I had to put this host into the affected state first: it carries an out-of-band /etc/sysctl.d/91-enroot.conf that sets the knob to 0, so Pyxis works here for the wrong reason. I moved that file aside, set the knob back to 1, ran the launch, installed the profile, ran the same launch again, and restored the original state afterwards.

Environment

OS:              Ubuntu 24.04.4 LTS
kernel:          6.8.0-106-generic
enroot:          3.2.0-1
apparmor_parser: AppArmor parser version 4.0.1
hardware:        DGX B300, 8x SXM

Before — restriction active, no profile

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
pyxis: importing docker image ...
error: pyxis: container start failed with error code: 1
error: pyxis: printing contents of log file ...
error: pyxis:     enroot-nsenter: failed to create user namespace: Permission denied
error: pyxis: couldn't start container
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
error: Failed to invoke spank plugin stack
srun: error: dgx01: task 0: Exited with exit code 1

exit status: 1

Install the profile

$ apparmor_parser -r /etc/apparmor.d/enroot-nsenter
$ echo $?
0

$ aa-status | grep enroot
   enroot-nsenter

After — same launch, profile loaded, restriction still active

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
srun: job 136 queued and waiting for resources
srun: job 136 has been allocated resources
pyxis: importing docker image ...
container-started-ok

exit status: 0

The knob reads 1 during both launches. That is the part I wanted on the record: the container starts with the OS security default left in place, which is what your review asked for and what the original patch did not do.

Afterwards the profile was unloaded and removed, 91-enroot.conf was put back, and the knob returned to its original value of 0, so the node is as it was.

Two notes on the run itself. The image is plain ubuntu:24.04 rather than one of our local squashfs images, so the reproduction does not depend on anything specific to this site. And pyxis_userns_allow_globally stayed at its default of false throughout — the global sysctl path was never taken.

[dholt]

The scoped AppArmor profile addresses the concern with the original global setting. A few convergence gaps remain in roles/pyxis/tasks/main.yml:

• Loading the profile only when its file changes means a failed first load, or an unloaded profile with an unchanged file, is not repaired on the next run. Reconcile the effective profile on reruns.
• With the restriction active and apparmor_parser absent, the default scoped path is silently skipped. Install the required userspace dependency or fail clearly.
• Turning the global fallback back off leaves its managed sysctl setting behind. Handle or clearly document that migration without overwriting unrelated site policy.

Please cover these transitions with focused regression tests. This does not require weakening the host-wide policy.

[100milliongold]

All three convergence gaps are addressed in eec281e.

1. Reconcile the effective profile on reruns

The load was gated on enroot_userns_profile.changed alone. The kernel's own list now decides as well:

- name: check whether the enroot-nsenter profile is loaded
  command: grep -q "^enroot-nsenter " /sys/kernel/security/apparmor/profiles
  register: enroot_userns_loaded
  changed_when: false
  failed_when: false
  when: ...

- name: load the enroot-nsenter AppArmor profile
  command: apparmor_parser -r /etc/apparmor.d/enroot-nsenter
  register: enroot_userns_load
  failed_when: false
  when:
    - ...
    - (enroot_userns_profile.changed | default(false))
      or (enroot_userns_loaded.rc | default(1) != 0)

Evidence

Three consecutive runs, with the kernel profile list stubbed to model each state:

run	profile file	in kernel list	load task
1	newly installed	no	changed
2	unchanged	yes	skipping
3	unchanged	no (unloaded by hand)	changed

Same three runs against this branch as it stands:

run	profile file	load task
1	newly installed	changed
2	unchanged	skipping
3	unchanged, profile gone from the kernel	skipping — not repaired

2. apparmor_parser absent is no longer a silent skip

- name: install the AppArmor userspace (Ubuntu)
  apt:
    name: apparmor
    state: present
  when:
    - is_compute
    - ansible_distribution == "Ubuntu"
    - apparmor_userns_knob.stat.exists | default(false)
    - not pyxis_userns_allow_globally

- name: fail when the AppArmor userspace is unavailable
  fail:
    msg: >-
      ... the scoped enroot-nsenter profile cannot be loaded and enroot would
      fail with "failed to create user namespace: Permission denied". Install
      the AppArmor userspace, or set pyxis_userns_allow_globally=true to clear
      the restriction for the whole host instead.
  when:
    - ...
    - not apparmor_parser_bin.stat.exists | default(false)

The profile is the default path, so its userspace is a dependency of the role rather than something to skip over. If it is still missing after the install, the play stops instead of reporting success while enroot keeps failing.

3. Turning the fallback off takes its setting with it

- name: drop the host-wide user namespace sysctl when the fallback is off
  ansible.posix.sysctl:
    name: kernel.apparmor_restrict_unprivileged_userns
    sysctl_file: /etc/sysctl.d/60-enroot-userns.conf
    state: absent
  register: enroot_userns_sysctl_removed
  when:
    - is_compute
    - apparmor_userns_knob.stat.exists | default(false)
    - not pyxis_userns_allow_globally

- name: reapply the remaining sysctl configuration
  command: sysctl --system
  changed_when: true
  when: enroot_userns_sysctl_removed.changed | default(false)

Removing the line alone would leave the running kernel at the lowered value until the next boot, so the remaining configuration is reapplied. sysctl --system was chosen over writing a value back because it restores whatever the distribution ships rather than a number this role invents.

[dholt]

The parser dependency and unloaded-profile handling address those review points. One migration issue remains in the global-fallback cleanup:

sysctl --system replays unrelated system settings, and it only runs when removal of the managed line reports a change. If restoration fails or is interrupted after that removal, the next run skips it and can leave the runtime restriction disabled.

Please limit restoration to the owned setting and verify the effective value independently of whether the file changed, or fail clearly for administrator reconciliation. Add the interrupted/failed-restoration case; do not reload unrelated host-wide policy.

[100milliongold]

Both problems are real, and 72e3d185 drops sysctl --system entirely.

What the old code got wrong

Two separate faults, as you split them:

1. sysctl --system replays every file under /etc/sysctl.d, /run/sysctl.d, /usr/lib/sysctl.d and /etc/sysctl.conf. Undoing one owned line should not reapply unrelated policy an administrator may have changed at runtime.
2. The reload was gated on enroot_userns_sysctl_removed.changed. Interrupt or fail the run between the removal and the reload and the file is gone while the runtime restriction is still off — and because the line is already absent, the next run reports no change and skips the reload. The node stays that way indefinitely. The file was standing in for the kernel, and they had already diverged at that point.

What it does now

The running value decides, read every time from /proc/sys/kernel/apparmor_restrict_unprivileged_userns, with no reference to whether the file changed:

1. Remove the managed line (unchanged).
2. Read the effective value.
3. If it is not 1, reapply only this key with systemd-sysctl --prefix=kernel.apparmor_restrict_unprivileged_userns. That replays no unrelated policy, and it honours the precedence of any other file that legitimately declares the key — which matters on this node, where an out-of-band /etc/sysctl.d/91-enroot.conf sets it to 0 and should win over anything this role assumes.
4. Re-read. If it is still not 1, nothing on disk declares the key any more, so the value left in the kernel is this role's own write. Undo just that write with sysctl -w. No file is created, so the boot default takes over from the next reboot and nothing is invented.
5. Re-read once more and fail with the current value if it is still not 1, naming the three directories to reconcile by hand. The role manages its own entry and will not override another one.

Step 2 is the answer to the interrupted case: it does not matter how the value came to be wrong, only that it is.

Binary location

systemd-sysctl is not on PATH; the task stats /usr/lib/systemd/systemd-sysctl and /lib/systemd/systemd-sysctl and uses whichever exists. Where neither does, step 3 is skipped and step 4 still restores the value, so a non-systemd host degrades to the targeted write rather than to sysctl --system.

One defect I introduced and fixed

The first version used selectattr('stat.exists') on the stat loop results. When that loop is skipped the results carry no stat key and the filter raises 'dict object' has no attribute 'stat'. I reproduced it, then confirmed Ansible short-circuits when: lists so the live code could not reach it — the preceding condition is false in exactly that case. Added selectattr('stat', 'defined') ahead of it anyway rather than leave correctness resting on condition order.

What I have not done

This is reasoning plus a syntax check and isolated Jinja/Ansible tests of the conditionals, not a run against a host in the affected state. The node I have access to carries the out-of-band file described above, so putting it into the affected state means moving that aside again. I can do that when the node is free and post the before/after, as with the earlier launch evidence. Say if you want that before merge.

[dholt]

The targeted replay and retry-independent reads address the previous broad-reload/interruption issues, but two policy-preservation gaps remain at this revision:

1. roles/pyxis/tasks/main.yml:192-200: a value of 0 after replay does not establish that no remaining configuration declares the key. If an administrator-owned file intentionally sets it to 0, replay correctly preserves that choice, then sysctl -w ...=1 overrides it and the final reconciliation failure is skipped. The same happens on a first scoped deployment with an existing site policy. Please remove the unconditional forced write and fail clearly for administrator reconciliation when targeted replay cannot establish the required effective value. Keep the reads/reconciliation independent of whether the managed file changed.
2. roles/pyxis/tasks/main.yml:130-134: ansible.posix.sysctl defaults to reload: true even with state: absent. If another key remains in that file, removal invokes sysctl -p on the entire file and reapplies that unrelated setting. Add reload: false to this removal task; the subsequent targeted step owns runtime reconciliation.

An independent review reproduced both cases with inert fixtures: actual current-head Ansible conditions select the forced write after an administrator's =0, and the pinned ansible.posix module issues the whole-file reload when another key remains. Controls with the two small corrections preserve the administrator's value and unrelated setting. These were isolated condition/module tests, not a host deployment.

Please retain regressions for both cases and the interrupted-removal retry. The scoped profile itself passed a non-loading syntax compilation; no broader redesign or feature removal is requested.

[100milliongold]

Both points are real. 974de010 and 99d35f16 address them.

1. The forced write overrode the administrator

Confirmed as described. A value other than 1 after the targeted replay does not
establish that the kernel is holding this role's own leftover write: an
administrator-owned file may declare the key and legitimately hold it down, in
which case the replay preserved their choice and sysctl -w ...=1 undid it. And
because that write then satisfied the final read, the override was silent — the
reconciliation failure never fired.

The role now asks the configuration which files declare the key before writing
anything, and writes only when none does:

- name: find configuration that still declares the user namespace restriction
  shell: |
    set -o pipefail
    grep -rlsE '^[[:space:]]*-?[[:space:]]*kernel[./]apparmor_restrict_unprivileged_userns[[:space:]]*=' \
      /etc/sysctl.conf /etc/sysctl.d /run/sysctl.d /usr/lib/sysctl.d /usr/local/lib/sysctl.d \
      2>/dev/null || true

Both spellings systemd accepts are matched — the dotted kernel.x form and the
path kernel/x form, each optionally prefixed with -. Checked against
fixtures covering all four shapes plus a commented-out line and a missing
directory.

When something else declares the key, the existing failure now names the files
rather than describing the situation in the abstract. The reads stay independent
of whether the managed file changed.

This also covers the first-scoped-deployment case you raised: a host where an
administrator's file already sets 0 now fails for reconciliation instead of
being quietly forced to 1.

2. state: absent reloaded the whole file

Confirmed from the module source rather than the docs alone
(ansible.posix as bundled with ansible 8.5.0,
plugins/modules/sysctl.py):

• 394: reload=dict(default=True, type='bool')
• 166-168: state: absent with the key present sets changed = True
• 194-195: if self.changed and self.args['reload']: self.reload_sysctl()
• 302: on Linux that is sysctl -p <sysctl_file>

So removing the managed line reapplied any other key an administrator had put in
that file. reload: false is now set on the removal. write_file is a separate
branch (192), so the removal itself is unaffected; the targeted steps that
follow own runtime reconciliation.

Regressions

tests/pyxis-userns/run-tests.py pins which tasks a host state selects. It
loads the real task file, evaluates the actual when: expressions, and asserts
run/skip:

host state	restore write	reconciliation failure
an administrator file declares the key at 0	skipped	fires
removal interrupted, nothing declares the key	runs	skipped
replay put the value back	skipped	skipped
already restricted	skipped	skipped

Two further assertions:

• the same administrator state replayed with the declaration condition taken
  back out, asserting the write would have run. Without that the suite could
  pass while proving nothing about the guard it exists to cover.
• the removal task carries reload: false.

Against the current branch: 6 passed, 0 failed. Against a mutant with both
corrections removed: 2 failed, naming each fault.

What this does not show

This is a condition-level test. It shows which tasks a host state selects, not
what those tasks then do on a host. I have not run it against a host in the
affected state — the node I have access to still carries the out-of-band
/etc/sysctl.d/91-enroot.conf described earlier, so reaching that state means
moving it aside again. I can do that and post before/after if you want it before
merge.

[dholt]

reload: false fixes the whole-file replay, and ordinary readable declarations of =0 are now preserved. The new discovery-based permission to force =1 still has two concrete gaps:

• roles/pyxis/tasks/main.yml:207-209,224-233: grep -r does not follow a policy-file symlink found inside sysctl.d. With 91-site.conf linking to a readable administrator file outside the searched directories, the actual scanner returns success with empty output, then the actual restore conditions select sysctl -w ...=1. Systemd can consume that symlinked drop-in. Valid systemd glob assignments are also outside this literal regex. Switching only to -R therefore does not establish complete policy discovery or prove who set the runtime value.
• main.yml:204-213,233: || true converts discovery read/traversal errors into successful empty output. An inert unreadable-file control produced raw grep rc=2, but task rc=0/empty output and a selected forced write. A failed earlier replay would stop the play; this defect matters where replay is unavailable or discovery fails after it.

Please use the smaller fail-clearly option from the prior review: keep removal, targeted replay and independent effective-value reads, but remove the forced-write shortcut when replay leaves an unresolved value. Use the existing administrator-reconciliation failure instead. A missing declaration does not distinguish an administrator's runtime-only choice from a previous role write. Discovery can supply diagnostics, not authorization to overwrite policy. Keep the scoped AppArmor profile and explicit global opt-in unchanged.

The six supplied tests pass, but discovery replaced with : and disabled initial/final read guards also leave them green. Add isolated scanner/read-chain controls and negative controls, not only precomputed declaration strings. Independent review and a separate parent fixture both reproduced the symlink gap without running sysctl.

I handled the new condition-test CodeQL XSS false positive separately; it is not an additional code-change request.