Audit: Are there cases where cancellation of a write could leak files?

[smklein]

#10063 was the source issue.

This issue tracks:

• Are there cases where we are performing asynchronous {file,directory} operations which create new filesystem entries
• ... which could also be cancelled...
• ... but which could have side-effects which occur after cancellation?

[smklein]

In nexus/src/app/background/tasks/support_bundle/collection.rs, we use cancellation of the collect_bundle_as_file task to potentially stop an ongoing bundle collection.

Bundle collection involves a ParallelTaskSet, which explicitly aborts all spawned tasks - but, as we observed in #10063 , this doesn't actually stop work which has been moved to a blocking task (like tokio::fs::write calls).

However, because the bundle collection is doing many writes to files within /var/tmp, we want this ordering when we cancel:

1. collect_bundle_as_file writes to /var/tmp
2. Cancellation aborts all ongoing tasks, and stops all writes
3. The support bundle collector should unwind the stack, and then drop the temporary directory, removing all usage of /var/tmp.

However, in reality, I believe the following can happen:

• After we abort all ongoing tasks, they still execute up to the next await point. They could write additional files, after step (3) has completed
• Furthermore, even if all ongoing tasks have been joined, if any of them handed off work to blocking tasks (like tokio::fs::write), then errant file creations within /var/tmp could happen long after the background task stack unwinds and completes.

[smklein]

Other cases:

test_destroy_published_when_instance_removed and test_failed_published_when_zone_gone are susceptible to this class of issues (they leave a runner up, and drop the temporary directory before the runtime), but I"m not sure it's currently possible to trigger a flake with them, because of the work that happens to be ongoing in the runner task.