Skip to content

fix(execution): fail an attachment job when the disk does, instead of escaping - #259

Merged
dovvnloading merged 1 commit into
mainfrom
fix/artifact-boundary-catches-disk-failure
Sep 7, 2026
Merged

fix(execution): fail an attachment job when the disk does, instead of escaping#259
dovvnloading merged 1 commit into
mainfrom
fix/artifact-boundary-catches-disk-failure

Conversation

@dovvnloading

Copy link
Copy Markdown
Owner

The problem

A full disk while staging a chat attachment leaves the job non-terminal for the rest of the installation's life and answers the route with an unhandled HTTP 500 instead of a stable error code.

Reproduced by making publish_artifact raise OSError(28, "No space left on device"):

Before After
raised to the caller OSError escaped the boundary AttachmentStagingError
job status left as queued failed

Root cause

_publish_bytes caught only one of the two exception types publish_artifact can produce:

except ExecutionRepositoryError:
    raise ArtifactBoundaryError("artifact_publish_failed") from None

publish_artifact re-raises. Its own handler cleans up the partial files and then propagates the original:

except Exception:
    target.unlink(missing_ok=True)
    temporary.unlink(missing_ok=True)
    raise

So a full disk, a permission error, or an antivirus lock on temporary.open("xb") / os.replace arrives unwrapped as OSError — and escapes the boundary entirely.

The caller is written to rely on that boundary:

try:
    artifact = self.boundary.stage_bytes(...)
except ArtifactBoundaryError as exc:
    code = self._boundary_code(exc.code)
    self._fail(job.job_id, code)          # <- never runs for OSError
    raise AttachmentStagingError(code) from None

create_job has already committed a row by this point, so when _fail is skipped the job simply stays queued. Nothing later reaps it — the retention pass only removes terminal jobs.

Not a deliberate distinction

publish_outputs, in the same class a few lines down, already treats both identically — catching ExecutionRepositoryError and OSError, each with its own rollback:

except ExecutionRepositoryError:
    ...
    except (ExecutionRepositoryError, OSError):
        raise ArtifactBoundaryError("artifact_cleanup_pending") from None
    raise ArtifactBoundaryError("artifact_publish_failed") from None

So the author already knew publish_artifact can raise OSError. _publish_bytes was simply out of step with its sibling.

The fix

except (ExecutionRepositoryError, OSError):
    raise ArtifactBoundaryError("artifact_publish_failed") from None

No rollback is needed here that is not already done: publish_artifact removes both the temporary and target files on its way out, and _publish_bytes has created nothing else.

Verification

The new test injects OSError(28) and asserts both halves of the contract — that a typed error reaches the caller, and that the job row is left terminal:

assert statuses == ["failed"], "the staging job was left non-terminal"

The job-status assertion is the one that matters: an exception-type-only test would pass on a fix that translated the error but still stranded the job.

Against the unfixed code it fails.

Check Result
python -m pytest -q 907 passed (906 on main + 1 new)
python -m mypy clean, 79 source files
python -m ruff check backend tests tools main.py app_factory.py clean
python tools/artifact_boundary_review.py --json --strict 12/12 passed

Run on Python 3.14, one of the versions in the compatibility matrix.

Compatibility and rollback

Widens one except clause. Every success path and every already-handled failure behaves exactly as before; the only change is for an exception that previously escaped unhandled. No API contract, stored data, or migration. Reverting the commit restores the previous behaviour exactly.

Limits

Jobs already stranded by this before the fix are not retroactively failed — they are non-terminal rows that the retention pass will not collect. Sweeping them needs a migration that can distinguish a genuinely-stuck job from a running one, which is a separate change.

The error code is the existing generic artifact_publish_failed. A disk-full condition is arguably worth its own code so the UI can say "not enough disk space", but that is a contract addition rather than a correctness fix.

🤖 Generated with Claude Code

… escaping

A full disk while staging an attachment left the job non-terminal for good and
answered the route with HTTP 500.

`_publish_bytes` caught only `ExecutionRepositoryError`:

    except ExecutionRepositoryError:
        raise ArtifactBoundaryError("artifact_publish_failed") from None

But `publish_artifact` re-raises: its handler removes the partial files and
then `raise`s the original, so a full disk, a permission error or an antivirus
lock arrives unwrapped as `OSError`. That escaped the boundary, so
`AttachmentStagingService`'s `except ArtifactBoundaryError` did not match,
`_fail` never ran, and the job it had already created stayed at `queued`
forever while the exception surfaced as an unhandled 500.

Reproduced by making `publish_artifact` raise `OSError(28)`:

    before: OSError escaped the boundary     job status left as: queued
    after:  AttachmentStagingError raised    job status left as: failed

`publish_outputs`, a few lines down in the same class, already catches
`ExecutionRepositoryError` and `OSError` and rolls back for both -- so this
was an inconsistency between two sibling paths rather than a deliberate
distinction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dovvnloading
dovvnloading merged commit 38331f9 into main Sep 7, 2026
7 checks passed
@dovvnloading
dovvnloading deleted the fix/artifact-boundary-catches-disk-failure branch September 7, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant