Skip to content

Keep retention progressing when the oldest expired runs are held #500

Open

Description

@rmcdaniel

A retention pass always selects the oldest expired, unarchived closed runs. A run that is skipped stays at the front of that order, and the next pass selects it again. When skipped runs fill a pass's limit, retention makes no progress on other eligible runs in that namespace.

Current behavior

Main 4e0f6458d890f4b34f1959c12dd385b2b1ea01f7:

  • HistoryRetentionEnforcer::expiredRunIds orders candidates by closed_at with a limit and no exclusion for runs that were recently skipped or failed.
  • The inline pass (runInlinePass) uses a limit of 1. One skipped run at the head stops inline pruning for that namespace.
  • pruneRun can skip for reasons that persist until an operator acts, for example external_payload_storage_driver_unavailable or activity_policy_backend_unsupported.
  • The scheduled history:prune pass uses a limit of 100, so it stalls only when at least that many such runs sit at the head. It does not print skip reasons and exits 0 when it only skipped.

This was found by reading source. It was not reproduced.

Expected

A held or failing run does not prevent pruning of other eligible runs. Operators can see which runs are held and why.

Acceptance

  • With a persistently skipped run at the head, inline and scheduled passes still prune eligible runs behind it.
  • Skipped and failed runs are reported with run id and reason through the retention status API and history:prune output. The command exits non-zero, or clearly reports no progress, when it made none.
  • Skipped runs are retried later without a tight loop on every pass.
  • Forever-retention, archived and hold semantics are unchanged.
  • Coordinate Rust parity through Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    priority:P3Lower-priority product improvement
    status:readyReady for implementation
    on Oct 19, 2026
  2. rmcdaniel commented on Oct 21, 2026

    @rmcdaniel
    MemberAuthor

    Native status (Rust Server, #325). Native retention landed in #595 and follows published PHP 2.5.14. It has the same head-of-line stall described here:

    • expired_run_ids orders by closed_at, id with a limit and excludes nothing that was skipped earlier.
    • The heartbeat inline pass prunes one run.

    Native is more exposed than PHP. It does not clean up external payload objects or message streams yet, so it skips any run that holds either, with external_payload_cleanup_unavailable or message_stream_cleanup_unavailable. That skip persists. On a namespace whose oldest expired run has such a reference, the native inline pass stops pruning that namespace.

    To keep the two servers compatible, I'll implement the same correction in native once this issue settles on a design. The design matters most where it touches shared state. If skipped runs get a durable retry-after or held marker, that becomes a column or table both servers write, and it belongs in the shared schema. If the scan instead excludes held runs with SQL predicates, native can follow without a schema change.

    Please record the chosen approach here before implementation. Native will follow it, and #325's port log tracks the gap until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingkind:defectA public product behavior is incorrectpriority:P3Lower-priority product improvementstatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions