[yuechao-qin]

What

The orchestrator loop is single-threaded, so the wall time of one failing log upload delays every
other container execution waiting to be processed. _retry gave every call 5 attempts a second
apart with no bound on the total, so a log read that cannot succeed spent ~30s of the loop
before raising.

Two changes: _retry learns a deadline, and the launcher — not the loop — decides what the
deadline is.

	before	after
attempts, Kubernetes log upload	5	1
sleeps between them	4 x 1s	0
worst case before raising	unbounded	10s
every other launcher	—	unchanged (None)

How

_retry(func, max_retries, wait_seconds, max_total_seconds)
                                        ^^^^^^^^^^^^^^^^^ new: stop once the next
                                                          attempt would start past it

The deadline is checked before sleeping, not before attempting, so the final attempt can still
overrun by one request timeout. Bounding that too would mean threading a timeout into func,
which is the callee's concern, not the retry helper's.

The three knobs then become class attributes on LaunchedContainer:

LaunchedContainer           log_upload_max_retries = None   -> orchestrator default (5)
                            log_upload_wait_seconds = None  -> orchestrator default (1.0s)
                            log_upload_max_total_seconds = None -> unbounded, as today
  LaunchedKubernetesJob     1 attempt, 10s ceiling  \
  LaunchedKubernetesContainer  1 attempt, 10s ceiling / shared constants, so the Job and
                                                       Pod paths cannot drift apart

How long a log read is worth waiting for is a property of the platform: reading from a local
informer cache and reading over a WAN are not the same bet. They sit on the launched container
rather than the launcher because a deployment may route through a launcher-router, in which case
the loop never holds the concrete launcher. Precedent: LauncherError.is_retriable.

The cost, measured

Measured on one deployment over 72 hours: 323 failing log uploads produced 1723 attempt log lines
— 1723 / 323 = 5.33, the five attempts — each costing the single loop thread a second of sleep
between tries. Queries and raw figures are in
Shopify/oasis-backend#827, an internal
repository.

A log upload that fails now raises sooner rather than differently; what makes a lost log stop
raising at all is the companion change, #418.

Verification

pytest --ignore=tests/instrumentation -q → 445 passed. 15 tests cover the deadline, the three
knobs, and the real Job and Container classes driven through the orchestrator's helper; removing
either launcher's budget turns exactly that launcher's parametrised case red.