Skip to content

Record payment and outcome of each call_runner attempt - #71

Closed
eliteprox wants to merge 6 commits into
livepeer:mainfrom
eliteprox:feat/call-runner-manifest-id
Closed

eliteprox wants to merge 6 commits into
livepeer:mainfrom
eliteprox:feat/call-runner-manifest-id

Conversation

@eliteprox

@eliteprox eliteprox commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • call_runner now writes every attempt to a RunnerCallRecord as it happens. Each RunnerAttempt records the payment challenge it answered (manifest_id), the signer auth_id it opened, whether the request went out with Livepeer-Payment headers (payment_sent), the runner's HTTP status, and how the attempt ended (outcome).
  • The result, the stream and any error from the call all point to the same record. A caller can pass in its own record= so it can still read it after cancelling the call (for example with asyncio.wait_for).
  • payment_sent and auth_ids on LivepeerGatewayError, LiveRunnerCallResult and LiveRunnerCallStream are read-only properties computed from the record. payment_sent is true once any attempt sent payment, so a gateway may switch to another runner only while it is false. auth_ids lists every signer session the call opened. The signer charges when it signs, so an auth id can appear before payment_sent is true.
  • Errors that are not a LivepeerGatewayError (for example a ClientPayloadError while reading an error body) are re-raised as LivepeerGatewayError with the record attached.
  • Attempt outcomes:
    • unreachable only when the connection itself failed, so the request never got to the runner;
    • timeout for a plain timeout, since the runner may already have the request;
    • also succeeded, payment_required, refused (503), http, payment (signer failure), disconnected, invalid_response and cancelled.
  • LivePaymentSession.auth_id reads the signer's AuthID from the payment state, decoded once per payment response.
  • open_stream now closes its HTTP session when reading an error body fails.
  • LiveRunnerCallStream.session_id carries the paid challenge's manifest_id, as LiveRunnerCallResult.session_id already does.

Runner selection and failover (RunnerSelectionCursor, Scope startup, RunnerRejection) are unchanged and match main. Making them use the record, and stop failing over once a payment is sent, is planned for a follow-up PR.

Test plan

  • uv run --frozen --group lint ruff check: no new findings on changed lines
  • uv run --frozen --group test pytest -W error::ResourceWarning: 267 passed
  • New tests run call_runner against a real aiohttp runner on a local port, with a fake signer, and cover:
    • a signer failure after a paid attempt;
    • an unpaid retry after a signer refresh, which still reports paid;
    • a refresh on the final attempt, which keeps the earlier auth ids;
    • a truncated 500 on a stream, which is wrapped and reports paid;
    • a cancelled call, whose record stays readable;
    • a refused connection (unreachable) versus a slow runner (timeout);
    • the outcomes for 503 and 400.
  • livepeer_core test suite passes against this branch

🤖 Generated with Claude Code

LiveRunnerCallStream.session_id carries the answered payment challenge's
manifest_id, as LiveRunnerCallResult.session_id already does for unary calls;
it is "" when no challenge was answered.

Errors raised from a call_runner attempt carry payment_sent, True when that
attempt's request already carried Livepeer-Payment headers, and manifest_id,
the paid challenge's manifest_id ("" when unpaid). A gateway may fail over to
another runner only while payment_sent is False, and can attribute the
network cost of a paid call that failed. When the challenge retries run out,
payment_sent covers every attempt and manifest_id is the last paid one.

manifest_id is the key the signer's create_signed_ticket Kafka event carries
as manifest_id; it is not the event's session_id (the signer state id) or
pm_session_id (the orchestrator ticket params' recipientRandHash).

Replayed onto livepeer/livepeer-python-gateway main without the multipart
commit this change was previously stacked on.
Selection keeps each failed attempt as capacity, unreachable, or other, and copies payment_sent and the paid manifest id onto the rejection so a gateway can stop after a paid attempt.
eliteprox and others added 3 commits October 7, 2026 14:48
Refactor the error handling in LivepeerGatewayError and related classes to replace the manifest_id with auth_ids, which now captures the signer auth IDs for each payment session. This change allows for better tracking of payment sessions across errors and runner rejections, ensuring that the system can accurately attribute costs and manage retries based on the associated auth IDs. Update tests to reflect these changes and ensure proper functionality.
Merge auth ids through one unique_auth_ids helper, attach the signed
session's auth id in a single place in _get_runner_payment, decode the
signer state's auth id once per payment response, drop the unused
PaymentSession.auth_id, and simplify the unreachable exception check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
call_runner now writes every attempt to a RunnerCallRecord as it happens:
the challenge answered, the signer auth id, whether the request left with
Livepeer-Payment, the runner's status, and how it ended. The result,
stream, and any error hold that same record, and a caller can pass its
own to read it after cancellation. payment_sent and auth_ids on errors,
results, and streams are derived from it, which replaces per-raise-site
stamping that missed signer failures, refresh retries, and non-gateway
exceptions.

Attempt outcomes report "unreachable" only when the request provably
never left; a bare timeout reads as "timeout". Unexpected exceptions are
wrapped in LivepeerGatewayError so they carry the record, and open_stream
now closes its session when reading an error body fails.

Runner failover (selection.py, scope.py, RunnerRejection) is reverted to
main and moves to a follow-up PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eliteprox eliteprox changed the title Carry the paid manifest_id on call_runner streams and errors Record payment and outcome of each call_runner attempt Oct 12, 2026
Comment thread src/livepeer_gateway/remote_signer.py Outdated
Comment on lines +365 to +366
self._state = state
self._auth_id = auth_id_from_signer_state(state)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we avoid storing _auth_id separately and keep it within state for downstream parsing by auth_id_from_signer_state

Comment thread src/livepeer_gateway/errors.py Outdated
Comment on lines +4 to +6
from typing import TYPE_CHECKING

if TYPE_CHECKING:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TYPE_CHECKING can usually be removed

…tate

RunnerAttempt, RunnerAttemptOutcome and RunnerCallRecord now live in errors.py,
so LivepeerGatewayError can reference RunnerCallRecord without a TYPE_CHECKING
import. LivePaymentSession.auth_id is parsed from the stored signer state
instead of being cached separately.
@eliteprox

Copy link
Copy Markdown
Collaborator Author

Moved to eliteprox#5, which targets releases/m2. Both review comments are addressed there: the call record types moved into errors.py, so the TYPE_CHECKING import is gone, and LivePaymentSession.auth_id now reads from the stored signer state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant