Retry transient failures for safe real Model Target requests

[adamtheturtle]

Problem

The Model Target contract tests can fail on a transient gateway response from the real Vuforia even when the code and the API contract under test are behaving correctly.

For example, PR #3522's Model Target job received an AWS ELB 502 Bad Gateway while making an unauthenticated GET request to a dataset download endpoint. The same job passed unchanged when rerun.

The existing retry mechanisms do not clearly cover this case:

• RETRY_ON_TRANSIENT_VWS_FAILURE handles selected SDK exceptions in fixture/setup helpers, not raw requests.Response objects returned by the Model Target test helpers.
• The repository-wide pytest-retry configuration marked this test for retries, but the failed CI attempt stopped after the first assertion failure and produced no retry report. This needs to be understood rather than assumed to protect these requests.

Proposed change

Add an explicit, bounded retry policy for safe requests made to the real Model Target backend:

• Initially limit automatic retries to GET requests, where repeating the request cannot create a dataset or consume training allowance.
• Retry transient transport failures such as connection/read timeouts and connection errors.
• Retry HTTP 502, 503, and 504 responses.
• Use a small bounded attempt count (for example, three attempts) with backoff and jitter, honoring a reasonable Retry-After value when present.
• After the retry budget is exhausted, retain the current detailed assertion failure, including the final status and response body.
• Do not apply this policy to the in-memory backends, so tests of deliberately configured 5xx responses remain immediate and deterministic.
• Do not automatically retry dataset-creation POST requests: their outcome can be ambiguous, and repeating them can consume quota or create duplicate resources.

The implementation should live in the shared Model Target request/test utility rather than in individual test cases.

Tests

• A real-backend request abstraction retries a transient response and returns the subsequent expected response.
• The retry stops after the configured limit and exposes the final response details.
• Non-transient 4xx responses are not retried.
• Mock-backend configured 5xx responses are not retried.
• Mutating requests are not retried by this policy.
• Add a regression test or documented diagnosis for why pytest-retry did not retry the failing parametrized Model Target test item.

Acceptance criteria

• A one-off 502/503/504 or network failure on a real Model Target GET does not fail CI.
• Persistent upstream failures still fail clearly after a short, bounded delay.
• The policy cannot consume additional Model Target training allowance or hide intentional mock failure responses.