[EventHubs] Retry a CBS authentication failure once on a fresh connection

[j7nw4r]

Summary

The uAMQP Event Hubs transport does not recover when a CBS PutToken operation returns a non-OK result. A synchronous send or receive can also return an exact amqp:unauthorized-access result after the connection authorization state becomes stale.

The failed public call can succeed after the client replaces the complete AMQP stack and authenticates on a fresh connection.

Motivation

A fresh connection can repair stale or damaged CBS state without making authorization failures globally transient. The recovery must remain bounded and must not retry credential rejection or asynchronous link failures.

Proposal

Allow one additional authentication attempt on a fresh connection per public producer or receiver call in the uAMQP transport.

• Leave the Rust AMQP transport unchanged.
• Honor RetryOptions.MaxRetries == 0; this value disables authentication recovery.
• Track the authentication recovery separately from ordinary transient retries and share it across nested phases of one public call.
• Recover only a non-OK CBS PutToken result or an exact synchronous amqp:unauthorized-access send or receive result.
• Do not recover credential GetToken failures, asynchronous link or attach authorization failures, or CBS open Invalid and Cancelled results.
• Apply the ordinary retry budget to a CBS open Error result and other transient failures that occur before CBS returns an authentication result.
• Replace the complete producer or receiver stack before the authentication attempt.
• Preserve the consumer-created PartitionClient handle and resume exclusively after the last delivered offset.
• Use the configured first-retry delay with the existing 0.8 to 1.3 jitter and MaxRetryDelay cap.
• RetryOptions.RetryDelay defaults to 800 ms. The default first-retry range is 640 to 1,040 ms.
• Use an Azure::Core::Context-aware wait.
• Stop after the second authentication failure and preserve the final public exception type and details.
• Preserve generation and in-flight safety during concurrent teardown.

This change requires explicit security-focused review.

Validation

• A producer operation with a failed CBS PutToken rebuilds once and succeeds within the same call.
• A receiver operation with a failed CBS PutToken rebuilds once and resumes from the correct position.
• An exact synchronous amqp:unauthorized-access transfer rebuilds once.
• RetryOptions.MaxRetries == 0 causes no additional authentication attempt.
• A credential GetToken failure causes no additional authentication attempt.
• An asynchronous link or attach authorization failure causes no additional authentication attempt.
• A transient CBS open error uses the ordinary retry budget.
• A second authentication failure stops recovery and preserves its original public details.
• Cancellation during the recovery delay returns promptly.
• Concurrent teardown cannot discard a stack that another thread rebuilt.
• Rust AMQP production behavior and tests remain unchanged.