Skip to content

PYTHON-5794 - Add prose tests to verify correct retry behavior when a…#2755

Open
NoahStapp wants to merge 10 commits intomongodb:masterfrom
NoahStapp:PYTHON-5794
Open

PYTHON-5794 - Add prose tests to verify correct retry behavior when a…#2755
NoahStapp wants to merge 10 commits intomongodb:masterfrom
NoahStapp:PYTHON-5794

Conversation

@NoahStapp
Copy link
Copy Markdown
Contributor

@NoahStapp NoahStapp commented Apr 14, 2026

… mix of overload and non-overload errors are encountered

PYTHON-5794

Changes in this PR

  • Add the two new prose tests required by DRIVERS-3446.
  • Fix retry behavior to comply with the backpressure spec correctly.

Test Plan

Added two new prose tests.

Checklist

Checklist for Author

  • Did you update the changelog (if necessary)?
  • Is there test coverage?
  • Is any followup work tracked in a JIRA ticket? If so, add link(s).

Checklist for Reviewer

  • Does the title of the PR reference a JIRA Ticket?
  • Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
  • Is all relevant documentation (README or docstring) updated?

… mix of overload and non-overload errors are encountered
Copilot AI review requested due to automatic review settings April 14, 2026 16:43
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds new prose-style tests and updates client retry logic to ensure that, after encountering a SystemOverloadedError, subsequent retryable (non-overload) errors can continue to use the increased retry budget (per the backpressure spec intent referenced in PYTHON-5794/DRIVERS-3446).

Changes:

  • Add new prose tests for “overload then non-overload” sequences for retryable reads and writes (sync + async).
  • Update _ClientConnectionRetryable retry eligibility to track a per-operation _max_retries budget that expands when an overload error is observed.
  • Add helper plumbing in retryable reads tests to synchronously reconfigure fail points from a CommandFailedEvent callback.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
test/test_retryable_writes.py Adds a new prose test asserting increased write retries after overload then non-overload errors.
test/test_retryable_reads.py Adds setup client + helper to reconfigure fail points from events, and a new prose test for reads.
test/asynchronous/test_retryable_writes.py Async equivalent of the new write prose test.
test/asynchronous/test_retryable_reads.py Async equivalent of the reads helper + new prose test.
pymongo/synchronous/mongo_client.py Adjusts retry logic to expand retry budget after overload errors and enforce via _max_retries.
pymongo/asynchronous/mongo_client.py Async equivalent of the retry budget changes.

sleepyStick
sleepyStick previously approved these changes Apr 14, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.67%. Comparing base (e1751ff) to head (409c626).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2755      +/-   ##
==========================================
+ Coverage   87.64%   87.67%   +0.02%     
==========================================
  Files         141      141              
  Lines       24396    24404       +8     
  Branches     4172     4176       +4     
==========================================
+ Hits        21383    21396      +13     
+ Misses       2115     2110       -5     
  Partials      898      898              
Flag Coverage Δ
auth-aws-rhel8-test-auth-aws-rapid-web-identity-python3.14-cov 35.01% <16.66%> (-0.02%) ⬇️
auth-aws-win64-test-auth-aws-rapid-web-identity-python3.14-cov 35.01% <16.66%> (-0.04%) ⬇️
auth-enterprise-macos-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.75% <33.33%> (-0.02%) ⬇️
auth-enterprise-rhel8-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.75% <33.33%> (-0.01%) ⬇️
auth-enterprise-win64-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.76% <33.33%> (-0.02%) ⬇️
auth-oidc-local-ubuntu-22-test-auth-oidc-default 48.61% <33.33%> (-0.02%) ⬇️
compression-snappy-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.32% <58.33%> (+4.23%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.49% <58.33%> (+0.09%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.06% <58.33%> (+4.53%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.99% <58.33%> (+4.24%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.32% <58.33%> (+4.24%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.48% <58.33%> (+0.09%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.06% <58.33%> (+4.53%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.98% <58.33%> (+4.23%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.32% <58.33%> (+4.24%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.48% <58.33%> (+0.09%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.06% <58.33%> (+4.53%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.98% <58.33%> (+4.23%) ⬆️
compression-zstd-ubuntu-22-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.96% <58.33%> (+4.23%) ⬆️
coverage-report-coverage-report 85.88% <100.00%> (+3.85%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.34% <58.33%> (+4.25%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.48% <58.33%> (+0.09%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.06% <58.33%> (+4.53%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.00% <58.33%> (+4.25%) ⬆️
encryption-crypt_shared-macos-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.71% <33.33%> (-0.03%) ⬇️
encryption-crypt_shared-macos-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov ?
encryption-crypt_shared-macos-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.39% <33.33%> (+<0.01%) ⬆️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.72% <33.33%> (-0.02%) ⬇️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.46% <33.33%> (-0.02%) ⬇️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.39% <33.33%> (-0.03%) ⬇️
encryption-crypt_shared-win64-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.61% <33.33%> (-0.07%) ⬇️
encryption-crypt_shared-win64-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.59% <33.33%> (+0.06%) ⬆️
encryption-crypt_shared-win64-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.48% <33.33%> (+<0.01%) ⬆️
encryption-macos-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.72% <33.33%> (-0.02%) ⬇️
encryption-macos-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov ?
encryption-macos-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.40% <33.33%> (-0.04%) ⬇️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.38% <33.33%> (-0.03%) ⬇️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 55.15% <33.33%> (-0.07%) ⬇️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 55.08% <33.33%> (-0.04%) ⬇️
encryption-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.72% <33.33%> (-0.03%) ⬇️
encryption-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.47% <33.33%> (-0.02%) ⬇️
encryption-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.39% <33.33%> (-0.02%) ⬇️
encryption-win64-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.61% <33.33%> (-0.02%) ⬇️
encryption-win64-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.58% <33.33%> (+0.05%) ⬆️
encryption-win64-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.48% <33.33%> (+0.01%) ⬆️
load-balancer-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 48.39% <66.66%> (+<0.01%) ⬆️
mongodb-latest-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.44% <58.33%> (+4.50%) ⬆️
mongodb-latest-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.31% <58.33%> (+4.19%) ⬆️
mongodb-latest-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.11% <50.00%> (-0.01%) ⬇️
mongodb-latest-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.21% <50.00%> (-0.01%) ⬇️
mongodb-latest-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.38% <58.33%> (+0.08%) ⬆️
mongodb-rapid-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.45% <58.33%> (+4.49%) ⬆️
mongodb-rapid-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.33% <58.33%> (+4.18%) ⬆️
mongodb-rapid-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.11% <50.00%> (-0.01%) ⬇️
mongodb-rapid-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.21% <50.00%> (-0.01%) ⬇️
mongodb-rapid-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.39% <58.33%> (+0.10%) ⬆️
mongodb-v4.2-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 57.17% <16.66%> (+2.65%) ⬆️
mongodb-v4.2-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 55.61% <16.66%> (+2.59%) ⬆️
mongodb-v4.2-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 56.67% <16.66%> (-0.02%) ⬇️
mongodb-v4.2-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 56.78% <16.66%> (-0.02%) ⬇️
mongodb-v4.2-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 57.33% <16.66%> (+2.65%) ⬆️
mongodb-v4.4-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.52% <58.33%> (+4.48%) ⬆️
mongodb-v4.4-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.56% <58.33%> (+4.21%) ⬆️
mongodb-v4.4-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.18% <50.00%> (-0.01%) ⬇️
mongodb-v4.4-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.26% <50.00%> (-0.01%) ⬇️
mongodb-v4.4-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.33% <58.33%> (+4.25%) ⬆️
mongodb-v5.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.72% <58.33%> (+4.49%) ⬆️
mongodb-v5.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.71% <58.33%> (+4.20%) ⬆️
mongodb-v5.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.37% <50.00%> (-0.01%) ⬇️
mongodb-v5.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.50% <50.00%> (-0.01%) ⬇️
mongodb-v5.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.56% <58.33%> (+4.23%) ⬆️
mongodb-v6.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.74% <58.33%> (+4.51%) ⬆️
mongodb-v6.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.72% <58.33%> (+4.21%) ⬆️
mongodb-v6.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.39% <50.00%> (-0.01%) ⬇️
mongodb-v6.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.56% <50.00%> (+<0.01%) ⬆️
mongodb-v6.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.71% <58.33%> (+0.09%) ⬆️
mongodb-v7.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.77% <58.33%> (+4.49%) ⬆️
mongodb-v7.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.71% <58.33%> (+4.20%) ⬆️
mongodb-v7.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.45% <50.00%> (-0.01%) ⬇️
mongodb-v7.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.55% <50.00%> (-0.01%) ⬇️
mongodb-v7.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.73% <58.33%> (+0.13%) ⬆️
mongodb-v8.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.45% <58.33%> (+4.51%) ⬆️
mongodb-v8.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.32% <58.33%> (+4.20%) ⬆️
mongodb-v8.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.11% <50.00%> (-0.01%) ⬇️
mongodb-v8.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.23% <50.00%> (+<0.01%) ⬆️
mongodb-v8.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.39% <58.33%> (+0.09%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 60.54% <58.33%> (+4.24%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 62.69% <58.33%> (+0.09%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 62.26% <58.33%> (+4.52%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 60.18% <58.33%> (+4.23%) ⬆️
ocsp-rhel8-test-ocsp-ecdsa-valid-cert-server-staples-latest-python3.14-cov 34.08% <8.33%> (?)
ocsp-rhel8-test-ocsp-rsa-valid-cert-server-staples-latest-python3.14-cov 34.08% <8.33%> (?)
pyopenssl-macos-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.76% <58.33%> (+0.11%) ⬆️
pyopenssl-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.73% <58.33%> (+0.09%) ⬆️
pyopenssl-win64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.65% <58.33%> (+0.05%) ⬆️
stable-api-accept-v2-rhel8-auth-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.33% <58.33%> (+4.24%) ⬆️
stable-api-accept-v2-rhel8-auth-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.98% <58.33%> (+4.23%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.28% <58.33%> (+4.20%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 60.92% <58.33%> (+4.51%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.95% <58.33%> (+4.22%) ⬆️
storage-inmemory-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.34% <58.33%> (+4.25%) ⬆️
storage-inmemory-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.98% <58.33%> (+4.23%) ⬆️
test-macos-arm64-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.31% <58.33%> (+4.24%) ⬆️
test-macos-arm64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.47% <58.33%> (+0.09%) ⬆️
test-macos-arm64-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.06% <58.33%> (+4.53%) ⬆️
test-macos-arm64-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.97% <58.33%> (+4.25%) ⬆️
test-macos-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.30% <58.33%> (+4.22%) ⬆️
test-macos-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.46% <58.33%> (+0.07%) ⬆️
test-macos-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov ?
test-macos-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.97% <58.33%> (+4.25%) ⬆️
test-numpy-macos-arm64-test-numpy-python3.14-python3.14-cov 32.59% <8.33%> (-0.05%) ⬇️
test-numpy-macos-test-numpy-python3.14-python3.14-cov 32.61% <8.33%> (-0.02%) ⬇️
test-numpy-rhel8-test-numpy-python3.14-python3.14-cov 32.61% <8.33%> (-0.02%) ⬇️
test-numpy-win32-test-numpy-python3.14-python3.14-cov 32.59% <8.33%> (-0.01%) ⬇️
test-numpy-win64-test-numpy-python3.14-python3.14-cov 32.59% <8.33%> (+<0.01%) ⬆️
test-win32-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.20% <58.33%> (+4.27%) ⬆️
test-win32-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.43% <58.33%> (+0.07%) ⬆️
test-win32-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.00% <58.33%> (+4.53%) ⬆️
test-win32-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.85% <58.33%> (+4.26%) ⬆️
test-win64-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.20% <58.33%> (+4.27%) ⬆️
test-win64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.42% <58.33%> (+0.06%) ⬆️
test-win64-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.04% <58.33%> (+4.57%) ⬆️
test-win64-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.85% <58.33%> (+4.27%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
NoahStapp and others added 6 commits April 14, 2026 13:42
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants