How to Retry Failed Python Requests

---

Why retries?

Web scraping and API calls frequently fail due to transient network/server issues or rate limits. Instead of crashing or hammering an endpoint, retry with delays and backoff.

Quick install

pip install requests

Status codes to watch

• 403 Forbidden - Often needs credentials or you’re blocked; retries won’t help much.
• 429 Too Many Requests - You’re rate-limited; slow down, back off, or rotate proxies.
• 500 Internal Server Error - Server issue; retry soon.
• 502 Bad Gateway - Upstream server glitch; retry later.
• 503 Service Unavailable - Service down; retries eventually help, once fixed.
• 504 Gateway Timeout - Network timeout; backoff and try again.

---

Examples

Basic GET + print status

A minimal request using requests.get and printing the status code.

import requests

def send_get_request(URL):
    r = requests.get(URL)
    print(r.status_code)

send_get_request('https://iproyal.com')

403 with basic auth

Retrying a GET with credentials to handle restricted endpoints.

import requests

def send_get_request(URL, credentials):
    r = requests.get(URL, auth=credentials)
    print(r.status_code)

login_details = ('username', 'password')
send_get_request('https://iproyal.com', login_details)

Retry Failed Requests With a Loop

Simple retry loop with fixed delays between attempts.

import requests
import time

def send_get_request(URL, retry):
    for i in range(retry):
        try:
            r = requests.get(URL)
            if r.status_code not in [200, 404]:
                time.sleep(5)
            else:
                break
        except requests.exceptions.ConnectionError:
            pass
    print(r.status_code)

send_get_request('https://dashboard.iproyal.com/login', 5)

Retry Failed Requests With HTTPAdapter

Using HTTPAdapter and Retry from urllib3 for exponential backoff.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

def send_get_request(URL):
    sess = requests.session()

    retries = Retry(total = 5,
                    backoff_factor = 1,
                    status_forcelist = [429, 500, 502, 503, 504])

    sess.mount('https://', HTTPAdapter(max_retries=retries))
    get_URL = sess.get(URL)
    print(get_URL.status_code)

send_get_request('https://iproyal.com')

Escape 429 With Proxies

Switching to a different proxy when hitting HTTP 429 (Too Many Requests).

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

def send_get_request(URL):
    sess = requests.session()

    proxies = {"http" : "http://USER:PASS@HOST:PORT"}

    retries = Retry(total = 5,
                    backoff_factor = 1,
                    status_forcelist = [500, 502, 503, 504])

    sess.mount('https://', HTTPAdapter(max_retries=retries))
    get_url = sess.get(URL, proxies=proxies)
    if get_url.status_code == 429:
        sess.get(URL, proxies=proxies)
        
    print(get_url.status_code)

send_get_request('https://iproyal.com')

---

Notes

• For timeouts, add timeout=N to requests.get(...) if needed.
• The loop strategy can be faster but easier to detect; backoff is gentler.
• For sticky sessions, maintain a list of proxies and switch on 429.