Handle API Errors with Python Requests

Resilient API Client

In vehicle telemetry and fleet management services, backend microservices frequently query external and internal REST APIs. Network conditions may be unstable, upstream services may throttle calls, and responses may occasionally return invalid payloads.

Implement a function robust_get that wraps requests.get with comprehensive error handling and retry logic.

Function Signature

import requests
from typing import Any, Dict, Optional

def robust_get(
    url: str,
    max_retries: int = 3,
    base_backoff: float = 0.5,
    timeout: float = 5.0,
    session: Optional[requests.Session] = None
) -> Dict[str, Any]:
    pass

Requirements

1. Transient Errors & Retries:

   • Transient HTTP status codes: 500, 502, 503, and 504.
   • Network-level errors: requests.exceptions.Timeout and requests.exceptions.ConnectionError.
   • Retry up to max_retries times when encountering transient errors. Use exponential backoff: sleep for base_backoff * (2 ** attempt) seconds before attempt attempt (0-indexed).

2. Rate Limiting (HTTP 429):

   • When receiving a 429 Too Many Requests status, treat it as retryable up to max_retries.
   • If the response includes a Retry-After header with an integer value, sleep for that number of seconds. If the header is missing or unparseable, fall back to the exponential backoff calculation.

3. Non-Retryable Errors:

   • Client errors (4xx status codes other than 429) must abort immediately without retrying.

4. JSON Parsing:

   • On a successful response (2xx), attempt to parse the body using .json().
   • If requests.exceptions.JSONDecodeError (or ValueError) occurs, do not retry; return an error response indicating invalid JSON.

5. Return Format: Return a dictionary containing:

   • "success": bool (True if valid 2xx status and parsed JSON, False otherwise)
   • "data": Any (parsed JSON on success, None on failure)
   • "status_code": Optional[int] (HTTP status code, or None if a network connection error prevented receiving a response)
   • "error": Optional[str] (a descriptive string if success is False, otherwise None)

Example 1

# Remote endpoint returns 503 twice, then 200 with valid JSON {"status": "ok"}
result = robust_get("https://api.example.com/telemetry", max_retries=3)
# Result:
{
    "success": True,
    "data": {"status": "ok"},
    "status_code": 200,
    "error": None
}

Example 2

# Remote endpoint returns 404 Not Found
result = robust_get("https://api.example.com/unknown", max_retries=3)
# Result:
{
    "success": False,
    "data": None,
    "status_code": 404,
    "error": "Client error: 404"
}