Query Machines and Mark Them Offline

Problem Statement

You are tasked with writing a Python function mark_offline(api_client, cluster_id) to manage nodes in a remote infrastructure cluster. Your goal is to identify all nodes that are currently in an 'active' state but have a 'maintenance_mode' flag set to 'pending', and transition them to an 'offline' state.

Requirements

1. Fetch the list of nodes for the given cluster_id from api_client.get_nodes(cluster_id).
2. Filter for nodes where status == 'active' and maintenance_mode == 'pending'.
3. For each target node, call api_client.update_node_status(node_id, 'offline').
4. Resiliency: Implement a retry mechanism. If an API call fails or returns a 5xx error, retry up to 3 times with exponential backoff.
5. Error Handling: If a node record is malformed (missing keys), skip it and log the error.

Interface

# Provided API Client interface
class APIClient:
    def get_nodes(self, cluster_id: str) -> list: # Returns list of dicts
        pass
    def update_node_status(self, node_id: str, status: str) -> bool: # Returns success
        pass

Examples

Input: Nodes: [{'id': 'n1', 'status': 'active', 'maintenance_mode': 'pending'}, {'id': 'n2', 'status': 'error', 'maintenance_mode': 'none'}]

Output: update_node_status('n1', 'offline') is called. n2 is ignored.

Edge Cases

• API client returns an empty list.
• The API returns 403 Forbidden or 404 Not Found (should not be retried).
• Large node counts exceeding API pagination (assume get_nodes returns all nodes for this exercise).