Implement a Resilient Aggregator over Three Services
DoorDashSoftware Engineer · OnsiteLast updated
Implement an asynchronous aggregation service that orchestrates calls to User, Payment, and Address downstream clients with dependency-aware concurrency, partial failure handling, and deadline enforcement.
Problem Statement
When a customer navigates to the checkout screen, the backend must construct a composite CheckoutContext by aggregating data from three separate downstream microservices:
UserService:get_user(user_id: str) -> UserProfile- Returns
UserProfile(user_id: str, default_address_id: str, payment_customer_id: str). - Critical dependency: The other two services depend on IDs returned by this call.
- Returns
AddressService:get_address(address_id: str) -> Address- Returns
Address(address_id: str, formatted_address: str). - Non-critical: If this service fails or times out, the context should return
address = Noneand record an error message without failing the entire checkout view.
- Returns
PaymentService:get_payment_methods(customer_id: str) -> list[PaymentMethod]- Returns a list of
PaymentMethod(method_id: str, card_type: str, last_four: str). - Semi-critical: Should retry once on network error (
ClientNetworkError) before failing. If it still fails, the aggregator returnspayment_methods = []and records an error.
- Returns a list of
Implement the class CheckoutAggregator and its primary method:
class CheckoutAggregator:
def __init__(
self,
user_service: UserClient,
address_service: AddressClient,
payment_service: PaymentClient,
):
...
async def get_checkout_context(
self, user_id: str, timeout_seconds: float = 1.0
) -> CheckoutContext:
...
Return Type
@dataclass
class CheckoutContext:
user: UserProfile
address: Optional[Address]
payment_methods: list[PaymentMethod]
errors: list[str] # Non-fatal errors encountered during partial failure
Requirements
- Dependency Execution:
AddressServiceandPaymentServicemust run concurrently as soon asUserServicefinishes. - Critical Failure: If
UserServiceraises an exception or times out, propagate aUserFetchErrorimmediately and do not invoke downstream services. - Deadlines & Cancellation: If the overall execution exceeds
timeout_seconds, any running downstream tasks must be promptly cancelled, and aTimeoutErrorshould be raised ifUserServicewas not resolved, or partial results returned if acceptable according to policy. - Retries: Retry
PaymentServiceat most once on transient network errors before falling back to empty results.
Example
# User found, address succeeds, payment fails transiently then succeeds
context = await aggregator.get_checkout_context(user_id="usr_123", timeout_seconds=0.5)
assert context.user.user_id == "usr_123"
assert context.address.formatted_address == "900 Market St, SF"
assert len(context.payment_methods) == 1
assert context.errors == []
Hints
Hint 1
Identify the dependency boundary: `UserService` must resolve first, whereas `AddressService` and `PaymentService` can be executed concurrently using `asyncio.gather` with `return_exceptions=True`.
Hint 2
Enforce the overall deadline using `asyncio.wait_for` around the composite aggregation or separate deadlines per stage to ensure in-flight coroutines get cancelled.
Hint 3
Implement an isolated retry wrapper for the payment client that intercepts `ClientNetworkError` and attempts exactly one retry before returning an exception or fallback.
Follow-up questions
- How would you integrate a Circuit Breaker pattern to avoid hammering `PaymentService` when it enters a sustained outage?
- If `AddressService` and `PaymentService` have disparate latencies (e.g., 50ms vs 400ms), how would you stream partial results over gRPC or Server-Sent Events (SSE) to the frontend?
- How would you ensure distributed tracing context (e.g., OpenTelemetry traceparent headers) is propagated across the concurrent downstream async calls?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.