Katkat

Implement a Resilient Aggregator over Three Services

DoorDashSoftware Engineer · Onsite

Last updated

Implement an asynchronous aggregation service that orchestrates calls to User, Payment, and Address downstream clients with dependency-aware concurrency, partial failure handling, and deadline enforcement.

Problem Statement

When a customer navigates to the checkout screen, the backend must construct a composite CheckoutContext by aggregating data from three separate downstream microservices:

  1. UserService: get_user(user_id: str) -> UserProfile
    • Returns UserProfile(user_id: str, default_address_id: str, payment_customer_id: str).
    • Critical dependency: The other two services depend on IDs returned by this call.
  2. AddressService: get_address(address_id: str) -> Address
    • Returns Address(address_id: str, formatted_address: str).
    • Non-critical: If this service fails or times out, the context should return address = None and record an error message without failing the entire checkout view.
  3. PaymentService: get_payment_methods(customer_id: str) -> list[PaymentMethod]
    • Returns a list of PaymentMethod(method_id: str, card_type: str, last_four: str).
    • Semi-critical: Should retry once on network error (ClientNetworkError) before failing. If it still fails, the aggregator returns payment_methods = [] and records an error.

Implement the class CheckoutAggregator and its primary method:

class CheckoutAggregator:
    def __init__(
        self,
        user_service: UserClient,
        address_service: AddressClient,
        payment_service: PaymentClient,
    ):
        ...

    async def get_checkout_context(
        self, user_id: str, timeout_seconds: float = 1.0
    ) -> CheckoutContext:
        ...

Return Type

@dataclass
class CheckoutContext:
    user: UserProfile
    address: Optional[Address]
    payment_methods: list[PaymentMethod]
    errors: list[str]  # Non-fatal errors encountered during partial failure

Requirements

  1. Dependency Execution: AddressService and PaymentService must run concurrently as soon as UserService finishes.
  2. Critical Failure: If UserService raises an exception or times out, propagate a UserFetchError immediately and do not invoke downstream services.
  3. Deadlines & Cancellation: If the overall execution exceeds timeout_seconds, any running downstream tasks must be promptly cancelled, and a TimeoutError should be raised if UserService was not resolved, or partial results returned if acceptable according to policy.
  4. Retries: Retry PaymentService at most once on transient network errors before falling back to empty results.

Example

# User found, address succeeds, payment fails transiently then succeeds
context = await aggregator.get_checkout_context(user_id="usr_123", timeout_seconds=0.5)
assert context.user.user_id == "usr_123"
assert context.address.formatted_address == "900 Market St, SF"
assert len(context.payment_methods) == 1
assert context.errors == []

Hints

Hint 1

Identify the dependency boundary: `UserService` must resolve first, whereas `AddressService` and `PaymentService` can be executed concurrently using `asyncio.gather` with `return_exceptions=True`.

Hint 2

Enforce the overall deadline using `asyncio.wait_for` around the composite aggregation or separate deadlines per stage to ensure in-flight coroutines get cancelled.

Hint 3

Implement an isolated retry wrapper for the payment client that intercepts `ClientNetworkError` and attempts exactly one retry before returning an exception or fallback.

Follow-up questions

  • How would you integrate a Circuit Breaker pattern to avoid hammering `PaymentService` when it enters a sustained outage?
  • If `AddressService` and `PaymentService` have disparate latencies (e.g., 50ms vs 400ms), how would you stream partial results over gRPC or Server-Sent Events (SSE) to the frontend?
  • How would you ensure distributed tracing context (e.g., OpenTelemetry traceparent headers) is propagated across the concurrent downstream async calls?