Katkat
Back

Investigate High Memory Usage

DoorDashSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question evaluates incident-response and operational debugging skills, focusing on diagnosing high memory usage in a distributed microservice architecture that includes caching and third-party integrations.

You are the on-call engineer for a delivery platform.

System context

  • Couriers use a mobile app to accept and complete deliveries.
  • The mobile app calls a public gateway service (Dasher Service), which then calls a Payment Card Integration Service.
  • For some merchants, the courier must pay in person using a prepaid debit card.
  • That card is funded programmatically during checkout through a third-party payment card provider.
  • The integration service also relies on Redis for card and account information caching.
  • The company is in the middle of migrating from a monolith to microservices.

High-level flow: Courier App -> Dasher Service -> Payment Card Integration Service -> Third-Party Card Provider

Payment Card Integration Service <-> Redis cache

Incident

It is 4:30 PM Pacific, during a busy period, and you are paged because the Payment Card Integration Service is showing much higher than expected memory utilization.

Explain how you would handle this on-call investigation. Your answer should cover:

  1. How you would assess severity and business impact.
  2. What metrics, dashboards, and logs you would check first.
  3. The most likely causes of high memory usage in this architecture.
  4. How you would determine whether the issue is caused by traffic, a recent deploy, Redis behavior, retries, or the third-party provider.
  5. Immediate mitigation steps you would consider.
  6. How you would communicate during the incident.
  7. What long-term fixes or follow-up actions you would propose after recovery.