Katkat
Back

Review an existing architecture for risks

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to analyze a distributed system architecture for single points of failure, scalability bottlenecks, and operational risks, while balancing engineering trade-offs.

Scenario

Imagine you are the primary engineer overseeing a 'Flash Sales' service that has been running for two years. The system currently handles 1,000 requests per second (RPS) but is slated for a major marketing event that expects 50,000 RPS.

Current Architecture:

  • Frontend: A monolithic web server running on three EC2 instances behind an Application Load Balancer.
  • Database: A single primary RDS (MySQL) instance with no read replicas.
  • Storage: Product images are served directly from the application server’s local disk.
  • Session Management: Stored in the application memory (in-memory hash maps).
  • Monitoring: Basic CPU/RAM alerts sent via email.

Your Task:

Identify the top three risks in this architecture that would cause the system to fail under the new traffic requirements. For each risk, propose a mitigation strategy. Finally, describe how you would prioritize these changes given that you only have three engineers and four weeks before the event.

Constraints & Requirements

  • You must address availability, data consistency, and latency.
  • Justify why you chose your top three risks over others.
  • Consider the 'blast radius' of your proposed changes.

Hints

Hint 1

Consider what happens to the user experience if one of the three application servers crashes or if the database is overwhelmed by read requests.

Hint 2

Evaluate the bottlenecks introduced by storing state (sessions) locally and serving static assets from the application server disk.

Hint 3

Think about the difference between 'quick wins' that improve stability and 'architectural refactors' that require significant testing and code changes.

Follow-up questions

  • How would you implement a 'Circuit Breaker' pattern to protect the database if it starts showing signs of high latency?
  • If the database becomes a bottleneck for writes specifically, what sharding or partitioning strategy would you suggest, and what are the trade-offs?
  • How would you ensure that your changes do not introduce new regressions during the four-week period?