Katkat
Back

Troubleshoot a single-node web outage

MetaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests your ability to diagnose a sudden service outage on a single-node web server and your ability to propose architectural changes to prevent future single points of failure.

Scenario

You are operating a single-node web server running a monolithic e-commerce application. Users are reporting that the site is completely unresponsive and returning 502 Bad Gateway errors. You have SSH access to the server. Walk through your diagnostic process to identify the root cause and explain how you would resolve the immediate issue. Finally, propose a more resilient architecture to ensure this does not happen again.

Your Task

  1. Troubleshooting Steps: Detail the commands or log files you would inspect to pinpoint the bottleneck (e.g., CPU, Memory, Disk, Network, or Application logic).
  2. Immediate Resolution: Suggest how you might restore service if the root cause is a resource exhaustion issue versus a process crash.
  3. Architectural Improvements: Design a high-level system architecture that removes this single node as a single point of failure (SPOF) while maintaining low latency.

Constraints

  • Assume the application uses a standard stack: Nginx (reverse proxy) and a Python/Gunicorn web application running on a single Linux virtual machine.
  • The database is currently hosted on the same node.
  • Focus on scalability and fault tolerance in your redesign.

Hints

Hint 1

Start by checking high-level resource utilization tools like 'top', 'htop', or 'iostat' to see if the system is starved for CPU or RAM.

Hint 2

Examine Nginx error logs and application-specific logs to determine if the 502 error is caused by the backend application process crashing or hanging.

Hint 3

Consider decoupling the database and implementing a load balancer to distribute traffic across multiple nodes.

Follow-up questions

  • How would you implement automated health checks to ensure that traffic is not routed to a node that has become unresponsive?
  • If the database is the bottleneck, what specific strategies would you use to scale it (e.g., read replicas, sharding, caching)?
  • How would you monitor this system to get alerted before a full outage occurs?