Katkat
Back

Troubleshoot a website outage with disk full

MetaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your systematic approach to diagnosing a production service outage caused by resource exhaustion and your ability to implement short-term and long-term mitigation strategies.

Scenario

You are an on-call engineer at a social media company. Your dashboard alerts you that the 'User Metadata Service' is returning 500 Internal Server Error codes for 100% of incoming traffic. This service runs on a cluster of Linux-based virtual machines behind a load balancer.

Upon checking the logs for one of the failing instances, you see the following error message repeatedly: IOError: [Errno 28] No space left on device

Your Task

Describe your step-by-step process to investigate and resolve this incident. Specifically, address the following:

  1. Immediate Triage: How would you verify the disk usage and identify the root cause of the capacity exhaustion?
  2. Mitigation: What steps would you take to restore service availability quickly without losing critical data?
  3. Root Cause Analysis: How would you investigate what created the large volume of data to prevent recurrence?
  4. Prevention: What architectural or monitoring improvements would you propose to prevent a similar outage in the future?

Constraints & Considerations

  • Assume the service uses local persistent storage for temporary logs and internal caches.
  • Consider the trade-offs between clearing files versus scaling resources.
  • Explain how you would safely inspect the file system without potentially exacerbating the outage.

Hints

Hint 1

Start by identifying which mount point is full using standard command-line utilities like 'df' and 'du'.

Hint 2

Distinguish between temporary log buildup and unexpected data ingestion, and consider the implications of deleting open file handles.

Hint 3

Think about how automated log rotation or monitoring thresholds could have prevented the system from reaching zero capacity.

Follow-up questions

  • What would you do if the disk was full because of a massive surge in legitimate user-generated traffic that the service is required to store?
  • How does the behavior of a 'deleted' file with an active process handle differ from a file that is truly removed from the disk?
  • If this service were moved to a containerized environment like Kubernetes, how would your troubleshooting steps change?