Katkat
Back

Prepare for an SRE Systems Fundamentals Round

ByteDanceSoftware Engineer · Onsite
EasySoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question tests your understanding of Linux process management, memory utilization, and the practical troubleshooting steps required to handle a production service experiencing memory exhaustion.

The OOM Killer Scenario

You are an SRE at a company running a high-throughput microservice in a Linux-based container environment. You receive an alert that the service has been terminated by the kernel's Out-Of-Memory (OOM) Killer. The service is responsible for processing large JSON payloads and uses a custom caching layer to speed up lookups.

Your Task

Describe your step-by-step diagnostic process to identify the root cause of this failure. Your answer should address the following:

  1. Which Linux system logs or command-line tools would you examine to confirm the OOM event and identify which process was targeted?
  2. If the service is memory-bound due to the caching layer, what strategies would you propose to mitigate the OOM risk without simply increasing the total RAM allocated to the container?
  3. Explain the difference between 'Resident Set Size' (RSS) and 'Virtual Memory Size' (VSZ) in the context of debugging this memory spike.

Constraints

  • Focus on the Linux environment and the specific mechanics of memory management.
  • Do not assume external monitoring tools like Prometheus or Datadog are functioning; focus on host-level investigation first.
  • Consider both application-level configuration and OS-level limits (e.g., cgroups/ulimit).

Hints

Hint 1

Check system-wide logs like /var/log/syslog, /var/log/messages, or the output of the 'dmesg' command to find specific OOM Killer trigger events.

Hint 2

Consider implementing or tuning memory eviction policies, such as Least Recently Used (LRU) caches, or exploring memory-mapped files for large data sets.

Hint 3

Remember that RSS represents the memory actually in RAM, while VSZ represents the total address space requested by the process, which can be inflated by shared libraries or memory-mapped regions.

Follow-up questions

  • How would you utilize 'cgroups' to proactively prevent a single container from starving the host node of memory?
  • If the OOM event occurs during high traffic bursts, how would you design a load-shedding mechanism to protect the process?
  • What are the trade-offs between swapping memory to disk versus letting the OOM Killer terminate a service process?