Katkat
Back

Reason About Unix Signals, Zombies, and Process Reaping

Hudson River TradingSite Reliability Engineer · Technical Screen
MediumSite Reliability EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

Assess understanding of Unix process lifecycles, signal handling mechanics, zombie state accumulation, and process reaping responsibilities in parent processes and PID 1.

Scenario

During a trading session, an SRE notices an alert indicating that a containerized market data adapter host is rapidly consuming process IDs (PIDs). Inspecting the container with ps -ef reveals hundreds of processes in a `` state:

UID        PID  PPID  C STIME TTY          TIME CMD
app          1     0  0 08:00 ?        00:00:02 /usr/bin/adapter-manager
app        102     1  0 08:01 ?        00:00:00 [market-worker] 
app        103     1  0 08:01 ?        00:00:00 [market-worker] 
app        104     1  0 08:02 ?        00:00:00 [market-worker] 
...

A junior engineer attempts to resolve the issue by running kill -9 102 inside the container, but the defunct process remains visible. They then propose issuing kill -9 1 to force the manager to restart.

Tasks

Provide a technical evaluation of the situation addressing the following four areas:

1. Signal Delivery and Interception

Explain how the Linux kernel delivers signals to a process. Contrast SIGTERM and SIGKILL, detailing:

  • How signal registration and signal masks (sigprocmask) apply.
  • Why kill -9 had no effect on PID 102.
  • Why issuing SIGKILL directly to PID 1 inside a container PID namespace may not terminate it, unlike on a standard Unix host.

2. Zombie Process Mechanics

Define what a zombie process actually is. Specifically explain:

  • What resources are freed immediately when a child process exits, and what specific kernel metadata remains allocated.
  • Why excessive zombie processes degrade the system even if they consume zero CPU and negligible resident memory.

3. Parent Process Responsibilities

Explain the mechanism by which parent processes properly reap child processes:

  • Outline the role of SIGCHLD and the wait() / waitpid() family of system calls.
  • Explain why a naive signal handler like the following can still leak zombie processes under high load: 
    void handle_sigchld(int sig) {
    wait(NULL);
}
  • Provide the correct implementation pattern using waitpid and explain its flags.

4. Container PID 1 Reaping Problem

Explain the "PID 1 zombie problem" common in Docker/OCI containers. Describe what occurs when a worker's parent process dies before the worker exits, and explain how init processes (such as tini, systemd, or modern container runtime --init flags) resolve this behavior.

Hints

Hint 1

A process enters the defunct state after terminating until its exit status is read by its parent via a system call.

Hint 2

Unix standard signals are not queued; multiple children terminating nearly simultaneously will coalesce into a single pending SIGCHLD signal.

Hint 3

In Linux containers, the process with PID 1 does not inherit default signal handlers from the kernel and must explicitly register handlers or be designed to handle child termination.

Follow-up questions

  • How does setting the signal action for `SIGCHLD` to `SIG_IGN` or specifying `SA_NOCLDWAIT` alter kernel behavior regarding child reaping?
  • If a container's PID 1 freezes or deadlocks without exiting, how can an operator from the host namespace inspect and clean up the leaked processes?
  • What is `prctl(PR_SET_CHILD_SUBREAPER, ...)` and how do modern service supervisors use it instead of delegating all orphaned process reaping to PID 1?