Troubleshoot a Host That Rejects SSH Connections

Scenario

You are an SRE on call when an automated alert flags that an internal trading infrastructure host prod-exec-04.infra is refusing SSH connections (Connection refused or Connection reset by peer). Out-of-band management (e.g., serial console or IPMI / BMC) is available, but remote shell access via SSH is completely down.

Your task is to outline a rigorous, layered troubleshooting plan to isolate the root cause. For each stage, specify the exact command or source of telemetry, the hypothesis being tested, and what the observation strictly proves versus what ambiguity remains.

Requirements

1. Client & Transport Layer (Outside-In)

   • Distinguish between connection failure modes: Connection timed out, Connection refused (TCP RST), and immediate teardown/reset after TCP handshake.
   • Detail client-side commands (e.g., ssh -vvv, nc/ncat, tcpdump, traceroute/mtr) and state what each failure signature rules out.

2. Host Network & Firewall Layer (Via Console)

   • Check interface state, routing tables, and packet filter rules (nftables, iptables, firewalld).
   • Differentiate between a packet drop (silent timeout) versus an active reject (ICMP port unreachable or TCP RST generated by REJECT rules).

3. Socket & Process Layer

   • Inspect listener sockets (ss -tulpn or lsof -i :22).
   • Verify whether sshd is running, bound to 0.0.0.0:22 / :::22 vs 127.0.0.1:22, or managed via a systemd socket unit (ssh.socket).
   • Check process state, resource limits (ulimit -n), and whether the process is hung in an uninterruptible sleep (D) state.

4. Authentication & Daemon Logging

   • Locate and interpret relevant daemon logs (journalctl -u ssh, /var/log/secure, /var/log/auth.log).
   • Account for common edge cases: TCP wrappers (/etc/hosts.allow, /etc/hosts.deny), PAM configuration errors, SELinux/AppArmor denials, or full disk / exhausted inode count preventing PID/privilege separation files from being written.