Katkat
Back

Design Robust Large-Input Processing

AttentiveSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question evaluates debugging and root-cause analysis, memory management, streaming/chunked data processing (including correctness across chunk boundaries), and system reliability aspects such as observability, testing, and recovery for large-input backends.

The string-reduction algorithm above is now part of a backend service.

Each job reads a text object from S3-compatible storage, applies the transformation, and writes the result back. A worker has 1 GB of memory and a 10-minute timeout, but some input objects are several GB in size. In production, some jobs crash.

Describe how you would:

  1. Triage and diagnose the crash.
  2. Determine whether the failure is caused by malformed input, an application bug, memory pressure, timeout, or a dependency issue.
  3. Redesign the service so it can handle very large inputs without first loading the entire object into memory.
  4. Preserve correctness when character runs span chunk boundaries.
  5. Add observability, testing, and recovery mechanisms so similar failures are easier to detect and fix in the future.