Katkat
Back

Design a Concurrent Image Processing Service

AnthropicSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient, distributed image-processing pipeline that safely scales from a single worker to hundreds of concurrent nodes while guaranteeing job lease safety, idempotent output writes, and clean cancellation.

Scenario

You are tasked with designing the core backend service for an asynchronous image processing platform. The service ingests large raw image assets and applies multi-step transformations (e.g., resizing, format transcode, background removal, and embedding generation).

Initially built as a single-process worker, the system must now scale out to hundreds of stateless, concurrent worker nodes. The design must guarantee that jobs are processed safely without duplicate execution side effects, partial file corruption, or phantom writes from stalled workers.

Requirements

Functional Requirements

  • Job Submission & Tracking: Clients submit processing requests containing source image URLs and desired transformation specs. Clients can poll or stream job status.
  • Job Cancellation: Clients can cancel in-flight jobs. Workers must halt execution promptly without publishing unfinished artifacts.
  • Worker Coordination: Safe distribution of work among concurrent worker nodes with dynamic leasing and ownership handover if a node dies.
  • Atomic Publication: Downstream consumers must never observe partially written images or corrupt metadata.

Non-Functional Requirements

  • Scale: Ingest up to 50,000 images per minute at peak. Image sizes range from 2 MB to 50 MB.
  • Latency & Execution Time: Job processing takes between 1 second and 60 seconds depending on the transformation complexity.
  • Fault Tolerance & Idempotency: Worker crashes, network partitions, and duplicate message deliveries must not lead to data corruption or duplicate billable operations.
  • Backpressure: The system must protect worker clusters and upstream storage from being overwhelmed during traffic spikes.

What Your Design Should Cover

  1. Job Lifecycle & State Machine: Define exact states (e.g., PENDING, ACQUIRED, PROCESSING, SUCCEEDED, FAILED, CANCELLED) and legal transitions.
  2. Leasing, Fencing, and Ownership Protocol: How workers claim work, maintain active leases, detect dead workers, and prevent "zombie" workers from overwriting fresher state.
  3. Idempotent Artifact Storage: The strategy for writing intermediate and final artifacts to object storage to avoid partial result exposure.
  4. Cancellation & Backpressure Mechanics: Mechanism for propagating cancellation signals to executing worker threads, and rate-limiting/shedding load when queues saturate.
  5. Data Model & High-Level Architecture: Schema for job tracking and the interaction diagram between the API gateway, coordinator/queue, workers, and blob store.

Hints

Hint 1

Use monotonic fencing tokens or lease epochs so the database and storage layers reject stale writes from paused workers.

Hint 2

Ensure artifact publishing is atomic by writing to isolated staging paths (e.g., with unique job attempt UUIDs) and performing a single metadata commit or atomic rename/manifest update.

Hint 3

Differentiate between cooperative cancellation via worker polling and forced lease expiration when deciding how to reclaim resources.

Follow-up questions

  • How does your leasing protocol prevent split-brain issues during an unexpected GC pause or network partition on a worker?
  • If an image pipeline consists of a DAG of multiple dependent tasks, how do your state machine and intermediate storage adapt to enable partial retries?
  • How would you implement priority queues so interactive user requests bypass bulk batch backfills without starving either pool?