Katkat
Back

How would you scale batch image pipelines?

AnthropicSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This problem evaluates your ability to design a distributed, fault-tolerant batch processing system for high-throughput image transformation tasks.

Scenario

You are tasked with building a system that processes a batch of 1 million high-resolution images stored in an object store (e.g., S3). The processing involves resizing, filtering, and running a lightweight machine learning model to generate metadata. The final images and metadata must be saved back to storage, and the process needs to complete within 4 hours.

Requirements

  • Functional: Support different processing workflows, allow for retries of individual failed image tasks, and ensure data integrity.
  • Non-functional: The system must be cost-efficient, observable (monitoring progress and errors), and capable of handling varying image sizes without crashing the worker nodes.
  • Assumptions: You have access to standard cloud-native building blocks (queues, object storage, serverless functions, or container orchestration).

Task

Design the architecture for this pipeline. Please address:

  1. How will you distribute the workload efficiently without overloading the system?
  2. What strategy will you use to handle intermittent failures or images that cause worker crashes?
  3. How would you handle backpressure if the downstream database for metadata writes becomes a bottleneck?
  4. How do you track the progress and status of the 1 million images?

Hints

Hint 1

Consider using a producer-consumer pattern where an orchestrator breaks the batch into smaller units of work pushed to a durable message queue.

Hint 2

Think about using a mix of ephemeral storage for processing and persistent storage for output, while implementing a 'dead-letter queue' for poisoned images that cause crashes.

Hint 3

Address the throughput mismatch by introducing a batching layer or a buffer before the database to aggregate metadata writes.

Follow-up questions

  • How would your architecture change if we needed to prioritize certain image batches over others (e.g., VIP user uploads)?
  • If the ML model is too large to fit in a standard container memory, how would you optimize the infrastructure?
  • How would you ensure cost-efficiency if the workload is highly sporadic but must finish within a strict deadline?