Katkat

Implement a Parallel Image Processor

AnthropicSoftware Engineer · Technical Screen

Last updated

Design and implement a concurrent batch image processor that executes a sequence of matrix transforms across worker threads or processes while isolating per-job failures.

Problem Statement

You are tasked with building a batch image processing engine that applies a series of transformations to 2D grayscale image grids. In production, transformations can be computationally expensive and individual operations may occasionally throw errors (e.g., due to corrupt input or division by zero).

Implement the class ParallelImageProcessor that processes a list of image jobs concurrently up to a specified concurrency limit. The processor must isolate failures: an unhandled exception or validation failure in one job must not terminate or corrupt other jobs in the batch.

Requirements

  1. Concurrency: Use a worker pool of size max_workers to process jobs concurrently.
  2. Fault Isolation: If a transform raises an exception or an input image is invalid (e.g., ragged matrix, empty dimensions, pixel values outside [0, 255]), catch the error, mark the job as failed, and record the error message. Do not fail the whole batch.
  3. Preserve Order: The returned list of JobResult objects must match the exact order of the input jobs.
  4. Immutability: Transformations should not mutate the original input image in-place.

Interface

from dataclasses import dataclass
from typing import Callable, List, Optional

@dataclass
class ImageJob:
    job_id: str
    image: List[List[int]]
    transforms: List[Callable[[List[List[int]]], List[List[int]]]]

@dataclass
class JobResult:
    job_id: str
    success: bool
    image: Optional[List[List[int]]]
    error: Optional[str]

class ParallelImageProcessor:
    def __init__(self, max_workers: int):
        pass

    def process_batch(self, jobs: List[ImageJob]) -> List[JobResult]:
        pass

Constraints

  • 1 <= max_workers <= 32
  • 0 <= len(jobs) <= 1,000
  • Each valid image is an H x W matrix where 1 <= H, W <= 1,000.
  • Valid pixel values are integers in the range [0, 255].

Example

def invert(img):
    return [[255 - p for p in row] for row in img]

def bad_transform(img):
    raise ValueError("Invalid operation")

jobs = [
    ImageJob(
        job_id="job-1",
        image=[[0, 255], [100, 200]],
        transforms=[invert]
    ),
    ImageJob(
        job_id="job-2",
        image=[[50, 50]],
        transforms=[bad_transform]
    ),
]

processor = ParallelImageProcessor(max_workers=2)
results = processor.process_batch(jobs)

# Output:
# [
#   JobResult(job_id="job-1", success=True, image=[[255, 0], [155, 55]], error=None),
#   JobResult(job_id="job-2", success=False, image=None, error="Invalid operation"),
# ]

Hints

Hint 1

Use `concurrent.futures.ThreadPoolExecutor` or `ProcessPoolExecutor` to distribute jobs across workers while retaining future handles to restore input ordering.

Hint 2

Validate the image grid structure (non-empty rows, rectangular dimensions, value bounds [0, 255]) before applying transforms.

Hint 3

Wrap the per-job pipeline execution inside a try/except block to intercept both validation errors and runtime exceptions thrown by transforms.

Follow-up questions

  • When should you choose `ProcessPoolExecutor` over `ThreadPoolExecutor` for CPU-heavy image transforms in Python given the Global Interpreter Lock (GIL)?
  • How would you implement a per-job timeout so that an infinite loop in a custom transform does not block a worker indefinitely?
  • If input images were several gigabytes each, how would you redesign the system to avoid high serialization and inter-process communication overhead?