Implement a Parallel Image Processor
AnthropicSoftware Engineer · Technical ScreenLast updated
Design and implement a concurrent batch image processor that executes a sequence of matrix transforms across worker threads or processes while isolating per-job failures.
Problem Statement
You are tasked with building a batch image processing engine that applies a series of transformations to 2D grayscale image grids. In production, transformations can be computationally expensive and individual operations may occasionally throw errors (e.g., due to corrupt input or division by zero).
Implement the class ParallelImageProcessor that processes a list of image jobs concurrently up to a specified concurrency limit. The processor must isolate failures: an unhandled exception or validation failure in one job must not terminate or corrupt other jobs in the batch.
Requirements
- Concurrency: Use a worker pool of size
max_workersto process jobs concurrently. - Fault Isolation: If a transform raises an exception or an input image is invalid (e.g., ragged matrix, empty dimensions, pixel values outside
[0, 255]), catch the error, mark the job as failed, and record the error message. Do not fail the whole batch. - Preserve Order: The returned list of
JobResultobjects must match the exact order of the inputjobs. - Immutability: Transformations should not mutate the original input image in-place.
Interface
from dataclasses import dataclass
from typing import Callable, List, Optional
@dataclass
class ImageJob:
job_id: str
image: List[List[int]]
transforms: List[Callable[[List[List[int]]], List[List[int]]]]
@dataclass
class JobResult:
job_id: str
success: bool
image: Optional[List[List[int]]]
error: Optional[str]
class ParallelImageProcessor:
def __init__(self, max_workers: int):
pass
def process_batch(self, jobs: List[ImageJob]) -> List[JobResult]:
pass
Constraints
1 <= max_workers <= 320 <= len(jobs) <= 1,000- Each valid image is an
H x Wmatrix where1 <= H, W <= 1,000. - Valid pixel values are integers in the range
[0, 255].
Example
def invert(img):
return [[255 - p for p in row] for row in img]
def bad_transform(img):
raise ValueError("Invalid operation")
jobs = [
ImageJob(
job_id="job-1",
image=[[0, 255], [100, 200]],
transforms=[invert]
),
ImageJob(
job_id="job-2",
image=[[50, 50]],
transforms=[bad_transform]
),
]
processor = ParallelImageProcessor(max_workers=2)
results = processor.process_batch(jobs)
# Output:
# [
# JobResult(job_id="job-1", success=True, image=[[255, 0], [155, 55]], error=None),
# JobResult(job_id="job-2", success=False, image=None, error="Invalid operation"),
# ]
Hints
Hint 1
Use `concurrent.futures.ThreadPoolExecutor` or `ProcessPoolExecutor` to distribute jobs across workers while retaining future handles to restore input ordering.
Hint 2
Validate the image grid structure (non-empty rows, rectangular dimensions, value bounds [0, 255]) before applying transforms.
Hint 3
Wrap the per-job pipeline execution inside a try/except block to intercept both validation errors and runtime exceptions thrown by transforms.
Follow-up questions
- When should you choose `ProcessPoolExecutor` over `ThreadPoolExecutor` for CPU-heavy image transforms in Python given the Global Interpreter Lock (GIL)?
- How would you implement a per-job timeout so that an infinite loop in a custom transform does not block a worker indefinitely?
- If input images were several gigabytes each, how would you redesign the system to avoid high serialization and inter-process communication overhead?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.