Katkat
Back

Design a Fair-Queueing Image-Upscaling Service, One GPU per Job

OpenAISoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design the backend for a SaaS image-upscaling API where every job runs on exactly one accelerator and a scheduler must divide a shared fleet across many paying customers, testing job-store design, weighted scheduling by accelerator-time rather than job count, and recovery from worker, accelerator, and scheduler failures.

LumenScale sells an HTTP API that takes a customer's image and returns a higher-resolution version, produced by a neural upscaling model that runs entirely on one accelerator per image. A model instance is never sharded across accelerators -- a job occupies exactly one accelerator for its whole runtime, from the moment it starts until the output is written out.

LumenScale has thousands of customer accounts split across three plans (Free, Pro, Enterprise) that all draw from a single shared fleet of accelerators. At peak, the service receives on the order of 3,000 upscale requests per minute. A job's runtime depends heavily on the requested scale factor and the input resolution: a small thumbnail upscaled 2x finishes in roughly 2 seconds, while a 4x upscale of a large photo can take 30-40 seconds. Across the full traffic mix the median job takes about 5 seconds and the average is closer to 8 seconds.

Design the system end to end. Treat the numbers above as given, but introduce and justify any additional assumption you need (fleet size, timeouts, limits) rather than leaving it unstated.

Clarifying Questions

  • Should fairness be computed per API key, per billing account, or per plan tier as a whole?
  • Should each account have a hard concurrency ceiling regardless of what its fair share would otherwise allow?
  • Does the upscaling model support resuming a partially finished image, or does a crash mean redoing that job from the start?
  • Do customers need visibility into queue position or an ETA, or is a terminal success/failure callback enough?
  • When the fleet is fully booked, does the service keep queuing new submissions indefinitely, or push back at the API layer?
  • Is the accelerator fleet uniform, or does it mix hardware generations with different memory limits or supported scale factors?

Part 1 -- Request flow, job store, and accelerator matching

Design the path from an uploaded image to a finished, downloadable result: the API surface, the durable record that tracks a job through its lifecycle, and the logic an idle accelerator uses to pick up its next piece of work.

Part 2 -- A fairness policy across customers

Design the rule that decides, every time an accelerator frees up, whose queued job runs next. Justify your choice of fairness unit, explain how a customer who has been quiet for a while and then resumes submitting is treated, and describe any limit that applies on top of the base fairness rule.

Part 3 -- Failure handling

Enumerate the ways this system can fail -- a worker process dying, an accelerator going unhealthy mid-job, a specific job crashing the model repeatedly, and the scheduler process itself losing its in-memory state -- and design detection and recovery for each, so that no customer's image is silently lost, delivered twice, or mischarged against their fair share.

Hints

Hint 1

Model the job store as a state machine (queued -> leased -> running -> done/failed) before touching the fairness rule -- scheduling is just a policy for choosing the next transition, not a separate system bolted on afterward.

Hint 2

One accelerator per job removes cross-accelerator coordination, but a 40-second job and a 2-second job now compete for the exact same resource -- decide what that means for how long a small job can be made to wait behind a large one.

Hint 3

If you measure fairness in jobs completed, a customer submitting many tiny thumbnails looks identical to one running a handful of huge batches -- pick a unit that actually reflects how much accelerator occupancy each customer is using.

Hint 4

Every recovery case in Part 3 -- a dead worker, a stuck accelerator, a crashed scheduler -- reduces to the same question: who currently owns this job, and how does the rest of the system know when that ownership has gone stale?

Follow-up questions

  • How would the fairness policy need to change if Enterprise customers are contractually promised a guaranteed minimum throughput rather than just a higher weight?
  • One customer's image keeps crashing the model after being retried on three separate accelerators -- what do you do with that job, and what, if anything, do you tell the customer?
  • How does accelerator matching change if half the fleet only has enough memory for up to 2x upscales while the other half supports up to 8x?
  • How would you detect in production that your accelerator-time accounting has quietly drifted, so that one customer is getting more than their weight entitles them to?