Katkat
Back

Design a PDF-to-Markdown Inference API

Mistral AISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

This question evaluates system-design and ML inference orchestration skills, assessing competency in composing multi-stage pipelines that manage heterogeneous resource profiles (CPU, GPU, memory) and trade-offs between latency and throughput.

Problem Statement

Design an inference service that converts PDF files into Markdown. Assume the following building blocks already exist and you do not need to implement them — your job is to compose them into a production service:

  • split_pdf(pdf) -> List[np.ndarray] — a CPU-intensive function that splits a PDF into individual pages and rasterizes each page into a NumPy array (image).
  • ocr(page_array) -> str (or batched ocr(batch) -> List[str]) — a GPU-intensive OCR engine that extracts text/layout from a page image.
  • to_markdown(ocr_output) -> str — a memory-intensive post-processing step that converts a page's OCR output into Markdown and/or assembles per-page results into a final document.

The work is therefore a three-stage pipeline with three different resource profiles (CPU → GPU → memory). The central design tension is that these stages contend for different resources and must scale independently.

You must design for two scenarios (Part 1 and Part 2 below).

Constraints & Assumptions

State these explicitly (or ask — see below) and design against them:

  • Scale: pages range from 1 to a few thousand; documents up to ~1,000 pages on the sync path. Many concurrent jobs on the async path (assume bursty traffic).
  • Resource asymmetry: CPU split, GPU OCR, and memory-heavy markdown each saturate a different resource; they must scale independently.
  • GPU is the cost driver: OCR throughput is gated by GPU utilization and batch size; idle GPUs are wasted money.
  • Correctness: the final Markdown must be assembled in original page order regardless of completion order.
  • Artifacts are large: a rasterized 1,000-page PDF is far too large to hold entirely in a single process's memory.
  • Sync SLA: a synchronous request must complete (or stream) within a bounded connection lifetime; assume a hard ceiling (e.g. minutes, not hours).

Clarifying Questions to Ask

  • What is the size distribution of PDFs (median vs. p99 page count), and the expected request rate / concurrency?
  • Is the synchronous response required to be one final ordered document, or may the API stream pages as they complete?
  • What are the latency SLAs — interactive (sync) vs. "result within N minutes" (async)?
  • Is multi-tenancy / fairness a requirement (do we need per-tenant quotas and isolation)?
  • What are the durability and retention requirements for source PDFs and outputs (and any compliance constraints on storing customer documents)?
  • Do we need partial results / progress, exactly-once delivery, or is at-least-once with idempotency acceptable?

Part 1 — Synchronous API for one very large document

A single user submits one very large PDF (e.g. 1,000 pages) and wants the full converted output as fast as possible, over a single request/response interaction. Optimize end-to-end latency for this one document.

What This Part Should Cover

  • A sync API contract and how the result is delivered (one ordered body vs. an incrementally streamed response), with a stated reason for the choice.
  • A page-level decomposition that overlaps CPU split, GPU OCR, and memory markdown into a pipeline rather than running them serially over all pages, plus a sense of why this is the dominant latency win.
  • An intra-job GPU batching scheme (micro-batching pages of the one document) and the throughput-vs-latency trade-off it implies, plus how the GPU is kept from idling.
  • A result-ordering mechanism that reassembles out-of-order page completions into correctly ordered Markdown.
  • Awareness that a single large document can exceed the connection ceiling, and what the contract does about it.

Part 2 — Asynchronous API for many concurrent requests

Many clients submit conversion requests concurrently, and each client is willing to receive the result later (poll, webhook, or download link). Here you optimize global throughput and resource utilization, not the latency of any single document.

Clarifying Questions for this Part

  • What delivery mechanisms must we support — polling, webhook callback, signed download URL, or all three?
  • Is there a per-tenant SLA or priority tier (e.g. paid jobs jump ahead), or is best-effort fair sharing acceptable?

What This Part Should Cover

  • An async API contract: submit → job_id, status/progress, and result retrieval (poll / webhook / signed download), with idempotent submit.
  • Decoupling via durable queues between the three stages so bursts are absorbed and each stage autoscales on its own signal.
  • Cross-job GPU batching — filling OCR batches from a shared queue across many jobs so small jobs ride along and the GPU stays saturated.
  • Fairness / scheduling so one giant document cannot starve small jobs (fair queuing, per-tenant caps, optional priority class).
  • Independent autoscaling of CPU / GPU / markdown pools, each keyed to its own bottleneck metric.

Hints

Hint 1

Where to start: Don't treat the document as one monolithic job that runs split → OCR → markdown sequentially over all 1,000 pages. The three stages have different bottlenecks (CPU, GPU, memory). What is the smallest unit you could process independently, and what would that buy you?

Hint 2

Pipelining: Think about whether a later stage really has to wait for the previous stage to finish for *every* page before it can start. Once you can process work at a finer granularity, what scheduling structure lets the three resources be busy at the same time, and what would you need to do to keep the GPU from sitting idle?

Hint 3

Delivering the result: A 1,000-page conversion can exceed normal HTTP timeouts, and pages won't necessarily finish in order. Think about how a long-running request can give the client useful output before the whole document is done, versus returning one ordered document. Whichever you pick, how do you reconstruct the correct final page order?

Hint 4

Decoupling: With many jobs in flight, think about what should sit *between* the CPU, GPU, and memory stages so a burst of uploads doesn't overwhelm the GPU pool, and so each stage can autoscale on its own metric. What component gives you that?

Hint 5

GPU efficiency: The GPU is the scarce, expensive resource, and it's most efficient when it processes full OCR batches — but a single small job may not have enough pages to fill one. Given that, where could the pages in a batch come from, and what scheduling policy stops one 1,000-page document from monopolizing the GPU and starving small jobs?

Follow-up questions

  • The sync path receives a 1,000-page PDF but the configured connection ceiling will be exceeded. What does your service do — reject, degrade, or transparently fall back to async — and what does the client see?
  • How do you guarantee **fairness** so that one tenant submitting many huge documents cannot monopolize the GPU pool and starve everyone else?
  • OCR quality is sometimes poor on certain pages (skew, low DPI). How would you add a **re-process / retry-with-different-settings** path (e.g. re-rasterize at higher DPI, deskew, swap OCR model) without reprocessing the whole document?
  • Suppose GPU OCR becomes the dominant cost. What levers (batch size, dynamic batching, quantization, autoscaling on queue depth, spot/preemptible GPUs) would you pull, and what are their risks?