Katkat
Back

Design a DAG Pipeline Runner for Data-Labeling Jobs

MercorMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteSystem Design

Last updated

Design a multi-tenant service that runs user-defined DAGs of automated steps (pre-processing, model pre-labeling, QA checks, exports), tracks every run, and survives crashes. Tests data modeling, separating the scheduler from workers, durable state, retries and idempotency, tenant fairness and observability.

A data-labeling platform lets each customer team define pipelines that prepare and post-process datasets. A pipeline is a directed acyclic graph of steps. A typical one: unpack an uploaded archive, run a model to pre-label each shard (a fan-out of 1 to 500 parallel steps), join the results, run automated quality checks, then export to the customer's bucket.

Design the service that stores these pipeline definitions, starts runs, executes steps in dependency order, and shows progress.

Functional requirements

  • Create versioned pipeline definitions: steps, dependencies, a container image or handler for each step, and parameters.
  • Start runs by API call, on a cron schedule, or when a dataset upload event arrives.
  • Execute ready steps in parallel when their dependencies allow it. Support dynamic fan-out (the number of shards is known only after the unpack step finishes) and fan-in.
  • Per-step retry policy, timeout, and an option to continue past non-critical failures.
  • Users can cancel a run, retry a failed step, and inspect status, logs and outputs for each step.

Non-functional requirements

  • About 2,000 tenants, 50,000 runs per day, and up to 5,000 runs active at once. Steps last from seconds to 6 hours.
  • No lost progress: if any scheduler or worker machine dies, runs continue without redoing steps that already finished.
  • Isolation: one tenant launching 10,000 runs must not starve the others.
  • The UI should show status changes within about 2 seconds.

Deliver

  1. Data model and the state machines for runs and steps.
  2. Component architecture: which piece decides what runs next, and which piece does the work.
  3. The exact path a step takes from blocked to succeeded, including the crash cases.
  4. Retry, timeout and idempotency semantics.
  5. Scaling, fairness between tenants, and observability.
  6. How old runs behave when a definition changes in the middle of a run.

Human review steps are out of scope for the first version. Mention how you would add them.

Hints

Hint 1

Put every state transition in a durable store. If the scheduler's memory disappears, what does it need in order to pick up where it left off?

Hint 2

Separate a stateless scheduler (it decides which steps are ready) from a pool of workers (they run the steps). Connect them with a durable queue that uses visibility timeouts or leases.

Hint 3

Assume at-least-once execution. A worker can finish a side effect and crash before it reports success. What makes a step safe to run twice?

Hint 4

For fairness, think about per-tenant queues or concurrency quotas, and a dispatcher that rotates between tenants.

Follow-up questions

  • A worker writes the export to the customer bucket and then crashes before reporting. Walk through exactly what prevents a duplicate or corrupted export.
  • How would you support conditional branches (if the QA score is below a threshold, run a relabel step) without breaking the DAG model?
  • One tenant's steps keep timing out and retrying, which eats GPU capacity. How do you contain it?
  • How would you backfill 10,000 runs against a new pipeline version without hurting live traffic?