Katkat
Back

Design a Job Scheduler That Can Scale by 100 Times

NextdoorSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a distributed, asynchronous job scheduler that starts at modest traffic but can seamlessly scale up 100x. Evaluates API design, data storage trade-offs, worker coordination, task deduplication, and failure handling across scale boundaries.

Scenario

Nextdoor relies on asynchronous background tasks for various neighborhood features: sending digests, geocoding new addresses, recalculating neighborhood boundaries, and dispatching push notifications. Currently, the platform runs a centralized scheduler handling a modest job volume. However, as international expansion and notification frequency grow, the scheduler must be redesigned to gracefully scale by 100x without requiring a complete rewrite.

Requirements

Functional Requirements

  • Job Submission: Clients can submit one-off delayed jobs or recurring (cron-style) tasks with an execution payload and execution timestamp.
  • Execution Guarantees: Ensure at-least-once execution of scheduled tasks.
  • Job Management: Support checking job status, canceling pending jobs, and querying execution history.
  • Priority & Retries: Support task priorities (e.g., immediate user-facing notifications vs. daily email digests) and configurable exponential backoff retry policies.

Non-Functional Requirements

  • Initial Scale: 500 jobs submitted per minute (~8.3 QPS), with peak execution of 1,000 jobs per minute. Total active delayed tasks in storage: ~1 million.
  • Target Scale (100x): 50,000 jobs submitted per minute (~833 QPS sustained, 2,500 QPS peak), with 100 million active scheduled tasks.
  • Precision: Delayed jobs should trigger within 1–5 seconds of their target execution time.
  • Reliability: No single point of failure; jobs must survive worker or scheduler crashes.

What Your Design Should Cover

  1. API Design: Core endpoints for submitting, canceling, and inspecting jobs.
  2. Data Storage & Indexing: Schema and choice of data stores for both initial scale (e.g., relational DB with indexing) and the 100x target (e.g., partitioned queues, distributed key-value stores, or sorted sets).
  3. Scheduler Architecture: Mechanism for identifying ready-to-run jobs (polling vs. push, wheel timers, or range-sharded scans) and dispatching them to worker pools.
  4. Concurrency & Execution: Avoiding duplicate execution under high concurrency; handling worker heartbeats and task leases/locks.
  5. 100x Migration Path: Explicitly explain how components transition from the baseline scale to 100x scale (e.g., DB sharding, queue decoupling, distributed locking bottlenecks).

Hints

Hint 1

At lower scale, an indexed relational table polled with `SELECT FOR UPDATE SKIP LOCKED` works well, but it degrades when polling millions of rows across high-concurrency dispatchers.

Hint 2

Consider decoupling the delay storage (e.g., time-bucketed partitions or sorted sets) from the actual worker execution queue (e.g., standard FIFO message brokers).

Hint 3

Think about task leasing: workers should claim tasks with an expiration time, heartbeating back to extend leases for long-running jobs to handle crashed nodes gracefully.

Follow-up questions

  • How would you prevent a 'thundering herd' problem if 100,000 jobs are scheduled to trigger at the exact same minute boundary (e.g., 09:00:00)?
  • How does your design achieve idempotent execution or prevent duplicate delivery when a worker crashes mid-task?
  • If clients request strict exactly-once scheduling semantics, what architectural changes and client-side contracts would be required?