Katkat
Back

Design a job scheduler with SLA and logs

RobinhoodSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed, fault-tolerant job scheduling system capable of meeting strict SLAs and providing audit-ready logging.

System Design: Distributed Job Scheduler

Design a distributed job scheduling system that allows users to submit tasks to be executed at a specific time or on a recurring schedule. The system must support thousands of concurrent jobs and ensure that task execution meets predefined Service Level Agreements (SLAs), such as 'start execution within 5 seconds of the scheduled time'.

Requirements

  • Functional:
    • Ability to schedule one-time tasks and cron-style recurring tasks.
    • A mechanism for workers to pick up tasks and report success or failure.
    • Support for job retries with exponential backoff if a task fails.
  • Non-Functional:
    • Scalability: Handle high job submission rates and scale the worker fleet dynamically.
    • Reliability: Ensure no job is lost even if a worker or scheduler node crashes (at-least-once execution).
    • Observability: Provide a logging interface to track the state transitions (queued, running, completed, failed) of every job for audit and debugging purposes.
  • Assumptions:
    • Assume a fleet of transient worker nodes.
    • System should be able to handle 10,000 scheduled tasks per minute.

Scope

  • Define the API endpoints for job submission and status monitoring.
  • Outline the data model for storing job definitions, state, and logs.
  • Describe the architecture, specifically how the scheduler identifies tasks ready for execution and ensures they are picked up by workers without duplicate execution.
  • Explain how you would implement SLA monitoring and alerting for missed deadlines.

Hints

Hint 1

Consider using a distributed message queue or a prioritized task store to decouple job submission from worker execution.

Hint 2

For task scheduling, think about using a 'time-bucketed' approach or a distributed heap structure to efficiently query tasks due in the next few seconds.

Hint 3

Use a distributed lock or a state-machine approach (e.g., status changing from 'queued' to 'in-progress') to ensure a task is only claimed by one worker at a time.

Follow-up questions

  • How would you handle a 'hot partition' where millions of jobs are scheduled for the exact same second?
  • How would you integrate a circuit breaker pattern to prevent the system from overwhelming downstream services if the scheduled jobs fail repeatedly?
  • If the logs grow to petabyte scale, what storage and retention strategy would you propose for the job history?