Katkat
Back

Design an async job system and cache layer

SalesforceSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to architect a distributed job processing system and integrate a high-performance caching layer to handle scheduling, execution, and state management at scale.

Design an Async Job Processing and Caching System

Design a distributed background job processing system that supports scheduled tasks, automatic retries, and state tracking. Additionally, implement a caching layer to store job metadata and status updates, ensuring the system remains responsive under high read/write loads.

Requirements

  • Functional Requirements:
    • Ability to enqueue jobs with a delay or for immediate execution.
    • Persistent state transitions (PENDING, PROCESSING, COMPLETED, FAILED).
    • Retry mechanism with exponential backoff for failed jobs.
    • API to poll or receive notifications on job status.
  • Non-Functional Requirements:
    • Handle 10,000 requests/second.
    • Ensure job idempotency (processing the same job twice should have no side effects).
    • Low latency access to job status (sub-10ms).
    • High availability and fault tolerance.

Expected Output

  1. API Design: Define the necessary endpoints for submitting jobs, checking status, and internal worker communication.
  2. Data Model: Propose a schema for job storage and describe how you handle state transitions consistently.
  3. Caching Strategy: Detail how you would implement a cache layer. Address cache eviction, consistency (write-through vs. write-behind), and mitigation strategies for "hot keys" (e.g., thousands of workers querying the same job status).
  4. Architecture: Describe the high-level components (e.g., job queues, task workers, scheduler, caching layer, persistent storage).

Hints

Hint 1

Consider using a message queue or a partitioned log (like Kafka) to handle the decoupling between job producers and consumers.

Hint 2

For the cache, consider using a sharded distributed cache like Redis and implement a locking mechanism or optimistic concurrency control for status updates.

Hint 3

Address the 'thundering herd' problem when many workers try to update the status of the same job simultaneously.

Follow-up questions

  • How would you handle priority queues if some jobs are more urgent than others?
  • What specific observability metrics would you implement to detect if a background task queue is lagging?
  • How would you ensure exactly-once processing, or if that is too expensive, how do you handle at-least-once delivery with idempotency?