Katkat
Back

Design GPU credit allocator

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

A multi-tenant GPU credit allocation system design (OpenAI software-engineer system-design screen): track and deduct GPU credits in real time across many nodes, issue/transfer/spend credits via idempotent APIs, and enforce budgets, hierarchical quotas, rate limits, and fair-share with scheduler integration. Covers prepaid vs postpaid, an immutable double-entry ledger with reservations, overspend prevention, failure recovery, scaling, and observability.

Question

Design a GPU credit allocation system for a multi-tenant compute platform (e.g., OpenAI's GPU infrastructure). Organizations are granted GPU credits (for example, a monthly allowance), and users run jobs that consume those credits across many GPU nodes. Your design should cover:

  1. Track real-time GPU consumption per job and deduct credits as work is performed, across multiple GPU nodes.
  2. Define APIs to issue/grant credits, transfer credits between scopes (org -> project -> user), top up credits, query balances, and spend/settle credits for jobs.
  3. Enforce budgets, per-user/per-project quotas, and rate limits in real time under high concurrency, preventing double-spend and overspend.
  4. Integrate with a job scheduler / admission controller to admit, throttle, or reject workloads based on available credits and limits.
  5. Support both prepaid and postpaid billing models.
  6. Enforce fair usage when GPUs are scarce (fair-share across users/orgs).
  7. Address idempotency, the consistency model, data model, failure recovery, scaling (partitioning, caching), audit logging/reporting, and observability/alerting.