Design a prompt playground

Design a prompt playground for developers and prompt engineers.

The product lets users write prompts, choose model settings, run prompts against AI models, stream results back to the browser in real time, save prompt versions, compare outputs, collaborate with teammates, and inspect cost, latency, and safety issues.

This is an open-ended system design problem. Drive it like a real interview: scope the requirements first, do back-of-the-envelope sizing, sketch a high-level architecture, then go deep on the hard parts (streaming, cancellation, tenant isolation, and cost control). State any assumptions you make explicitly.

Constraints & Assumptions

Assume the following scale unless you state different assumptions:

• 100,000 monthly active users.
• 1 million prompt runs per day.
• Some runs stream tokens back to the browser in real time.
• Users belong to workspaces or organizations (multi-tenant).
• Prompt content and model outputs may contain sensitive data.

Treat external model providers as a dependency you call over the network: calls cost real money per token, have variable latency, can be rate-limited or temporarily unavailable, and may refuse a request.

Clarifying Questions to Ask

A strong candidate scopes the problem before designing. Good questions to raise with the interviewer (these scope the whole system; per-Part clarifications appear under the relevant Part below):

• What is the read:write split? How many runs vs. how many reads of history/usage dashboards, and how interactive (bursty) is traffic across the day and time zones?
• What is the average run duration and output length (tokens), and what fraction of runs stream vs. fire-and-forget? This sizes concurrent connections and storage.
• Are models internal only, external providers, or both — and do we need a fallback model when one is degraded?
• What are the latency targets — especially time-to-first-token for streamed runs — and what durability guarantee do we owe a run if the browser disconnects mid-stream?
• How strict is tenant isolation and data retention (e.g. per-workspace encryption, configurable deletion, regulatory deletion requests)?
• Is real-time multiplayer co-editing in scope, or is versioning + comments sufficient for v1?

The interviewer will expect you to cover the following. Treat each as a part of your answer.

Part 1 — Core user flows and requirements

Lay out the functional and non-functional requirements, the key user flows (author a prompt, run it, watch it stream, save a version, compare outputs, review history), and what you are explicitly leaving out of scope.

What This Part Should Cover

• Functional vs. non-functional split with the non-functional list anchored to this product's stakes (low time-to-first-token, durability of in-flight runs, tenant isolation, cost control).
• An explicit out-of-scope list (e.g. training/fine-tuning, billing/payments, cursor-level co-editing) — naming what you are not building shows judgment.
• End-to-end user flows that connect authoring → run → stream → save version → compare → review history, not just a feature list.

Part 2 — APIs and data model

Define the core entities and a sensible API surface. Pay attention to what must be preserved so a run is reproducible later, and where large/sensitive content lives.

What This Part Should Cover

• A clear entity model (workspace, membership, prompt, immutable version, run, run output, quota, audit) with reproducibility built in: a run pins a version and snapshots the effective config + variable values.
• A storage split argument — relational metadata vs. object-stored large bodies — justified by what each field is queried by and how big it gets.
• A coherent API surface for authoring, running, streaming, comparing, and usage — including where reproducibility breaks down (nondeterminism at temperature > 0).

Part 3 — High-level architecture

Sketch the components and how a request flows through them. A good answer makes one structural decision very clear.

What This Part Should Cover

• One load-bearing structural decision stated up front — separating the authoring/collaboration plane from the execution plane — and used to organize the rest.
• A request flow through named components (gateway → orchestrator → queue → workers → model gateway → providers; a separate streaming tier) with each component's responsibility clear.
• A model gateway that centralizes provider-specific concerns so the rest of the system stays provider-agnostic.

Part 4 — Prompt execution, streaming, retries, and cancellation

This is the core of the problem. Walk through the run lifecycle and be precise about how tokens stream to the browser, when you retry, and how cancellation actually stops billing.

What This Part Should Cover

• A precise run lifecycle (queued → running → succeeded/failed/canceled) with the streaming path decoupled from the worker so a run survives a dropped browser.
• A retry policy that distinguishes retryable (timeout, 429, 5xx) from deterministic errors and refusals, plus idempotency on POST /runs to prevent double-charging.
• Cancellation across process boundaries — how cancel intent reaches the worker, what aborts the upstream generation to stop billing, and how the cancel-vs-completion race resolves.

Part 5 — Versioning, collaboration, and run history

Explain how editing produces versions, how teammates share and comment, and how run history is stored and queried at scale.

What This Part Should Cover

• A versioning model where editing produces immutable versions and comments live separately, so discussion never mutates a version.
• A justified scope cut on concurrent editing (last-write-wins draft + version history vs. OT/CRDT), tied to whether live multiplayer is core.
• Run history at scale driven by the 1M/day number: partitioning/retention, bodies in object storage, and heavy aggregations pushed off the transactional primary.

Part 6 — Security, privacy, rate limiting, and abuse prevention

Prompt content and outputs may be sensitive, so this part is load-bearing. Cover tenant isolation, encryption, secret handling, rate limits vs. spend quotas, and abuse prevention.

What This Part Should Cover

• Tenant isolation enforced server-side on every query via a session-derived workspace_id, plus encryption in transit/at rest and role-based access.
• Secret handling and log hygiene — provider keys confined to the gateway, sensitive prompt/output content kept out of logs/traces.
• Two distinct controls — rate limits (protect the system) and spend quotas (protect the budget) — enforced at the right layers, plus retention/deletion controls and abuse/anomaly detection.

Part 7 — Observability, cost tracking, and scalability tradeoffs

Describe what you measure to debug a slow or failed run, how you compute and store authoritative cost, and the major tradeoffs you are making (and why).

What This Part Should Cover

• The right metrics and end-to-end tracing to debug a single slow/failed run (queue wait, time-to-first-token, tokens/sec, error/refusal/timeout/cancel rates, per-dimension cost).
• Authoritative cost computed at run time and stored on the run, with a clear reason why recomputing from a code constant at read time is a trap.
• Named tradeoffs (async vs. sync, storage split, conservative output caching, graceful degradation) argued rather than asserted.