Design an On-Demand Cloud Notebook Service for ML Engineers

Design a cloud notebook service for machine learning engineers: on request, it provisions a workspace running a Jupyter-style notebook server backed by one or more GPUs, which an engineer connects to from their browser, runs training and analysis code in, pauses when they're not using it, and resumes later with their files and installed packages intact.

Only the product's shape is given here — confirm specifics with the interviewer and treat the open points below as clarifying questions to raise early.

Constraints and clarifications to settle

• No scale, latency, or cost targets are given — state the numbers you assume (e.g., how many engineers, how many concurrent workspaces, how many GPU types).
• A workspace runs arbitrary user code and must be isolated from other workspaces and from the platform's own control plane.
• A workspace is stateful and long-lived: it can hold hours or days of uncommitted notebook state, installed packages, and large datasets or model checkpoints.
• GPUs are an expensive, limited resource — idle GPU-backed workspaces should not silently burn budget indefinitely.

Clarifying questions worth raising

• How many engineers use this, and how many workspaces run concurrently at peak?
• What GPU types and counts must a single workspace support, and can that change after creation (e.g., engineer wants to add a second GPU later)?
• How fast must a new workspace become usable, and how fast must a paused one resume? Is a short wait acceptable if a GPU of the right type isn't immediately free?
• What must survive a pause: only files, or also installed packages, running background processes, and open terminal sessions?
• Is a workspace built from a base image plus a setup script (e.g., a Dockerfile-like spec), or does it start blank and the engineer installs everything by hand?
• Should idle workspaces pause automatically, and what counts as idle — no browser connection, or no GPU utilization, or both?

What a strong answer covers

• A clear lifecycle and API: create, connect, pause, resume, delete, with explicit states and the valid transitions between them.
• A latency budget for workspace creation and resume, and which steps (image pull, GPU attach, storage mount) can be done ahead of time versus only on demand.
• A persistent storage design that survives a pause/resume and, ideally, a host failure, separate from the ephemeral compute the notebook server runs on.
• A connection/routing design that keeps working when a resumed workspace lands on a different physical host than before.
• An idle-detection and auto-pause policy, with the cost and user-experience tradeoff it represents.
• Isolation between workspaces, and between a workspace and the platform's control plane.