Katkat
Back

Choose File Storage Tiers Using Access Predictions

AppleMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

Design an end-to-end machine learning and decision system that predicts file access likelihood to automatically transition cloud-stored user files across storage tiers to minimize operational cost while meeting retrieval latency SLAs.

Scenario

A major personal cloud storage service hosts billions of user assets (photos, documents, backups, videos). Storing all data in low-latency 'Hot' storage (e.g., standard object storage) is prohibitively expensive. Moving objects to 'Cold' storage (e.g., archive tiers) drastically cuts daily storage costs per gigabyte, but introduces retrieval fees and restoration latency (minutes to hours).

Your task is to design an automated lifecycle management system that evaluates files and decides whether to keep them in Hot storage or demote them to Cold storage (and vice versa) based on predicted future access.

Requirements

Functional Requirements

  • Predict the probability of file access over a fixed planning horizon (e.g., the next 30 or 90 days).
  • Translate access probabilities and file metadata into an optimal tier assignment that minimizes total expected cost: Cost = Storage_Cost + E[Retrieval_Cost] + Transition_Cost.
  • Enforce user experience constraints: files with high sensitivity to immediate retrieval latency must not be archived prematurely.

Non-Functional & Scale Requirements

  • Scale: 10 billion active objects across 100 million users; hundreds of petabytes under management.
  • Execution cadence: Offline batch processing running daily or weekly over the catalog.
  • Safety: Avoid 'thrashing' (frequently demoting and re-hydrating objects, which incurs compounded egress/transition fees).
  • Evaluation: Prevent temporal data leakage during model training and offline backtesting.

Key Areas to Cover

  1. Formulation & Feature Engineering: Define the target variable, prediction horizon, and features (file age, type, size, historical access frequency/recency, user activity patterns, client device type).
  2. Model Training & Calibration: Discuss model choice and explain why well-calibrated probabilities are strictly required for cost-based decision making.
  3. Decision Policy & Optimization: Derive the decision rule or threshold for demoting a file based on expected financial cost and transition overhead.
  4. Data Leakage & Evaluation: Detail how to split train/validation/test sets over time to avoid forward-looking bias, and define both ML metrics (Brier score, calibration curve) and business metrics (cost savings vs. retrieval delay rate).
  5. System Architecture: Outline the batch pipeline to score billions of items without overwhelming metadata stores.

Hints

Hint 1

Because downstream optimization compares expected dollar costs, raw ranking metrics like AUC-ROC are insufficient; probability calibration (e.g., Platt scaling or isotonic regression) is essential.

Hint 2

Formulate the tiering decision as a break-even analysis where the cost of storing in Hot for horizon T is compared against Cold storage cost plus probability of retrieval times retrieval fee plus transition fee.

Hint 3

To prevent data leakage in temporal access logs, use expanding-window or rolling-window temporal splits, ensuring feature calculation strictly excludes future interactions beyond the cutoff date.

Follow-up questions

  • How would you adjust your decision framework if an object's cold retrieval requires a minimum retention duration (e.g., AWS Glacier 90-day minimum storage charge)?
  • How can you handle 'cold start' files where a user has just uploaded hundreds of photos with zero access history?
  • If inference on 10 billion files daily is too compute-intensive, how would you design a two-stage filtering or triage mechanism to score only the most uncertain or impactful files?