Katkat
Back

Choose Fast or Cheap Models

GoogleSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

This question evaluates a candidate's competency in ML system design, focusing on balancing cost-versus-latency trade-offs for inference routing, operational metrics, workload segmentation, and reliability.

You are building an AI-powered product and must choose between two inference options for each request:

  • Option A: higher cost per token, but lower latency
  • Option B: lower cost per token, but higher latency

How would you decide when to use each option? Discuss the trade-offs across user experience, latency, quality, reliability, and operating cost. Also explain what metrics you would track, how you would segment different workloads, and whether you would use a dynamic routing strategy instead of a single global choice.