Katkat
Back

Stabilize LLM inference and estimate needed repeats

CitadelData Scientist · Technical Screen
MediumData ScientistTechnical ScreenML System Design

Last updated

This question tests the ability to design an LLM inference pipeline that ensures result stability through statistical aggregation and estimates the minimum number of inference samples required to achieve a target confidence level.

Scenario

You are designing a production inference pipeline for an LLM that extracts quantitative financial sentiment scores from unstructured text. Because LLMs are stochastic, multiple runs on the same input yield different outputs. You need to design a system that minimizes variance and guarantees a stable output before serving the downstream consumer.

Task

  1. Describe an architectural approach to stabilize the inference output. How would you determine the optimal number of repeated inferences (n) to reduce variance while maintaining latency constraints?
  2. Assume you have collected a batch of 100 historical inferences for a set of inputs. Define a statistical method to quantify the 'stability' of these outputs.
  3. If a stakeholder requires the mean output to be within an error margin of 'e' with a confidence level of 95%, how do you derive the number of repeats (n) needed for a new, unseen input?

Constraints & Requirements

  • Provide a pseudo-code or workflow logic to handle the orchestration of multiple inference calls.
  • Discuss the trade-offs between 'temperature' settings and the number of repetitions.
  • Address how you would measure the correlation (e.g., Pearson) between your stabilized outputs and a benchmark or historical expert labels to ensure the stability process hasn't introduced bias.
  • Assume the model latency per request is 200ms and the budget for the total inference process is 1 second.

Hints

Hint 1

Consider using the Central Limit Theorem to model the distribution of the mean of your repeated inferences.

Hint 2

Think about the relationship between variance, sample size, and the standard error of the mean (SEM).

Hint 3

Stability can be quantified by calculating the variance or the coefficient of variation (CV) across the outputs for a single prompt.

Follow-up questions

  • How would you handle outliers in the inference batch? Would you use the arithmetic mean, median, or a trimmed mean, and why?
  • If the inference budget is extremely tight, how would you shift your approach from increasing sample size (n) to adjusting decoding parameters?
  • How would you monitor for 'drift' in stability as the underlying model version or the input data distribution evolves over time?