Katkat
Back

Manage Prompt and Model Behavior in Production

Luma AISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

This question tests the ability to design an end-to-end configuration management and observability system for LLM applications that ensures reproducibility, safety, and rapid recovery from performance drift.

Scenario

You are building a high-scale RAG-based application using LLMs. The team often modifies system prompts, changes model endpoints, swaps retrieval strategies, or adjusts decoding parameters (temperature, top-p) to improve user experience. Recently, a subtle change in the system prompt caused an increase in 'refusal' responses that was not caught until 24 hours later.

Requirements

Design a system to manage these configurations and prevent silent regressions. Your design should address:

  1. Configuration Management: How do you store and version prompt templates, model metadata, and hyper-parameters so that every request is tied to a specific, immutable 'experiment ID'?
  2. Staged Rollouts: Describe how to safely ship a new system prompt to 5% of traffic and monitor its impact on downstream metrics (e.g., faithfulness, latency, token costs).
  3. Observability & Attribution: How do you reconstruct the exact execution state (which retrieval context was injected, which prompt was used) for a specific production log entry to debug a hallucination?
  4. Incident Response: Design a mechanism for immediate rollback to the 'last known good' configuration without redeploying application code.

Constraints

  • The system must support thousands of requests per second.
  • Sensitive user data used in prompts must be handled according to privacy standards (e.g., PII masking in logs).
  • The configuration service should provide high availability; it cannot be a single point of failure for inference requests.

Hints

Hint 1

Consider decoupling the configuration service from the application code by using a sidecar pattern or a cached remote configuration store.

Hint 2

Think about how to implement a 'shadow mode' where new configurations process traffic in parallel to the production path without returning results to the end user.

Hint 3

For incident analysis, explore how to capture 'input-context-output' triples as structured data rather than unstructured text logs.

Follow-up questions

  • How would you integrate automated evaluation metrics (like RAGAS or LLM-as-a-judge) into your CI/CD pipeline to gate configuration deployments?
  • If a model provider changes their underlying API behavior (silent drift), how does your system help detect that shift compared to changes you made to your prompt?
  • How would you handle configuration dependencies, such as a specific prompt that is only compatible with a specific model version?