Generate Consistent Outputs for Similar Inputs

Scenario

In an automated medical chart review and data extraction platform, users submit clinical text snippets, notes, or standardized queries to generate structured records (such as risk factor extraction or billing code suggestions). Generating these outputs requires costly model inference and deterministic post-processing. Often, incoming requests contain minor lexical variations, whitespace differences, or synonymous phrasing that represent the exact same semantic intent.

Design a system that returns identical, consistent outputs for semantically similar inputs, caches previous generation results safely, and avoids redundant computation during traffic spikes.

Requirements

Functional Requirements

• Input Processing: Accept raw text queries or clinical input payloads and return the structured generation result.
• Similarity Detection: Determine whether an incoming input is identical or semantically equivalent to a previously evaluated input based on a configurable similarity threshold.
• Output Consistency: Guarantee that similar inputs map to the same canonical output for a given pipeline version.
• Versioning & Updates: Support model or rule-set updates, ensuring stale outputs are either purged, tagged with version numbers, or recomputed when the generation logic evolves.

Non-Functional Requirements

• Latency:
  • Under 100 ms p99 for similarity cache hits.
  • Raw computation takes ~1.5 to 3 seconds p99 when cache misses occur.
• Throughput: Handle an average of 3,000 queries per second (QPS) with peak loads of 8,000 QPS.
• Concurrency Control: Prevent "thundering herd" / cache stampedes when dozens of identical or near-identical inputs arrive simultaneously.
• Consistency & Durability: Canonical records and mappings must be durably stored and resilient to node failures.

What Your Solution Should Cover

1. Similarity Strategy: How similarity is determined (e.g., canonicalization rules, locality-sensitive hashing, or embedding vector search) and where the boundary between identical and divergent outputs lies.
2. Concurrency & Request Coordination: How concurrent near-identical requests are collapsed into a single compute job (e.g., SingleFlight, distributed locking, or pub/sub).
3. Data Model & Storage: Schemas for storing canonical representations, embeddings/indexes, cached outputs, and version metadata.
4. Lifecycle & Invalidation: How changes to prompt templates, model checkpoints, or extraction rules are rolled out without returning inconsistent results.