Design a Text-to-Image Retrieval System
Gimlet LabsMember of Technical Staff, ML Systems · OnsiteLast updated
Design permission-aware text-to-image retrieval with safe ingestion, multimodal embeddings, metadata filtering, approximate search, reranking, relevance evaluation, versioned updates, deletion, and duplicate control.
Interview Prompt
Design a system where images are uploaded offline and users enter natural-language questions or descriptions to retrieve the most relevant images. Cover ingestion, multimodal representations, metadata and access control, vector search, reranking, evaluation, updates, and failure cases such as duplicate or unsafe images.
Constraints & Assumptions
- Images can be large and arrive in batches.
- Text queries must retrieve visual concepts even when no matching caption was supplied.
- Results must respect tenant and object-level permissions.
- An image update or deletion must propagate to every derived representation.
Clarifying Questions to Ask
- Is relevance based on visual content, OCR text, metadata, or all three?
- What corpus size, query rate, and freshness target apply?
- Can the system use human relevance feedback?
Follow-up questions
- How would you handle a query for text visible inside an image?
- How would you diversify results so near-duplicate images do not fill the page?
- What changes if users can submit an image as the query?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.