Katkat
Back

Design Chatbot Personalization Memory

MicrosoftSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem evaluates the ability to design a scalable, low-latency memory system for an LLM-based chatbot that stores and retrieves user-specific facts to provide personalized responses.

Problem Statement

Design a personalization memory layer for a chatbot that stores long-term user information (e.g., "I prefer dark mode," "My dog is named Rex"). The system must allow the chatbot to recall these facts during a conversation to improve relevance. The user base is in the millions, and the system must handle high-concurrency read/write operations with a latency budget of under 100ms for memory retrieval.

Requirements

  1. Data Storage & Modeling: How should facts be extracted, stored, and updated? Consider the schema and the trade-offs between structured vs. unstructured storage.
  2. Retrieval: How do you retrieve only the relevant subset of memories for a new query without scanning the entire history?
  3. Consistency & Conflict: How do you handle contradictory information (e.g., the user previously said "I am a vegetarian" but later says "I had steak for dinner")?
  4. Scale & Lifecycle: How do you handle unbounded user history? How do you implement data privacy (e.g., user deletion requests) while maintaining performance?

Deliverables

  • Describe the architecture including storage choices (e.g., Vector DBs, KV stores, Relational DBs).
  • Define the flow from user input to final LLM response generation including memory injection.
  • Provide an approach to filtering and ranking memory candidates to ensure the LLM context window remains optimized.

Hints

Hint 1

Consider a hybrid approach using a vector database for semantic retrieval of facts and a relational database for managing metadata and expiration policies.

Hint 2

Think about using a background processing pipeline for fact extraction so that it does not block the primary response path.

Hint 3

To handle contradictions, consider assigning timestamps or confidence scores to facts and implementing a resolution policy during the retrieval phase.

Follow-up questions

  • How would you measure the performance and accuracy of the memory retrieval system?
  • How would you ensure that sensitive or PII-heavy facts are stored securely and comply with GDPR or similar regulations?
  • How would you handle cold-start issues for new users who have no existing memory profile?