Katkat
Back

Design the Security Architecture for an LLM Chatbot Product

DecagonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

A system design question on securing a chatbot built on a large language model that talks with people outside the company. It tests threat modeling for prompt injection and data leakage, user authentication and tool authorization outside the model, tenant isolation, sensitive-data handling, abuse controls, auditing and incident response.

Design the security for a company's chatbot product: a production chatbot that holds conversations with people outside the company. Assume the chatbot is built on a large language model (LLM).

Cover how you would protect the chatbot, the data it can see, the systems it can reach and the people who talk to it. Start by establishing what the chatbot does and who uses it, then build a threat model, then design the controls and the architecture that enforces them.

Clarifying Questions

  • Is the chatbot run for many business customers, each exposing it to its own users (multi-tenant), or for a single organization?
  • Does it only answer questions from a knowledge base, or can it also take actions in other systems through tools, such as looking up or changing a user's account?
  • Are the people chatting anonymous visitors, signed-in users of the customer's product, or both?
  • What sensitive data can appear in conversations or in the knowledge it reads: personal data, payment details, health information?
  • Is the language model hosted by the company or called through a third-party API?
  • Which regulatory, data-residency and retention requirements apply?

Hints

Hint 1

Start from attackers and assets: List who could try to misuse the chatbot (the people chatting with it, the content it reads, other customers of the product, insiders) and what each of them could reach, before picking any control.

Hint 2

Separate what the model says from what the system does: Decide which decisions you would never leave to the language model alone, and where in the request path those decisions should be enforced instead.

Follow-up questions

  • A document in a customer's knowledge base contains hidden instructions telling the bot to reveal other users' data. Walk through what stops that attack at each layer.
  • Suppose the bot gets a tool that moves money, such as issuing refunds. How do you stop a user from talking it into refunds they are not entitled to?
  • How would you detect in production that your controls are being bypassed, and what would you do in the first hour after detection?
  • A customer requires that its users' conversations stay in one region and are never used to train models. What changes in the design?