Katkat
Back

Design an LLM agent with RAG and tools

Walmart LabsMachine Learning Engineer · Onsite
EasyMachine Learning EngineerOnsiteMachine Learning

Last updated

This problem tests the ability to architect an LLM-based agent capable of retrieving information and performing actions, focusing on orchestration, tool execution, and system reliability.

Scenario

You are tasked with designing an AI customer service agent for an e-commerce platform. The agent must handle two types of requests:

  1. Answering product-related questions using a product knowledge base (Retrieval-Augmented Generation).
  2. Managing order statuses by interacting with an internal SQL database (Tool Execution).

Design Requirements

  1. Orchestration: Describe how the agent decides between searching the knowledge base and querying the database.
  2. Tool Integration: Explain how you define the tool schema for an LLM to ensure it can invoke the database query function reliably.
  3. Safety and Guardrails: How do you prevent the agent from performing unauthorized actions (e.g., deleting orders) or leaking sensitive PII from the knowledge base?
  4. Model Parameters: Explain how the choice of temperature impacts the agent's performance in this specific context.

Expectations

  • Outline the flow from receiving the user prompt to delivering the final response.
  • Propose a data model or interface for the tools.
  • Briefly describe how you would evaluate the agent's accuracy and tool-use precision.

Hints

Hint 1

Consider using an agentic framework that utilizes function calling (tool calling) capabilities available in modern LLM APIs.

Hint 2

Think about implementing a ReAct (Reasoning and Acting) loop where the model writes a thought process before outputting a tool call or a final answer.

Hint 3

Focus on systemic constraints like input validation, prompt sandboxing, and strict schema enforcement to ensure safety.

Follow-up questions

  • How would you handle a scenario where the agent encounters an error while executing the SQL tool, such as a timeout or a schema mismatch?
  • If the knowledge base becomes massive (millions of documents), how would you optimize the retrieval component?
  • How would you implement a human-in-the-loop mechanism for high-stakes actions like canceling an order?