Katkat
Back

Design a Conversational Agent for Finding Vacation Rentals

Walmart LabsMachine Learning Engineer · Onsite
EasyMachine Learning EngineerOnsiteML System Design

Last updated

Design a multi-turn assistant that turns a traveler's free-form trip description into structured, grounded rental search results. Tests where to use an LLM versus deterministic search, clarification strategy, safety against hallucination and injection, and evaluation against classic search.

A vacation-rental marketplace wants to go beyond its filter-heavy search page. Travelers should be able to describe a trip in their own words, for example "somewhere quiet near a lake for 6 adults and a dog over the second weekend of July, ideally with a hot tub, under $400 a night". An assistant then asks a follow-up question if needed, searches the listing catalog, and refines its recommendations over several turns.

Design this agentic search assistant. Focus on the traveler's experience and the major system choices rather than on model internals.

Context

  • Catalog: around 2 million listings. Structured fields include location, capacity, bedrooms, price calendar, availability calendar, pet policy, and amenities. Unstructured fields include host description, house rules, and guest reviews.
  • Availability and price change constantly and are served by an authoritative availability service (p95 under 80 ms).
  • Traffic: up to 50 assistant sessions starting per second at peak, averaging 4 turns per session.
  • The assistant only finds and explains listings. It does not book or message hosts.

Requirements

  1. Turn free-form goals into a structured search state, and keep that state across turns, including when the traveler changes their mind ("actually make it 8 people").
  2. Separate hard constraints (dates, guest count, pets allowed) from soft preferences (quiet, hot tub, lake view) and treat them differently.
  3. Ask a clarifying question only when the answer would materially change the results.
  4. Every claim shown about a listing (price, "allows dogs", "5 min walk to the lake") must be backed by catalog data or a quoted review, and price and availability must be fresh.
  5. Behave sensibly when nothing matches, when the request is ambiguous, or when the agent is unsure.
  6. Bound latency and cost: the first useful response within about 4 seconds, with a capped number of model and tool calls per turn.

Deliver

  • The conversation flow for one request from start to finish.
  • Components, the tools the agent may call, and the data flow.
  • Safeguards against hallucinated facts, runaway tool loops, stale data, and instructions hidden in listing text.
  • An evaluation plan comparing the assistant with the existing filter search, offline and online.
  • Where an agent is not worth it, and the classic search page should be used instead.

Hints

Hint 1

Think of the LLM as an orchestrator around a normal filter-and-rank stack, not a replacement for it.

Hint 2

Keep a typed search state with hard constraints and soft preferences separated, and update it with diffs each turn.

Hint 3

Ask a clarifying question only when its answer would significantly change the candidate set.

Hint 4

Every fact shown about a listing should map back to a field or review, and price and availability need a fresh authoritative check.

Follow-up questions

  • How would you personalize results using a returning traveler's past bookings without making the assistant feel creepy?
  • The model sometimes treats a soft preference as a hard filter and returns nothing. How do you detect and fix that?
  • How would you extend the assistant to complete bookings safely?
  • What would you log per turn to debug a bad recommendation reported by a user?