Katkat
Back

Design a scalable search service with sharding

AirtableSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed search architecture capable of handling high-throughput indexing and low-latency queries across a massive dataset.

Problem Statement

Design a scalable search service for a platform like Airtable, where users store large, structured datasets in tables. The system must support full-text search across multiple columns, handling both rapid document updates and high-concurrency read queries.

Requirements

  • Functional: Users should be able to perform keyword searches across their specific tables with sub-200ms latency. The system must reflect data updates near-real-time.
  • Non-functional: The system must scale to petabytes of data, maintain high availability, and be resilient to the failure of individual index nodes.

Your design should address:

  1. Data Ingestion Pipeline: How to process updates from the primary database and trigger index refreshes.
  2. Sharding Strategy: How to distribute the search index across a cluster to balance load and storage.
  3. Query Path: The flow from receiving a user request to retrieving and aggregating results from shards.
  4. Consistency: How to handle the trade-off between write consistency and read-latency (e.g., refresh intervals vs. immediate visibility).

Examples

  • If a user updates a cell in a row of 1 million records, how quickly does that record appear in search results?
  • If a single table is 500GB, how should it be partitioned across your nodes to ensure search efficiency?

Hints

Hint 1

Consider using an inverted index structure and explain how you would split this index horizontally across multiple nodes.

Hint 2

Think about the role of a message queue or change data capture (CDC) mechanism in decoupling the primary data store from the search index updates.

Hint 3

Evaluate whether to use document-based sharding or term-based sharding, keeping in mind the need for 'scatter-gather' query execution.

Follow-up questions

  • How would you handle global 'search all' queries if a user has permissions across multiple tables stored on different shards?
  • What strategies would you implement to rebalance the cluster if one specific shard becomes a 'hot spot' due to a viral or highly active table?
  • How does your architecture change if you need to support multi-tenant isolation, where specific shards are dedicated to high-paying enterprise customers?