Katkat
Back

Implement Rate-Limited Wikipedia Crawler

GleanMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteCoding & Algorithms

Last updated

This question evaluates a candidate's ability to implement a rate-limited web crawler, testing skills in crawl frontier management, prioritization by page-title initial, deduplication, rate limiting, and robust error handling.

Implement a rate-limited Wikipedia crawler.

You may use an AI coding assistant during the interview, such as Cursor or Claude Code. You are expected to understand, modify, and explain the generated code.

Requirements:

  • Start from a given Wikipedia page title, for example Google.
  • Fetch the page and extract outgoing Wikipedia links.
  • Maintain a set of visited page titles.
  • Track the first character of each visited page title, considering the English letters A through Z case-insensitively.
  • When choosing the next page to crawl, prioritize pages whose first character has not yet appeared among visited pages.
  • For example, if the start page is Google, then G has already appeared, so links whose titles start with G should not be prioritized while other unseen starting letters remain available.
  • Once all 26 starting letters have appeared, choose additional pages to crawl randomly from the remaining frontier.
  • Respect a configurable rate limit, such as at most R requests per second.
  • Avoid crawling the same page twice.
  • Handle common failures such as invalid pages, duplicate links, HTTP errors, and empty frontiers.

Design and implement the crawler interface. You may assume helper functions for fetching page HTML or API responses if necessary, but the crawling logic, prioritization, deduplication, and rate limiting should be implemented clearly.