Katkat
Back

Crawl Same-Domain Links

AnthropicSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenCoding & Algorithms

Last updated

This question evaluates understanding of web crawling and link traversal algorithms, including URL normalization, same-domain filtering, deduplication, and cycle-safe graph traversal.

Implement a Python function that crawls a website starting from a seed URL and returns all unique pages reachable within the same domain.

Assume a helper function get_links(url) -> list[str] is available and returns all hyperlinks found on that page.

Requirements:

  • Start from a single input URL.
  • Only follow links whose hostname matches the hostname of the seed URL.
  • Do not visit the same URL more than once.
  • Handle cycles safely.
  • Continue until there are no more unseen same-domain links.

You may choose either BFS or DFS traversal. Be prepared to discuss how you would normalize URLs to reduce duplicate visits.