Katkat

Resolve a Package Dependency Graph with Concurrent Fetches

AnthropicSoftware Engineer · Onsite

Last updated

Discover a package's transitive dependency graph from a slow, flaky registry, first with a sequential BFS and then with a bounded pool of worker threads. Tests atomic visited-set reservation, explicit retry state, and correct termination detection with in-flight work.

You are writing the dependency-resolution step of a package manager. Given one or more root packages, you must discover the full transitive dependency graph by asking a remote registry for each package's manifest. The registry is slow (tens to hundreds of milliseconds per call) and occasionally flaky, so a sequential walk takes minutes on large projects.

You are given:

def fetch(name: str) -> list[str]:
    "Returns the names of the package's direct dependencies."
    # raises TransientError  -> timeout / 5xx, may succeed on retry
    # raises NotFoundError   -> package doesn't exist, never retry

and must return:

@dataclass
class Resolution:
    deps: dict[str, list[str]]   # every successfully fetched package -> its direct deps
    failed: dict[str, str]       # package -> reason ("not found", "gave up after N attempts")
    depth: dict[str, int]        # depth assigned when the package was first discovered

Part 1: Sequential walk

Implement resolve_sequential(roots, fetch, max_retries=2, max_depth=None). Each package must be fetched successfully at most once, even though shared dependencies and cycles are common (a -> b -> a is legal in some ecosystems). A package is retried at most max_retries times after a transient error. Roots have depth 0. If max_depth is set, packages at that depth are fetched but their dependencies are not explored.

Part 2: Concurrent walk

Implement ConcurrentResolver(fetch, workers=8, max_retries=2, max_depth=None).resolve(roots) using at most workers simultaneous fetches. Requirements:

  • Two workers that discover the same package at the same moment must not both fetch it.
  • "Fetched successfully", "failed permanently", and "waiting for a retry" are distinct states.
  • The call returns once all work is finished: the queue is empty and no fetch is still in progress (an in-progress fetch may discover new work).
  • No busy-waiting, and no holding a lock while calling fetch.

Example

graph:  app  -> web, db
        web  -> http, log, ghost        (ghost does not exist)
        db   -> log, pool
        http -> log
        pool -> log
        log  -> (none)
"pool" fails transiently once.

resolve(["app"]) with 4 workers:
  deps keys -> ['app', 'db', 'http', 'log', 'pool', 'web']
  failed    -> {'ghost': 'not found'}
  pool is fetched twice (one retry), log exactly once

Constraints

  • Up to 50,000 packages. The registry allows at most workers concurrent requests per client.
  • Without max_depth, the set of resolved packages must be identical to the sequential version's.

Hints

Hint 1

Mark a package as seen when you enqueue it, not when you finish fetching it.

Hint 2

In the concurrent version, the check 'already seen?' and the act of reserving must happen in one critical section.

Hint 3

An empty queue does not mean you are finished. What else must be zero?

Hint 4

Never call fetch while holding the lock. Use a Condition so idle workers sleep until new work arrives or everything is done.

Follow-up questions

  • How would you add exponential backoff for transient failures without blocking a worker thread during the wait?
  • The registry starts returning HTTP 429. How would you adapt concurrency dynamically?
  • How would you make the depth limit deterministic regardless of thread scheduling?
  • How would you cache manifests across runs, and how would you invalidate them when a package publishes a new version?