Katkat
Back

Find frequent IPs from a huge file

AsanaSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question evaluates a candidate's competency in large-scale data processing and algorithm design, focusing on memory- and disk-constrained frequency estimation, streaming and approximation concepts.

You are given a very large log file (too large to fit in memory). Each line contains an IPv4 address (e.g., 203.0.113.7).

Design an approach to answer one of these variants under memory constraints:

  • Variant A: Find the single most frequent IP.
  • Variant B: Find the top K most frequent IPs.

Explain how you would implement it in practice (data structures, disk usage, complexity), and how your approach changes if:

  • Memory is limited to, say, 256MB.
  • The file is tens/hundreds of GB.
  • You only need an approximate answer.