Katkat
Back

Design a malicious-URL checking service using an isMalicious API

LinkedInSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests your ability to design a high-throughput, low-latency URL reputation checking service, focusing on caching strategies, scalability, and system resilience.

Problem Statement

Design a malicious-URL checking service that provides a real-time assessment of whether a given URL is malicious. You have access to an existing third-party API isMalicious(url) which is expensive (high latency and rate-limited) but provides the ground truth.

Functional Requirements

  • Provide an API checkUrl(url) that returns a boolean indicating if the URL is malicious.
  • The system must provide low-latency responses, even if the third-party API is slow.
  • The system must handle high request volume (e.g., 50k requests per second).

Non-Functional Requirements

  • Minimize the number of calls made to the third-party isMalicious API.
  • The system should be highly available; it should fail gracefully if the third-party service is down.
  • Malicious URL status can change over time; implement a strategy to update or invalidate cached data.

Deliverables

  • High-level system architecture.
  • API design and data storage choice (e.g., database schema for reputation storage).
  • Caching strategy (e.g., cache eviction, TTL, or bloom filters for negative lookups).
  • Strategy for handling rate limits and service outages of the third-party dependency.

Hints

Hint 1

Use a multi-layered caching approach with a distributed cache like Redis to store previous results and reduce calls to the third-party API.

Hint 2

Implement a Bloom filter in front of the cache to quickly identify URLs that are definitely not in the malicious database to save cache memory.

Hint 3

Use an asynchronous update pattern (e.g., a message queue) to refresh the reputation of frequently accessed URLs so the user doesn't wait for a slow third-party API call.

Follow-up questions

  • How would you handle a scenario where the third-party API experiences a total outage? Should you fail closed or fail open?
  • How would you shard the data if the number of known malicious URLs reaches several billion entries?
  • How would you measure the effectiveness of your caching strategy and the health of the downstream service?