Katkat
Back

Design a file upload and scanning report system

CrowdstrikeSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a distributed, asynchronous system that handles high-throughput file uploads, coordinates multi-engine security scanning, and provides reporting capabilities.

Problem: Secure File Upload & Scanning System

Design a system that allows authenticated users to upload files for security scanning. Once a file is uploaded, it must be scanned by multiple third-party antivirus engines. The system should generate a comprehensive report of the findings and notify the user when the scan is complete.

Functional Requirements

  1. Users can upload files (up to 500MB each).
  2. The system triggers concurrent scans using multiple internal or third-party engines.
  3. Users can query the status and final report of their file uploads.
  4. Files must be quarantined if any scanner identifies a threat.

Non-Functional Requirements

  1. High availability and durability for uploaded files.
  2. Scalable processing to handle spikes in traffic.
  3. Observability to monitor scan engine health and latency.
  4. Security-first design (e.g., scanning should happen in an isolated environment).

Expected Components

  • API Design: Outline the core endpoints for uploading, status checking, and report retrieval.
  • Data Model: Describe the schema for tracking file metadata and scanning results.
  • Architecture: Provide a high-level design, focusing on the asynchronous workflow using queues or message brokers.
  • Error Handling: How do you handle scan engine failures, partial results, or corrupted files?

Hints

Hint 1

Consider using an object storage service with signed URLs for direct client uploads to keep large file traffic off your main application servers.

Hint 2

Use a message queue (like RabbitMQ or Kafka) to decouple the upload receipt from the intensive scan processing logic.

Hint 3

For the scanning layer, implement a worker pattern where individual containers perform the work in isolated sandboxes to prevent cross-contamination.

Follow-up questions

  • How would you handle a scenario where one specific scan engine consistently times out or returns false positives?
  • If the system experiences a massive influx of uploads (e.g., a burst of 10,000 files in one minute), how do you prevent the scan workers from being overwhelmed?
  • How would you implement a 'webhook' notification system so that external services can be automatically alerted when a scan is finished?