Katkat
Back

Design a Large-File Analysis Report System

AirwallexSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Tests system design skills in architecting a secure, multi-tenant pipeline for large file ingestion and asynchronous batch analytics with resumable uploads and idempotent job processing.

Scenario

Global financial platforms frequently require enterprise clients to upload massive transaction ledgers, bank statements, or batch payment files (e.g., multi-gigabyte CSV, XML, or JSON files). Once uploaded, the platform must validate, parse, and analyze these records to generate detailed reconciliation and compliance reports.

Design an authenticated large-file ingestion and asynchronous analysis service that delivers resumable uploads, robust job idempotency, versioned report generation, and strict tenant isolation.

Requirements

Functional Requirements

  • Resumable Uploads: Support uploading files up to 5 GB reliably over potentially unstable connections (chunked/multipart uploads).
  • Async Analysis Engine: Trigger an asynchronous validation and processing pipeline once the upload is assembled and verified.
  • Idempotency & Deduplication: Avoid reprocessing identical files or duplicate webhook/job triggers.
  • Versioned Reports: Produce versioned analysis reports (e.g., PDF/JSON summaries of anomalies and reconciled balances) accessible to authorized tenant users.
  • Tenant Isolation: Guarantee complete data segregation between tenants in storage, transit, and compute.

Non-Functional Requirements

  • Reliability & Fault Tolerance: Interrupted uploads must be resumable without restarting from byte 0. Worker crashes must not leave jobs in a stuck state.
  • Scalability: Handle 10,000 uploads per day with peak bursts of 50 concurrent large-file jobs.
  • Security: Presigned temporary access, encryption at rest and in transit, strict RBAC.
  • Latency: File upload initialization and completion API calls must respond within 200 ms. Asynchronous analysis should complete within 10 minutes for a 5 GB file.

What to Cover

  1. API Design: Define the endpoints for initiating multipart uploads, registering chunks, completing uploads, and retrieving report status.
  2. Storage Architecture: Detail the object storage layout, chunk assembly, and lifecycle rules.
  3. Asynchronous Processing Pipeline: Describe queue architecture, worker pools, chunking/streaming parser strategies, and job state management.
  4. Idempotency & Failure Recovery: Detail how you handle network drops, duplicate requests, worker termination mid-processing, and poisoned records.
  5. Multi-Tenancy & Isolation: Explain how you enforce tenant separation at both the data tier (storage/database) and compute tier (rate limits, resource fair sharing).

Hints

Hint 1

Avoid routing gigabytes of binary payload through your application gateway; use client-to-object-storage uploads with presigned multipart URLs.

Hint 2

Derive an idempotency key using tenant ID, file content hash (e.g., SHA-256), and schema version to prevent redundant processing runs.

Hint 3

Prevent noisy-neighbor starvation at the compute tier by implementing per-tenant fair-share job queues rather than a single FIFO queue.

Follow-up questions

  • How would you handle a scenario where an uploaded 5 GB CSV has a syntax error at line 5,000,000? How should partial validation and error reporting be structured?
  • How would you design the queuing system to ensure a single tenant uploading 50 files cannot starve other tenants waiting on single-file reports?
  • How can you notify the client when the report is ready without relying on frequent client-side polling?