Katkat
Back

Evaluate Architecture and Capacity Trade-offs

AtlassianSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This problem tests your ability to design a document storage and processing system, specifically focusing on capacity estimation, deduplication strategies, and security considerations.

The Scenario

You are tasked with building a document management service that allows users to upload, store, and retrieve documents (PDFs, DOCX, images). The system is expected to handle 10,000 document uploads per day, with an average file size of 5 MB. Documents must be stored for at least 5 years.

Your Task

  1. Capacity Planning: Calculate the total storage required for one year of operations. How much storage would you need if the system needs to maintain two backups of all data in geographically redundant regions?
  2. Deduplication: To save space, the company wants to implement deduplication. Describe an architecture to identify identical documents at ingestion time. How does this impact the upload performance?
  3. Security: Describe how you would ensure that the stored documents are encrypted at rest and that only authorized users can access specific files. What components are needed in your architecture to handle key management and authorization checks?

Constraints & Considerations

  • Assume 10,000 files/day at 5 MB each.
  • Your system should handle occasional spikes in traffic.
  • Explain the trade-offs between 'client-side' hashing for deduplication vs. 'server-side' hashing.
  • Mention how you would handle document versioning if a user updates an existing file.

Hints

Hint 1

Start by calculating daily storage requirements (10,000 files * 5 MB) and extrapolate to a yearly total including the backup multiplier.

Hint 2

Consider using cryptographic hashing (like SHA-256) to generate a unique fingerprint for each file to detect duplicates before the writing process.

Hint 3

For security, think about the separation of concerns between your storage layer (like S3) and your application server, utilizing pre-signed URLs or dedicated proxy services.

Follow-up questions

  • If the document count grows to 10 million uploads per day, how would you modify the deduplication process to avoid a database bottleneck?
  • How would you handle a scenario where a user requests the permanent deletion of a document that is part of a shared, deduplicated storage pool?
  • What metrics would you monitor in this system to ensure that storage costs remain within budget and the deduplication service is performing efficiently?