Katkat
Back

Design a Cloud-Agnostic Upload Adapter

LegoraBackend Engineer · Onsite
MediumBackend EngineerOnsiteSoftware Engineering Fundamentals

Last updated

Design a provider-neutral object storage upload client that unifies cloud SDK behaviors while supporting streaming, multipart uploads, error normalization, and provider-specific escape hatches.

Problem Statement

Your team maintains applications that store binary assets across multiple cloud storage providers (such as AWS S3, Google Cloud Storage, and Azure Blob Storage). Each provider SDK handles uploads, chunking, retry semantics, and error codes differently. Directly coupling services to specific SDKs makes cloud migrations and multi-cloud architectures difficult.

Design a cloud-agnostic upload library (StorageClient) that abstracts differences between storage providers without flattening or crippling provider-specific capabilities.

Requirements

  1. Core Interface & Polymorphism:

    • Define clean abstractions for basic single-shot uploads, streaming uploads, and initiated multipart/resumable uploads.
    • Support client-supplied idempotency tokens or deterministic deduplication keys.

  2. Multipart & Streaming Lifecycle:

    • Provide a unified session lifecycle for large files (initiate, upload_part, complete, abort).
    • Allow caller control over chunk size and concurrency, while respecting individual provider limits (e.g., minimum chunk sizes).

  3. Normalized Error Handling & Retry Boundaries:

    • Translate vendor-specific exceptions (e.g., S3 NoSuchBucket, GCS 404 Not Found) into a standardized hierarchy of domain exceptions (BucketNotFoundError, TransientStorageError, StorageQuotaExceededError, etc.).
    • Clarify retry ownership: distinguish which layer (underlying cloud SDK vs. adapter vs. consumer) manages transient network retries versus session-level resumability.

  4. Provider Escape Hatches & Contract Testing:

    • Expose a clean mechanism for callers to access provider-specific metadata, storage classes, or the underlying raw client without breaking type safety.
    • Describe a contract testing strategy to ensure all provider implementations satisfy the adapter specification under identical test suites.

Deliverables

  • Class and interface definitions (in Python, TypeScript, or Go style).
  • Concrete code or pseudocode demonstrating error translation and a multipart upload session.
  • Architectural explanation addressing retry policies, escape hatches, and contract testing.

Hints

Hint 1

Model multipart uploads as an explicit state machine or session object rather than a single monolithic method call.

Hint 2

Use an adapter or mapper registry to normalize vendor error codes and HTTP statuses into a cohesive domain exception hierarchy.

Hint 3

For escape hatches, consider a typed options bag with an optional generic provider payload or a method to safely unwrap the underlying client.

Follow-up questions

  • How does your abstraction handle differences in minimum part sizes across clouds (e.g., AWS 5 MiB vs. GCS 256 KiB chunks)?
  • If an upload fails halfway through a 50 GB file, how does your adapter avoid re-uploading already committed chunks after a client crash?
  • How would you structure a contract test harness using local emulators (e.g., LocalStack, Azurite) to run in CI pipelines?