Katkat
Back

Design a Scalable Audio AI Processing Platform

DavidaiSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

Tests the candidate's ability to design a resilient, distributed ML platform for audio processing that supports both interactive and batch workloads, efficient GPU scheduling, and granular failure recovery.

Scenario

You are tasked with designing the core backend platform for an audio AI service. The platform ingests user audio to perform tasks such as Speech-to-Text (ASR), speaker diarization, voice isolation, and translation. The system must support two distinct usage patterns: latency-critical interactive requests (e.g., live meeting assistants, dictation) and large-scale asynchronous batch jobs (e.g., processing multi-hour call center recordings or podcast archives).

Requirements

Functional Requirements

  • Dual Ingestion: Support both interactive requests (audio clips under 60 seconds requiring sub-2-second turnaround) and batch processing (files up to 2 hours / 1.5 GB).
  • Modular Processing DAG: Execute multi-stage pipelines (e.g., chunking/VAD -> separation -> ASR -> post-processing).
  • Segment Lineage & Fault Tolerance: Break long-form audio into chunks with explicit lineage tracking to allow idempotent, chunk-level retries instead of re-running the entire job on worker failure.
  • Quality Gates & Fallbacks: Run automated confidence scoring after model inference; route low-confidence segments to higher-accuracy fallback models.

Non-Functional Requirements & Scale

  • Scale: Ingest 50,000 hours of audio per day, with peak concurrency of 10,000 concurrent streams.
  • GPU Utilization: Maximize GPU hardware saturation through dynamic batching and mixed-precision execution.
  • Capacity Isolation: Guarantee interactive requests meet their SLA without batch jobs starving the compute pool.

What Your Design Should Cover

  1. High-Level Architecture: Ingestion tier, object storage data path, workflow orchestration, and worker topology.
  2. Large-Object & Chunk Handling: How files are ingested, chunked, stored, and streamed efficiently to workers without saturating network/memory.
  3. GPU-Aware Scheduling: Dynamic batching across variable-length audio clips, queue partitioning (interactive vs. batch), and noisy-neighbor mitigation.
  4. Data Model & Lineage: Entity relationships for jobs, segments, checkpoints, and quality evaluations to support idempotent retries.

Hints

Hint 1

Decouple the control plane from the data plane by keeping heavy audio files in object storage and passing segment references/presigned tokens through the orchestrator.

Hint 2

Address GPU utilization for variable-length audio by bucketing audio chunks by duration before feeding them to dynamic batching inference servers like Triton.

Hint 3

Isolate workloads either through dedicated GPU pools or strict priority-based scheduling with preemption to protect interactive SLAs from batch backlogs.

Follow-up questions

  • How would you optimize dynamic batching for audio segments of varying durations to avoid compute waste from excessive zero-padding?
  • How does your design handle segment boundary artifacts when audio is split across words or speaker transitions during chunking?
  • What strategies would you implement for cold-start latency when scaling GPU worker pools up during sudden traffic spikes?