Katkat
Back

Design a Synchronous Text Classification API

Hive.AiSoftware Engineer · Technical Screen
EasySoftware EngineerTechnical ScreenML System Design

Last updated

This question tests the ability to architect a robust, scalable, and maintainable synchronous ML inference service, focusing on operational reliability and multi-tenant management.

Problem Statement

Design a synchronous RESTful API that classifies incoming text strings into one of several predefined categories (e.g., Sentiment, Toxicity, or Topic). The service must be highly available, support multiple concurrent models, and maintain strict performance SLAs.

Functional Requirements

  • Multi-Model Support: The API should support versioned model artifacts (e.g., v1.2, v1.3).
  • Tenant Management: Requests must be authenticated, and API usage must be throttled per tenant ID.
  • Synchronous Inference: The client must receive the classification result immediately within the HTTP response.

Non-Functional Requirements

  • Reliability: Implement a strategy to prevent system collapse under high load (bounded overload).
  • Deployment: Define a mechanism for 'warm' deployments and instantaneous rollbacks to previous model versions.
  • Observability: Track performance and model drift while adhering to privacy standards (e.g., PII masking in logs).

Deliverables

  • Define the API contract (endpoints, request/response structure).
  • Sketch the high-level architecture (load balancer, worker nodes, artifact storage).
  • Explain how you would handle model versioning and traffic shifting.
  • Describe how the system handles transient failures and resource exhaustion.

Hints

Hint 1

Consider using a sidecar pattern or a dedicated model-server (like TorchServe or TF Serving) to abstract inference from the API logic.

Hint 2

Use a configuration service (e.g., Etcd or Consul) to manage model version metadata and dynamically route traffic to specific model artifacts without redeploying the API container.

Hint 3

Implement a 'leaky bucket' or 'token bucket' algorithm at the ingress layer to enforce per-tenant rate limits and provide clear feedback via 429 status codes.

Follow-up questions

  • How would you implement zero-downtime model updates if a new model requires significantly more GPU memory than the previous version?
  • If the inference time exceeds the client's timeout threshold, how can we improve perceived performance without changing the API to be asynchronous?
  • How do you ensure that the observability layer doesn't inadvertently log sensitive PII sent within the text classification payload?