Katkat
Back

Design an LLM-Based Coding Assistant

MetaMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteML System Design

Last updated

Design an end-to-end LLM-powered coding assistant supporting inline code completion and repository-level chat under strict latency constraints.

Scenario

Design an end-to-end intelligent coding assistant integrated into developer IDEs (similar to GitHub Copilot or internal developer tools at scale). The system must provide real-time inline ghost-text code completions as developers type, as well as an interactive multi-turn chat panel for debugging, refactoring, and answering questions about large codebases.

Requirements

Functional Requirements

  • Inline Code Completion: Low-latency single-line and multi-line completion (Fill-in-the-Middle / FIM).
  • Repository Context Awareness: Ground completions and chat answers in the developer's local workspace, recent edits, open tabs, and imported dependencies.
  • Chat & Code Modification: Multi-turn dialogue capable of generating diffs or unit tests from user queries.
  • Safety & Filtering: Real-time filtering of PII, secrets, and copyrighted or vulnerable code patterns.

Non-Functional Requirements & Scale

  • Latency:
    • Inline completion: Time to First Token (TTFT) < 100 ms; end-to-end p95 latency < 250 ms.
    • Chat interface: TTFT < 800 ms with streaming tokens.
  • Scale: 100,000 concurrent active developers generating up to 10,000 peak requests per second (RPS) for completion.
  • Availability: 99.9% service availability.

Key Components to Cover

  1. Data Pipeline & Training:
    • Pretraining, fine-tuning, and instruction alignment datasets.
    • Deduplication, syntax filtering, secret scrubbing, and FIM data preparation.
  2. Context Retrieval & Prompt Construction:
    • Gathering relevant project context (AST parsing, ctags, embeddings/vector search) within token limits.
    • Dynamic prompt assembly strategies.
  3. Model Architecture & Serving Strategy:
    • Sizing strategy (e.g., small 1B–7B models for completions vs. larger 30B–70B models for chat).
    • Latency optimization: continuous batching, PagedAttention, KV-caching, speculative decoding, and quantization.
  4. Evaluation & Guardrails:
    • Offline metrics (HumanEval, SWE-bench, pass@k, perplexity).
    • Online metrics (suggestion acceptance rate, retention, keystroke savings).
    • Licensing and security guardrails.

Hints

Hint 1

Differentiate the architecture and serving tiers between inline completion (strict sub-200ms budget) and conversational assistant queries.

Hint 2

Utilize Fill-in-the-Middle (FIM) training and local context retrieval (such as recent edit history and AST imports) to improve completion accuracy.

Hint 3

Address inference serving bottlenecks using continuous batching, prompt prefix caching, and speculative decoding.

Follow-up questions

  • How would you implement speculative decoding using a 1B draft model and a 7B target model to decrease completion latency?
  • How can you avoid leaking internal secrets or emitting verbatim GPL-licensed code in completions?
  • If offline pass@1 increases after a model update but the online acceptance rate drops by 4%, how would you debug the root cause?