Katkat
Back

Design Jira bug-to-team classification system

AtlassianMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end ML pipeline for classifying incoming Jira tickets to appropriate engineering teams based on textual and historical metadata.

Problem Statement

Design a system that automatically routes incoming Jira bug tickets to the most appropriate engineering team. Atlassian manages thousands of teams across various products, and misrouting leads to significant delays in resolution time.

Requirements

  • Functional: Automatically predict the 'Assignee Team' for new tickets. Provide an 'I am unsure' confidence threshold where human intervention is triggered.
  • Non-functional: The system must handle high throughput (thousands of tickets per minute). Low latency is important, but a 1-2 second delay for classification is acceptable. Provide a feedback loop to improve the model over time.

Scale Assumptions

  • 500,000+ new tickets per day across the entire platform.
  • 5,000+ distinct engineering teams.
  • Tickets contain a title, description, reporter, component tags, and associated system logs.

Deliverables

  1. Data Model: Define the features used to represent a ticket.
  2. Model Architecture: Suggest a model approach (e.g., embeddings, LLM-based, or ensemble).
  3. Human-in-the-loop: Describe the routing logic when model confidence is low.
  4. Evaluation: Define metrics to measure success, accounting for both classification accuracy and the cost of misrouting.

Hints

Hint 1

Consider how to handle the cold-start problem for newly formed teams or products with little historical data.

Hint 2

Think about the trade-off between using a large, complex model (like a transformer) versus a lightweight model for initial routing.

Hint 3

Address how to handle 'concept drift' when team responsibilities or project structures change over time.

Follow-up questions

  • How would you handle a scenario where a bug could legitimately belong to two different teams?
  • If you decide to fine-tune an LLM for this task, how would you manage the data privacy and token cost implications?
  • How would you design an A/B testing framework to measure the impact of this system on the Mean Time to Resolution (MTTR)?