Katkat
Back

Derive insights and improve complaint resolutions

WayfairData Scientist · Online Assessment
EasyData ScientistOnline AssessmentAnalytics & Experimentation

Last updated

This problem tests the ability to join disparate datasets to identify trends, evaluate the effectiveness of resolution strategies, and design an A/B test to validate potential improvements in customer satisfaction.

Scenario

You are a Data Scientist at Wayfair analyzing customer support tickets. You have access to three tables:

  1. complaints: [complaint_id, user_id, category, timestamp]
  2. resolutions: [complaint_id, resolution_type, resolution_time_hours]
  3. satisfaction_scores: [user_id, complaint_id, csat_score] (Scale 1-5)

Task

  1. Root Cause Analysis: Identify which category of complaints has the lowest average csat_score and the highest resolution_time_hours. Provide the SQL logic or analytical framework to extract this.
  2. Impact Assessment: Suppose the business wants to introduce a "Proactive Refund" policy for specific categories. Explain how you would estimate the causal impact of this policy on csat_score if you only have observational historical data.
  3. Experiment Design: Propose an A/B test design to determine if a new AI-chatbot resolution path improves csat_score compared to human-only resolution, keeping the sample size and power in mind.

Constraints

  • Assume csat_score is missing for 20% of cases; explain how you would handle this missing data bias.
  • Consider that resolution_time_hours may have extreme outliers due to unresolved long-standing claims.

Example

  • If category 'Shipping' has an average CSAT of 2.1 and average resolution time of 48 hours, while 'Product Quality' has CSAT 3.5 and 12 hours, you would prioritize 'Shipping' for intervention.

Edge Cases

  • Complaints that remain unresolved (null resolution_type).
  • Users with multiple complaints in a short timeframe (serial complainers).

Hints

Hint 1

Use a JOIN operation across the three tables filtered by complaint_id to create a unified view of the customer journey.

Hint 2

For the causal inference part, consider propensity score matching or an instrumental variable approach to control for selection bias.

Hint 3

For the A/B test, identify the primary metric (CSAT) and guardrail metrics (e.g., total cost of resolutions) to ensure the intervention doesn't hurt profitability.

Follow-up questions

  • How would you account for seasonal spikes in complaints, such as during holiday sales events?
  • If the chatbot resolution significantly reduces time but lowers CSAT, how do you decide if the trade-off is worth it for the company?
  • What statistical test would you use to compare the CSAT distribution between the control and treatment groups, given that CSAT is an ordinal variable?