Build and Defend a Baseline Model from a CSV

Scenario

You are handed a tabular dataset, cloud_account_monthly.csv, containing 600,000 monthly account records spanning January 2021 to December 2023 for an enterprise cloud service. Each row represents an account snapshot at month-end, with fields including account_id, snapshot_date, compute_usage_hours, storage_growth_pct, open_support_tickets, contract_tier, days_since_last_login, total_monthly_spend, and account_status (Active, Suspended, Churned).

The business team wants an automated early-warning system to identify accounts likely to churn within the next 60 days so customer success managers can intervene.

Tasks

Walk the interviewer through how you would take this CSV from exploration to a defended baseline ready for production consideration.

1. Target Definition and Leakage Prevention

• Formulate the precise target variable y and prediction window (observation window vs. outcome window).
• Identify common sources of data leakage in this schema (e.g., features updated post-churn, target leakage, global preprocessing).
• Detail how you ensure feature values reflect only information known at the cutoff timestamp.

2. Validation Split Strategy

• Propose and justify your validation scheme (e.g., standard k-fold, stratified split, temporal split, group-based split).
• Explain why naive random splitting across rows will invalidate your offline evaluation.

3. Baseline Model and Evaluation Metrics

• Define at least one rule-based or heuristic baseline and one simple statistical/ML baseline before training complex gradient-boosted trees.
• Specify your primary and secondary evaluation metrics under severe class imbalance (e.g., 2% monthly churn rate). Explain why accuracy and ROC-AUC can be misleading here.
• Describe how you select the operational decision threshold based on false-positive costs (interventions take CSM time) versus false-negative costs (lost account revenue).

4. Code Sanity Check and MLOps Plan

• If an AI coding assistant generates a preprocessing script that runs StandardScaler().fit_transform(X) and target-encodes contract_tier across the entire dataframe before splitting, explain the exact flaw and how you fix it.
• Outline your production deployment plan: batch scoring cadence vs. real-time API, drift monitoring metrics (e.g., Population Stability Index, feature drift), automated rollback criteria, and model retraining triggers.