Katkat
Back

Review Preprocessing Code and Tests

Capital OneData Scientist · Onsite
EasyData ScientistOnsiteCoding & Algorithms

Last updated

This problem tests your ability to identify common pitfalls in data science pipelines, specifically concerning data leakage, handling missing values, and software engineering best practices for reproducibility.

Scenario

You are reviewing a colleague's script designed to preprocess a dataset before training a fraud detection model. The dataset contains transaction amounts and customer metadata.

The Code snippet

import pandas as pd
from sklearn.impute import SimpleImputer

def preprocess(df):
    # 1. Fill missing transaction amounts with the global mean
    mean_val = df['amount'].mean()
    df['amount'] = df['amount'].fillna(mean_val)
    
    # 2. Remove outliers
    threshold = df['amount'].quantile(0.99)
    df = df[df['amount'] <= threshold]
    
    return df

Your Tasks

  1. Identify Bugs: Review the code for logic errors that would invalidate a machine learning model's performance in a real-world production environment.
  2. Improvement: Briefly explain how you would restructure this code to handle both train and test splits correctly.
  3. Engineering: Explain how you would ensure the preprocessing environment is reproducible for other data scientists on the team.

Constraints

  • Assume the dataset is roughly 1GB.
  • The model will be used for real-time inference.

Hints

Hint 1

Consider whether calculating the mean and quantiles on the entire dataset leads to 'information leakage' from the test set into the training set.

Hint 2

Think about how you would handle new, unseen data points arriving in production that were not present in your training set.

Hint 3

Consider the difference between a standalone script and a version-controlled package or containerized environment.

Follow-up questions

  • How would you unit test the `preprocess` function to ensure it handles edge cases like all-null columns or empty dataframes?
  • If the 'amount' feature is highly skewed, is using the global mean an appropriate imputation strategy? What alternatives would you suggest?
  • How would you integrate this preprocessing step into a scikit-learn Pipeline object to ensure it is applied consistently during cross-validation?