Analyze a Regression Dataset and Build a Safe File-Processing Utility

Problem Statement

In quantitative research pipelines, raw incoming datasets must be preprocessed deterministically without mutating original source files. Furthermore, high-dimensional regression models require screening out multi-collinear and redundant features before training.

You need to implement two modular utilities:

1. sanitize_dataset(src_path: str, dst_path: str, required_columns: list[str]) -> int

   • Read a CSV file from src_path and write a sanitized version to dst_path.
   • Filter out rows that have null/empty values in any of the required_columns or cannot be parsed as floats in numeric columns.
   • Ensure atomic and non-destructive writes: write to a temporary file first, and only replace dst_path upon full success. The original src_path must never be modified.
   • Return the count of valid rows retained.

2. select_uncorrelated_features(df: pd.DataFrame, target_col: str, collinearity_threshold: float = 0.85) -> list[str]

   • Calculate the absolute Pearson correlation of each candidate feature column with target_col.
   • Rank candidate features in descending order of their absolute correlation with target_col.
   • Greedily retain features: add the top-ranked feature, then iterate through the remaining candidates. A candidate feature is only included if its absolute pairwise correlation with all currently selected features is strictly less than collinearity_threshold.
   • Return the list of selected feature column names in order of selection.

Function Signatures

import pandas as pd

def sanitize_dataset(src_path: str, dst_path: str, required_columns: list[str]) -> int:
    pass

def select_uncorrelated_features(
    df: pd.DataFrame,
    target_col: str,
    collinearity_threshold: float = 0.85
) -> list[str]:
    pass

Constraints

• df contains N rows and M numeric columns (2 <= N <= 100,000, 2 <= M <= 500).
• 0.0 <= collinearity_threshold <= 1.0.
• All features in df (excluding target_col) have non-zero variance.
• If two candidate features tie in correlation with the target, break ties alphabetically by column name.

Example

# Input DataFrame
# Feature correlations with target 'y':
# 'feat_A': 0.90
# 'feat_B': 0.80 (corr with feat_A is 0.92)
# 'feat_C': 0.50 (corr with feat_A is 0.10)

selected = select_uncorrelated_features(df, target_col='y', collinearity_threshold=0.85)
# Output: ['feat_A', 'feat_C']
# 'feat_B' is discarded because abs(corr('feat_A', 'feat_B')) = 0.92 >= 0.85.

Edge Cases to Consider

• Write operation interrupted midway (temp file should be cleaned up; destination should not be corrupted).
• Candidate features having identical correlation with the target.
• A feature having zero variance (standard deviation of zero leads to NaN correlation).