Katkat
Back

Audit and Extend a Time-Series ML Pipeline

VoleonMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Audit a time-series ML pipeline for leakage, tune with chronological validation, and add inference with saved preprocessing and timestamp-correct features.

You are given an ML pipeline using time-series data from a CSV. Audit it for bugs and possible feature leakage, adjust model parameters using a valid evaluation process, and extend the pipeline so it can perform inference on new data.

Constraints & Assumptions

  • The source does not provide the dataset, code, target, or confirmed bug. Treat leakage as a hypothesis to investigate, not an established diagnosis.
  • Define the prediction time, target horizon, and available information before validating features.
  • This is a pipeline engineering and modeling discussion, involving file input, data transformation, model fitting, and inference rather than a single read-only query or deterministic algorithm console.
  • If data contains several entities, preserve entity boundaries as well as time ordering.

Clarifying Questions to Ask

  • What outcome is predicted, at what timestamp, and how far into the future?
  • Which fields are actually available at that prediction time, including reporting delays?
  • Is this one series or a panel of entities? Can the same entity appear in all time splits?
  • Must inference run in batches or online, and what history is available for lagged features?

Part 1 — Audit data and features

Describe how to find implementation bugs and determine whether features contain future information.

What This Part Should Cover

  • Schema, timestamps, sorting, missingness, duplicates, and entity grouping.
  • Target construction, lag/rolling windows, as-of joins, and transform-fit boundaries.
  • Reproduction of a suspected bug with a small traceable example.

Part 2 — Tune and evaluate

Explain how to compare parameter changes without leaking information from the evaluation period.

What This Part Should Cover

  • Chronological or walk-forward splits appropriate to the target horizon.
  • Training-only fitting of learned preprocessing and validation-only parameter selection.
  • A simple baseline, appropriate metrics, and untouched final evaluation.

Part 3 — Add inference

Define the artifacts and data contract required to reproduce training-time feature logic on new inputs.

What This Part Should Cover

  • Saved model, preprocessing state, feature order, schema, and version information.
  • Historical context and timestamp-correct feature construction.
  • Missing-feature behavior, prediction outputs, and training/inference parity checks.

Hints

Hint 1

Reconstruct one prediction timestamp: For a suspect feature, list every row and field used to compute it and ask whether each was available when that prediction would have been made.

Follow-up questions

  • Why can a rolling mean still leak after a chronological train/test split?
  • What would you check if offline metrics were strong but inference performance deteriorated immediately?