Katkat
Back

Evaluate and Monitor a Paid Electricity Price Forecast

Weiss Asset ManagementData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to design an evaluation framework for time-series forecasting models and systematic troubleshooting of performance degradation in an automated trading pipeline.

Scenario

You are working for a power trading desk. Your team purchased a proprietary 'next-week' electricity price forecast model from a third-party vendor. The model currently operates on a rolling window basis to inform hedging decisions. Over the last five business days, the vendor model’s Mean Absolute Error (MAE) relative to real-time market settlement prices has spiked by 40%.

Task

  1. Evaluation Framework: Propose a methodology to determine if this proprietary forecast actually provides 'alpha' compared to a naïve baseline (e.g., a 7-day seasonal persistence model). How do you adjust for transaction costs and electricity market volatility constraints?

  2. Diagnostic Strategy: The model performance degraded significantly over the last five days. Outline a structured approach to identify whether this is due to data leakage, a shift in market regime, or a pipeline failure.

Deliverables

  • Define the metric for 'economic value' in this context.
  • Describe how you would conduct a 'clean' backtest that prevents look-ahead bias.
  • List the specific system and data checks you would perform to isolate the cause of the recent performance drop.

Hints

Hint 1

Consider measuring economic value by simulating the P&L difference between executing trades based on the model versus the baseline.

Hint 2

For the performance drop, check if the input features (e.g., weather data or gas prices) experienced a distribution shift or if the API feeding the model began delivering nulls or stale values.

Hint 3

Use a walk-forward cross-validation approach to ensure that your backtesting window never includes information from the future of the test period.

Follow-up questions

  • How would you handle the issue of 'look-ahead bias' if the vendor’s model incorporates macro-economic indicators that are released with a lag?
  • If the performance decline is confirmed to be a regime shift (e.g., a sudden cold snap), how would you adjust the model update frequency or feature set?
  • What statistical test would you use to determine if the difference in predictive performance between your model and the baseline is statistically significant?