Design an ML Forecasting and Pacing System for Ad Bidding

Scenario

On a major professional social network, advertisers run targeted campaigns (e.g., Sponsored Content, Job Posts) with specific daily or lifetime budgets and bidding objectives (e.g., target CPA, target CPC, or maximum delivery). Without intelligent pacing, campaigns often exhaust their budgets early in the morning during non-optimal traffic spikes, missing valuable impressions later in the day. Conversely, overly conservative pacing results in budget under-delivery.

Design an end-to-end ML forecasting and budget pacing system that smooths delivery throughout the day, optimizes advertiser ROI, and deterministically prevents budget overspend.

Requirements

Functional Requirements

• Traffic & Price Forecasting: Forecast inventory volume and clearing price distributions per audience segment over a 24-hour horizon.
• Dynamic Pacing Strategy: Dynamically compute pacing parameters (such as bid multipliers or auction participation probabilities) per campaign to spend the budget evenly or proportional to high-value opportunities.
• Deterministic Budget Enforcement: Guarantee zero budget overrun at the advertiser level regardless of ML errors or prediction lag.
• Time-Correctness & Delayed Feedback: Train forecasting and conversion models without lookahead bias, accounting for delayed attribution (e.g., clicks and conversions happening minutes to days later).
• Fallback & Safety: Automatically revert to safe heuristics during model inference failures or sudden auction market drift.

Non-Functional Requirements

• Scale: 500,000 active daily campaigns; 5 billion auction opportunities per day (~60,000 QPS average, 150,000 QPS peak).
• Auction Latency: Bid modification and pacing checks during the live auction must execute in under 5 ms (p99).
• Update Frequency: Pacing coefficients should refresh every 1 to 5 minutes based on live spend tracking.

Discussion Points

Your proposal should address:

1. High-Level Architecture: Separation between the offline ML training pipeline, the near-line pacing control loop, and the online real-time bidding path.
2. ML Formulation: Feature engineering, loss formulation, and model architecture for supply/price forecasting, alongside the feedback control mechanism (e.g., PID controller, dual-gradient descent) for pacing.
3. Deterministic Guardrails: Architecture of the real-time budget counter and kill-switch mechanism decoupled from ML scoring.
4. Data Pipeline & Calibration: How to handle delayed attribution labels without leaking future states, and how to calibrate pacing parameters.
5. Experimentation & Monitoring: Strategy for A/B testing pacing algorithms in a shared auction marketplace and detecting distribution drift.