Design and Scale Quantitative Optimization Workflows

Context & Problem Statement

You are tasked with designing a quantitative portfolio optimization and simulation platform. The platform serves two primary use cases: running daily production optimizations for wealth management (e.g., automated tax-loss harvesting and multi-account asset location) and running multi-period backtests across thousands of portfolio variations to evaluate alpha models and rebalancing strategies.

Optimizers run complex numerical routines (such as Quadratic Programming or Mixed-Integer Programming) to balance alpha, transaction costs, tax implications (e.g., short-term vs. long-term capital gains, wash sales), and tracking error to a benchmark.

Requirements

Functional Requirements

• Optimization Workflow: Support defining and executing single-period optimizations and multi-period backtests across multiple accounts (e.g., taxable vs. tax-exempt retirement accounts).
• Point-in-Time Correctness: Ensure all inputs (prices, dividends, splits, tax lots, risk models) are strictly as-of the simulation timestamp to prevent look-ahead bias or restatement leakage.
• Lineage & Reproducibility: Record immutable lineage for all runs, including code version, input datasets, solver configuration, and output trades/weights, enabling exact deterministic replays.
• Monitoring & Validation: Automatically detect solver divergence, infeasibility, or anomalous turnover before persisting or routing orders.

Scale & Non-Functional Requirements

• Scale: Execute approximately 30,000 optimizer runs during an end-of-day rebalancing batch window within 90 minutes.
• Job Characteristics: An individual optimizer run typically consumes 1 to 2 CPU cores and takes between 3 to 15 seconds, depending on portfolio constraint complexity.
• Determinism: Given identical inputs and solver seeds, results must be bit-for-bit reproducible.
• Fault Tolerance: The system must tolerate worker crashes or transient solver failures without restarting the entire batch.

What to Cover

1. High-Level Architecture: Interaction between the orchestration engine, data services, worker pool, and storage layers.
2. Point-in-Time Data Storage & Serving: How to store and efficiently retrieve bitemporal financial data and tax lot states across 30,000 accounts.
3. Worker Orchestration & Scaling: Dynamic scheduling, worker pool sizing, caching shared data (e.g., factor covariance matrices), and mitigating the straggler problem.
4. Data Lineage & Artifact Storage: Tracking provenance and metadata for auditing and compliance.
5. Engineering & Research Trade-offs: Trade-offs between exact MIP formulation vs. heuristics for tax-loss harvesting, and distributed vs. local execution environments.