Design GenAI Fine-Tuning and Agent Tradeoffs

Scenario

Design an internal financial research agent for a quantitative hedge fund. Quantitative researchers and analysts will use this agent to parse unstructured market reports, analyze earnings call transcripts, extract key financial metrics, and execute multi-step research workflows (e.g., pulling internal factor risk exposures, running custom backtests, and summarizing regulatory filings).

The platform must deliver accurate, deterministic data for downstream quantitative models while supporting conversational research exploration.

Functional Requirements

• Domain Adaptation: Adapt open-source foundation models (e.g., Llama-3-70B) to understand proprietary firm terminology, financial metrics, and specific tabular extraction formats.
• Agentic Workflow: Support multi-step tool use, including querying internal timeseries databases, searching vector stores of SEC filings, and invoking backtest APIs.
• Deterministic Output: Output structured JSON schemas validated against strict type constraints for programmatic ingestion.
• Guardrails & Auditability: Provide end-to-end tracing, tool execution validation, hallucination detection on numerical figures, and full reproducibility.

Non-Functional Requirements

• Throughput & Scale: Support ~1,000 concurrent researchers, averaging 50,000 agent interactions per day, with peak spikes during earnings seasons (up to 5x).
• Latency: Single-turn tool execution P95 < 2.5 seconds; multi-step agent research workflows P95 < 12 seconds.
• Resource Constraints: Maximize GPU utilization across limited internal clusters (e.g., NVIDIA H100/A100 nodes).
• Reliability: Agent loops must terminate safely with zero infinite loops and graceful fallback if intermediate tools fail.

What Your Design Should Cover

1. Model Adaptation Trade-offs: Compare Full Fine-Tuning (FFT), LoRA, QLoRA, and pure in-context learning (RAG). Discuss memory footprint, training throughput, serving costs, and catatrophic forgetting risks for each.
2. Agent Architecture & Orchestration: Define the agent pattern (ReAct, Plan-and-Solve, or DAG-based state machines). Detail state management, tool execution sandboxing, and context window budget management.
3. Schema Validation & Constrained Decoding: Compare system prompt enforcement, function calling APIs, and grammar-constrained decoding (e.g., outlines, JSON schema masks at logit generation).
4. Serving & Multi-Adapter Infrastructure: Architecture for serving multiple task-specific adapters on a shared base model (e.g., vLLM Multi-LoRA / S-LoRA).
5. Observability & Evaluation: Metrics for agent step-success rates, tool choice accuracy, and numerical hallucination auditing.