Design an AI Fraud-Triage Agent Over a 1 TB/Day Payments Event Stream

A payments company wants an AI agent woven into its fraud pipeline: deterministic rules and models flag suspicious transactions, and the agent investigates each flagged case — pulling related transaction history, merchant context, and device signals — before deciding whether to hold, allow, or escalate it to a human reviewer. The one number you're given up front is volume: the pipeline ingests roughly 1 TB of raw transaction and session events per day.

Treat this like any other system-design round: what's being graded is how the pieces fit together end to end — how data moves, how the service grows under load, how it recovers from a partial outage — and specifically not the internal mechanics of training the fraud model or the agent's underlying LLM.

Constraints and Clarifications

• Beyond the ~1 TB/day raw-volume figure, nothing else about the system is pinned down — surface what you're assuming rather than guessing silently.
• Treat the agent itself as a component with a request/response API, a per-call latency, and a per-call dollar cost — not something you need to design the training process for.
• Treat each underlying fraud detection model similarly: a component you call, not something you build from scratch in this conversation.

Clarifying Questions

• Is 1 TB/day raw event volume (every transaction and session event), or specifically the volume of events that reach the agent after upstream filtering?
• Does that daily figure represent every merchant on the platform combined, or just one large account — and if it's shared across many merchants, does one of them having a bad day need to stay contained rather than slowing investigations for everyone else?
• What exactly does the agent do: investigate and produce a recommendation for a human, make an automated hold/allow decision within some dollar threshold, or both depending on case severity?
• What's the acceptable latency from a transaction being flagged to the agent's decision landing — sub-second, within a few seconds, or is a few minutes acceptable for anything that isn't blocking checkout in real time?
• What's the required retention window for the raw events, the upstream flags, and everything the agent looked at — and if a chargeback dispute surfaces six months later, does someone need to be able to reconstruct exactly why a given verdict was reached?

What a Strong Answer Covers

• Gets the numbers out before designing. Converts 1 TB/day into bytes/sec and a rough event rate (depends on assumed average event size, state that assumption), and separately estimates what fraction of that volume is expected to actually reach the agent after cheap deterministic filtering — the agent should only ever see a small, already-suspicious slice of the full stream, not all of it.
• Puts the agent in the right place in the pipeline. A cheap, fast, deterministic layer (rule checks, a lightweight scoring model) sits directly in the hot path and handles the overwhelming majority of transactions in milliseconds; the agent sits downstream of that layer, invoked only for the minority of events that the fast layer flags as ambiguous or suspicious — this is what keeps both latency and per-call agent cost bounded.
• Treats the agent's input as attacker-influenced. A meaningful share of the data an investigation reads — merchant-provided fields, device fingerprints, free-text order notes — is at least partly shaped by the person the system is trying to catch. The design should avoid feeding that content directly into anything that controls an irreversible action (auto-approving a large transaction, auto-unlocking an account) without a layer that isn't purely trusting the agent's own narrative of what it found.
• Designs for idempotent, replayable investigation. The agent's work on one flagged event should be safe to retry (a crash mid-investigation shouldn't double-charge or double-release a hold) and its reasoning trail should be logged well enough that a human reviewer — or an auditor months later — can see exactly what data the agent looked at and why it reached its conclusion.
• Addresses tenancy and blast radius. If multiple merchants share the pipeline, one merchant's traffic spike or one merchant being specifically targeted by fraud shouldn't degrade investigation latency for everyone else — through either per-tenant rate limiting/quotas or physically separate processing lanes for the largest tenants.
• Converts cost per agent call into a budget. With a stated or assumed investigation rate (events/sec reaching the agent) and a stated or assumed cost per agent call, derives a daily dollar figure, and discusses what happens when that number is too high: tighter upstream filtering, a cheaper fast-path model for the bulk of ambiguous cases, or batching multiple signals into one agent call instead of one call per signal.