Ingest and Serve RAG Sessions with Up to 200 Documents

Design ingestion and serving for an enterprise Agent/RAG conversation system. A session can upload up to 200 documents, including PDF, DOCX, TXT, and CSV, totaling hundreds of megabytes. Users then ask questions across those documents and hold multi-turn conversations.

The reported targets are more than 100,000 active sessions, end-to-end response time below 3 seconds, and time to first streamed token below 500 milliseconds. Explain how you would interpret, budget, and validate these targets while preventing document processing and provider throttling from overwhelming the service.

Constraints & Assumptions

• The source does not specify document page counts, parsing complexity, output length, request arrival rate, or provider quotas.
• Active sessions are not automatically simultaneous model generations.
• Upload, parsing, chunking, embedding, retrieval, and generation are distinct stages. Clarify which stages the response targets include.
• Documents and vectors expire after 24 hours in this source's broader requirements; serving and caches must respect that boundary.

Clarifying Questions to Ask

• Do the query targets apply only after ingestion is complete, and does total response time include the final generated token?
• May queries use only the ready subset of documents, or must every document be searchable first?
• What is the distribution of concurrent queries and generated output lengths across active sessions?
• Which provider request/token quotas and regional latency constraints are available?

Part 1 — Upload and Asynchronous Ingestion

Design the path from document upload to a searchable index without buffering or parsing every file in the web gateway. Explain status reporting, retries, and the indexing boundary visible to queries.

What This Part Should Cover

• Authorized bounded uploads and durable object references.
• Idempotent parsing/chunking/embedding stages and per-document readiness.
• A consistent corpus generation or declared partial-readiness policy.

Part 2 — Meet and Measure Query Latency

Allocate the critical path and describe retrieval, context assembly, generation, and caching. Explain when semantic caching can safely reuse an answer and when it cannot.

What This Part Should Cover

• Separate TTFT and total-response budgets, with percentiles and output bounds clarified.
• Authorization-, corpus-, conversation-, and model-aware cache identity.
• Measurement of feasibility rather than a guarantee unsupported by provider or workload data.

Part 3 — Scale Through Quota Pressure

Explain admission control, provider 429 handling, and backpressure across ingestion and interactive queries.

What This Part Should Cover

• Session concurrency versus active-request concurrency.
• Bounded request/token budgets, class separation, and fair admission.
• Retry delays and visible degradation without silently claiming the latency target was met.