Stream and Cancel Agent Responses Without Unsafe Retries

Design streaming, cancellation, and failure recovery for a multi-turn Agent/RAG conversation system. Users must be able to stop generation, while the call chain can include an orchestrator, a model, vector retrieval, and external tools with side effects.

The source requests immediate interruption of the model call and billing when a user clicks Stop. Explain which parts can be controlled locally and which require explicit provider support.

Constraints & Assumptions

• Streamed text may already have reached the client before cancellation or failure.
• A closed browser connection does not by itself prove that remote generation stopped.
• Tool operations may write data or charge money, so arbitrary retries can duplicate effects.
• The source supplies no universal provider cancellation, refund, request-lookup, or idempotency guarantee.

Clarifying Questions to Ask

• Is server-to-client token delivery sufficient, or is bidirectional interaction on one connection required?
• Does the provider expose cancellation and authoritative request status, and what happens to in-flight billing?
• Which tools are read-only, idempotent, or irreversible?
• Should a stopped turn remain as partial history, and can a later retry create a new attempt?

Part 1 — Stream and Stop

Compare SSE and WebSocket for this application. Trace a stop request from the UI through stream delivery, orchestration, model execution, and pending tools.

What This Part Should Cover

• Protocol choice based on traffic direction and operational behavior.
• An authorized cancellation operation identified by session, turn, and attempt.
• Immediate local suppression versus confirmed upstream cancellation and billing uncertainty.

Part 2 — Recover the Agent Chain

Describe deadlines, retries, durable operation identity, and partial outcomes when a component times out or crashes.

What This Part Should Cover

• Different retry treatment for retrieval and side-effecting tools.
• Attempt fencing and a recorded canceled, failed, complete, or uncertain outcome.
• No duplicate tool effects caused by blindly replaying an entire agent turn.