Katkat
Back

Stream LLM Output While Committing Reliable DAG Results

Qualified HealthSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Stream provisional LLM output to clients while durably validating and committing complete structured results before downstream DAG tasks execute.

A workflow runs an LLM task whose output must appear incrementally in an interactive client. Downstream DAG nodes, however, require the complete structured result before they can run.

Design a path that provides responsive token streaming while committing the final result reliably and unblocking downstream work only when that result is complete and valid.

Constraints & Assumptions

  • Client-visible partial text is not equivalent to a committed task result.
  • A client can disconnect and reconnect while generation continues.
  • Generation may fail or produce an invalid structured result after partial text has already been displayed.
  • The source supplies no exact replay-retention, latency, or output-format requirements; identify those decisions explicitly.

Clarifying Questions to Ask

  • Must a reconnecting client replay every fragment, or is receiving the latest snapshot sufficient?
  • Which schema or validation rule makes the final result usable by downstream tasks?
  • Can generation continue without a connected client, and what cancellation policy applies?
  • Must provisional output be durable, or only the final artifact and task outcome?

Hints

Hint 1

Identify the commit signal: Receiving the last visible token is not necessarily the same event as validating and durably storing the result required by the next DAG node.

Follow-up questions

  • What should the UI show if a retry replaces a failed attempt that already streamed text?
  • How can reconnecting clients distinguish missing fragments from a new attempt?
  • Why should a slow browser not hold the task's completion transaction open?