Streaming AI Responses: From Tokens to a Live Chat

In Topic 05, you learned how AI agents use tools to perform tasks. Now, let's understand how to make an AI chatbot show its answer as it is being generated, instead of making the user wait for the entire response.

Think about ChatGPT: you ask a question, and the answer starts appearing almost immediately, word by word.

That is called streaming.

1. What is streaming?

Without streaming, the user waits until the entire answer is ready.

With streaming, the application sends each piece of the answer to the browser as it arrives.

• Without streaming: The complete response appears at once.
• With streaming: The same response grows as new text arrives.

The model already generates tokens one after another. Streaming simply allows the application to deliver pieces without waiting for the final token.

Remember: Streaming doesn't necessarily make the model finish sooner. It lets the user start reading sooner.

2. Generation vs. delivery

These are two different jobs.

• Generation: The LLM produces tokens.
• Delivery: Your application sends the generated text through the backend to the browser.

Imagine a restaurant:

• The kitchen cooking food is generation.
• The waiter bringing dishes to your table is delivery.

The waiter can bring dishes one at a time instead of waiting for the whole meal. The kitchen hasn't become faster, but you can start eating sooner.

3. TTFT vs. total response time

Two metrics matter when measuring streaming.

Suppose a response takes 8 seconds in total.

These times are illustrative, not measured results.

One more detail: before generating a response, the model must process its input, including the system prompt, conversation history, and new message. A large input can increase the wait before the first token. Streaming helps deliver output once generation starts; it doesn't remove the initial input-processing phase.

4. Token vs. delta vs. network chunk

These three terms sound similar, but they mean different things.

• Token: A small piece of text the model works with.
• Delta: A piece of new text sent by the AI provider.
• Chunk: A piece of data delivered over the network.

For example:

Model tokens:
"Spring" | " Boot" | " is" | " great"

Provider deltas:
"Spring" | " Boot" | " is great"

Browser network chunks:
"Spring Boot" | " is great"

The text is the same, but the boundaries differ.

Why? The provider, backend, network, and browser can each group or split the data differently.

Important rule: Never assume that one network chunk equals one word or one token. Append each piece to the text you already have.

5. HTTP streaming, SSE, and WebSockets

All three can help deliver live data, but they are not the same thing.

1. HTTP chunked streaming

The server sends one HTTP response body in pieces instead of waiting for the complete body.

Good for simple streamed text.

2. SSE (Server-Sent Events)

A text-based event format over HTTP. The server can send text deltas and named events such as done or error.

Good for chat responses with explicit completion and error signals.

3. WebSockets

A persistent connection where both client and server can send messages at any time.

Good for voice, multiplayer games, and live collaborative editing.

Here's the key difference:

For a normal AI chat, the client sends a question and mostly listens to the response. HTTP streaming or SSE is usually enough; WebSockets are not automatically required.

One practical detail from the course: browser EventSource is GET-based, so if your chat API sends a question in a POST body, you can read SSE using fetch() instead.

6. How to stream a response in Node.js

With the OpenAI Responses API, enable streaming in the request and process events as they arrive.

const events = await client.responses.create({
  model: "YOUR_MODEL",
  input: "Explain streaming in simple terms",
  stream: true,
});

for await (const event of events) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write(event.delta);
  }
}

What is happening here?

1. stream: true asks the API to stream the response.
2. for await...of reads the events as they arrive.
3. response.output_text.delta identifies a new text piece.
4. process.stdout.write() prints the piece immediately without adding a new line after each one.

The code above demonstrates streaming to a terminal. In a real chat application, the backend also needs to forward each piece to the browser.

7. How the browser reads a stream

A common mistake is to use this:

const response = await fetch("/api/chat/stream", {
  method: "POST",
  body: message,
});

const reply = await response.text();

response.text() waits for the entire response body before returning the text. So even if the backend streams correctly, the browser displays everything at once.

Instead, read the response body with a ReadableStream reader.

const response = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "text/plain" },
  body: message,
});

if (!response.ok) {
  throw new Error(`Request failed: ${response.status}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();

// Create one assistant message bubble.
const bubble = addMessage("assistant", "");

while (true) {
  const { value, done } = await reader.read();

  if (done) break;

  const text = decoder.decode(value, { stream: true });
  bubble.textContent += text;
}

// Flush any remaining decoded text.
bubble.textContent += decoder.decode();

This example assumes an addMessage(role, text) helper that creates and returns a message element.

Three things to remember:

• reader.read() returns bytes, not a JavaScript string.
• TextDecoder converts those bytes to text.
• { stream: true } helps preserve characters that span multiple network chunks, such as the rupee symbol or emoji.

Also, keep appending text to the same bubble. One AI response should appear as one growing message, not dozens of separate messages.

Use textContent rather than innerHTML for untrusted model output so that generated HTML is displayed as text rather than interpreted as markup.

8. Saving conversation history correctly

Remember Topic 04: a chatbot stores conversation history so the model can use earlier messages.

Streaming introduces a small challenge: the assistant's response arrives in multiple pieces.

Suppose the model produces:

Piece 1: "Java"
Piece 2: " is"
Piece 3: " a language"

You should save one assistant message, not three.

Wrong

assistant: "Java"
assistant: " is"
assistant: " a language"

Correct

assistant: "Java is a language"

The solution is to use a buffer.

async function* streamReply(message) {
  const pieces = [];

  const events = await client.responses.create({
    model: "YOUR_MODEL",
    input: message,
    stream: true,
  });

  for await (const event of events) {
    if (event.type === "response.output_text.delta") {
      pieces.push(event.delta);
      yield event.delta;
    } else if (
      event.type === "error" ||
      event.type === "response.failed"
    ) {
      throw new Error("The model stream failed");
    }
  }

  // Save one complete assistant message after successful completion.
  history.addTurn(message, pieces.join(""));
}

Here, yield sends each piece to the caller immediately. At the same time, pieces collects the complete answer for history.

history.addTurn() is illustrative application code, not a built-in JavaScript method.

What if the stream fails?

• Completes successfully: Save the complete response.
• Provider fails halfway: Do not save a partial response as a completed answer.
• Client disconnects: Stop consuming the stream and avoid saving an incomplete answer.

The course notes that some applications may deliberately save partial responses with a "stopped" flag. The important part is to decide how partial responses should be handled.

One more backend detail: HTTP status and headers are sent before the streamed body. Once the response has started, you cannot change its status to 502 if the provider subsequently fails. SSE can communicate a named error event inside the stream.

9. The entire architecture

Streaming must work across the entire pipeline. If the backend buffers the complete response, a reverse proxy delays forwarding, or the browser calls response.text(), the user may still receive the answer all at once.

The course highlights Cache-Control: no-cache and X-Accel-Buffering: no for streaming routes, and recommends checking proxy buffering when diagnosing delays.

10. Interview revision

"What is streaming?"

Sending response pieces as they arrive rather than waiting for the full response.

"Does streaming make generation faster?"

Not necessarily. It mainly reduces time to first token and perceived latency.

"What is TTFT?"

Time from the request until the first output piece becomes visible.

"Token vs. chunk?"

A token is a model text unit; a chunk is a unit of data delivered over the network.

"SSE vs. WebSocket?"

SSE is server-to-client event streaming; WebSockets support communication in both directions.

"Why is response.text() unsuitable for live display?"

It waits for the complete response body.

"Why use TextDecoder?"

The browser receives bytes and must decode them into text.

"Why buffer the response?"

To save the streamed pieces as one complete assistant message.

"What happens if a stream fails halfway?"

The response may be incomplete; the application needs an explicit error and history-saving policy.

"What is Flux<String> in Spring AI?"

A stream of zero or more string values over time, followed by completion or failure.

Quick quiz

1. What does streaming primarily improve?

• It guarantees the model generates faster
• The user can see text sooner
• It removes the need for conversation history

2. What does TTFT measure?

• Time until the first output piece appears
• Time to save the conversation
• Total time until every response is finished

3. Which option supports server-to-client events with named done and error signals?

• SSE
• A normal completed JSON response
• A database transaction

4. Why is response.text() a problem for live streaming?

• It returns bytes only
• It waits for the full response body
• It automatically creates too many bubbles

5. How should the assistant's streamed reply be saved?

• As a separate message for every chunk
• Only the first chunk
• As one complete message after successful completion

Final memory trick: Streaming is not just turning on an option in the LLM API. The provider, backend, HTTP connection, browser reader, and UI must all pass the pieces through correctly. Buffer those pieces separately so conversation history still contains one complete assistant response.