Streaming AI Responses: From Tokens to a Live Chat

Ask ChatGPT a long question and watch the screen. The answer does not appear in one block after eight seconds. Text shows up in under a second, and then it grows, word by word, as if someone is typing. That is called streaming. This note explains what is happening at each layer, from the model all the way to the chat bubble on the screen.

Note 5 gave the model tools so it could act. Here we go back to a plain chat and look at how the answer travels. Note 4 already had a first streaming endpoint (section 27 there). This note goes further. It covers why streaming feels faster, what a "chunk" really is, how chunked HTTP, SSE and WebSockets differ, how to save the history when the reply comes in pieces, and how the browser reads a stream before it has ended. The code is in Spring Boot, Node.js and Python under code/06-streaming-chat/, plus a small Node script that reads a stream the way a browser does.

Here are the words used in this note. Each one is explained again the first time it shows up, but this table is the short version:

1. The Model Writes One Token at a Time

From note 2 you know the core idea. A model looks at the text so far and predicts the next token. Then it adds that token and predicts again.

Give it this:

The capital of India is

It does not produce "New Delhi." in one step. It goes like this:

The capital of India is           -> predicts " New"
The capital of India is New       -> predicts " Delhi"
The capital of India is New Delhi -> predicts "."

Here is the same loop as a picture:

Notice the loop. The model never has a whole sentence ready to hand over. The answer exists only up to the last token it wrote.

Two everyday pictures of the same thing:

• A friend typing a long WhatsApp message. The words exist one by one, in order, even if you only see them when they press send.
• A cricket commentator. They describe ball one, then ball two. Nobody has the commentary for the whole over in advance.

This "one piece after another" nature is what makes streaming possible.

The model already produces the answer in pieces. Streaming is about not hiding that.

2. Generation and Delivery Are Two Different Jobs

Two separate things happen when an AI app answers you.

Generation is the model's job. It writes token after token.

Delivery is our app's job. It moves the text from the AI provider, through our backend, to the user's browser.

Look at the two boxes. Streaming changes the right box only. The model generates exactly the same way with or without streaming. What changes is whether delivery waits for the last token or passes each one on.

Think of a restaurant:

• The kitchen cooks the dishes. That is generation.
• The waiter brings them to your table. That is delivery.

A waiter can wait until all five dishes are ready and bring them together. Or they can bring each dish the moment it is ready. The kitchen does not cook any faster either way. Only your waiting experience changes.

Or a newspaper. The press prints it (generation). The delivery boy brings it (delivery). Faster bicycles do not make the press print faster.

3. Without Streaming: Wait, Then Everything

This is a normal, non streaming request:

The model was writing the whole time. But the backend waited until the last token was ready, and only then sent one complete answer. If that takes eight seconds, the user stares at an empty box or a spinner for eight seconds.

From the user's side it feels like this:

Send
 ...........
 ...........
 ...........
Whole answer appears at once

It is like ordering food on an app that shows no tracking at all. You see "Order placed", then nothing for forty minutes, then "Delivered". The food was moving the whole time. You just could not see it.

4. With Streaming: Pieces as They Are Ready

With streaming, each piece is passed on as soon as it exists:

The user sees:

Spring
Spring Boot
Spring Boot is a framework
Spring Boot is a framework that ...

The whole answer may still take eight seconds. But useful text appears after about 0.8 seconds, and the user is reading while the model is still writing.

This is the food app with live tracking. "Restaurant accepted", "Food is being prepared", "Rider picked up", "Rider is 2 km away". Delivery is not faster. Waiting feels much shorter because you can see progress.

It is also a live cricket score. You do not wait for the match to end to learn the result. You see every run as it happens.

Streaming does not make the model finish sooner. It lets the user start sooner.

5. Time to First Token vs Total Time

There are two different numbers here. Keep them apart.

Say a reply behaves like this. These numbers are illustrative, not measured:

Time to first token (TTFT) is about 0.8 seconds. It measures how quickly the user starts seeing output.

Total time is about 8 seconds. It measures how long the whole answer takes.

Here are both timelines side by side:

The last box is at 8 seconds in both rows. What differs is the boxes in the middle. Without streaming, the time to first token and the total time are the same number: 8 seconds.

The idea of "feels faster" has a name: perceived latency. Latency means waiting time, so this is how long the wait feels, not how long it is. On a chat screen, this is the number people notice.

Everyday versions of the same trick:

• A restaurant brings the starter while the main course is still cooking. Dinner takes as long as before, but you are eating within minutes.
• A train station board shows "Platform 4, arriving in 6 minutes". The train is not early. You are just not standing there wondering.

6. The Model Reads Before It Writes

Why is the first token not instant? Because before the model can write anything, it has to read everything it was given.

That input can be large. From note 4 you know it holds the system prompt, the earlier conversation and the new message.

Notice the two phases. Streaming helps only in phase 2. During phase 1 there is nothing to stream yet. Providers often call phase 1 "prefill".

So a longer input usually means a longer wait before the first token. A chat that sends 30 old messages on every turn makes the user wait a bit longer each time before text appears. That is one more reason to manage the context, as note 4 explained.

Think of an exam. A good student reads the whole question before writing the first word of the answer. A two line question gets an answer started quickly. A full page case study takes a minute of reading first.

Or a shop till. The cashier must scan every item before the receipt starts printing. Forty items take longer to scan than four.

7. Tokens, Provider Chunks and Network Chunks Are Different

This one causes a lot of confusion, so take it slowly.

The model makes tokens. But your app never sees them exactly as the model made them. Several layers on the way join some pieces and split others.

Four tokens became three deltas and then two browser reads. The text is the same. Only the cut points moved.

Who moves them?

• The provider may group a few tokens into one delta.
• Our backend may write two deltas before the network sends a packet.
• The network, proxies and the browser may join or split pieces again.

A piece can be a single letter, a word, a few words or part of a word. You cannot know in advance.

Here is a post office picture. You write a letter word by word (tokens). You put pages into envelopes (deltas). The post office puts envelopes into big mail bags (network chunks). The person who receives it does not care how many bags there were. They read the letter.

And a water tap. Water comes out as one smooth flow, but inside the pipe it moved in uneven pushes. You fill the glass either way.

Never write code that depends on where a chunk starts or ends. Just add each piece to what you already have.

8. Streaming Is Still One HTTP Response

A normal HTTP API works like this:

REQUEST -> server works -> server finishes -> RESPONSE (complete body)

Ask for GET /users/10 and you get one finished JSON object. That is right for a user record. It is wrong for an AI answer, because the answer grows over time.

People sometimes think streaming means leaving HTTP. It does not. It is still one request and one response. The only difference is that the server starts sending the body before it knows the whole body.

In HTTP/1.1 this is called chunked transfer. The server says Transfer-Encoding: chunked in the headers (the info lines before the body), and then sends the body in pieces. Each piece starts with its size. A piece of size zero means "that was the end".

The status and headers go first, once. Then come as many body chunks as needed. One request, one response, a body that arrives slowly.

On the wire it looks roughly like this (sizes are in hex):

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

7
Refunds
d
 usually take
d
 5 to 7 days.
0

You never write this by hand. Express, FastAPI and Spring do it for you when you write to the response bit by bit. HTTP/2 and HTTP/3 send the body in small labelled frames instead, but the idea is the same: the body arrives in parts.

A shop till is a good picture. A normal response is a cashier who prints the receipt in the back and then hands it to you. A streaming response is the till printing line by line in front of you. Same receipt, one transaction.

9. SSE: the Server Talks, the Client Listens

Plain chunked text is enough to stream an answer. But sometimes you want more structure. You want to say "this is a piece of text", "this is the end", or "something went wrong". SSE gives you that.

SSE stands for Server-Sent Events. It is a simple text format sent over one long HTTP response with Content-Type: text/event-stream. Each event is a few lines, and a blank line ends it:

data: {"delta":"Your "}

data: {"delta":"refund "}

data: {"delta":"is on its way."}

event: done
data: {}

The data: line holds the content. An optional event: line gives the event a name. Our servers use done at the end and error if the provider fails halfway.

After the question, everything flows one way. The client only listens.

Like a railway station announcement system. The station speaks, many people listen, and nobody talks back through the speaker. Or a radio commentary of a cricket match: one voice, one direction, a stream of small updates.

Why do our servers put JSON inside data:? Two small traps in the format:

• A new line ends a data: line. A reply with line breaks, such as a list, would be cut into pieces.
• SSE readers drop one space right after data:. A delta like " Boot" would arrive as "Boot", and "Spring" and "Boot" would join as "SpringBoot".

Inside JSON, {"delta":" Boot"} keeps its space and \n stays a new line.

The Spring server writes data:{...} with no space after the colon. Node and Python write data: {...}. Readers treat both the same.

10. WebSockets: Both Sides Talk

A WebSocket is a different kind of connection. It starts as an HTTP request that asks to "upgrade". After that, the connection stays open and both sides can send messages whenever they like.

Notice the client message in the middle. That is what WebSockets add.

A phone call is a WebSocket. Both people can talk at any moment. A radio is SSE. One side talks, the other listens.

For a normal chat answer you rarely need a WebSocket. After the user sends the question, almost all the data flows one way:

Client -> question
Server -> piece
Server -> piece
Server -> piece
Server -> done

Even "stop generating" does not need a WebSocket. The browser can just close the stream, and the server notices that the client went away.

WebSockets earn their place when both sides really do send a lot: live voice, a shared document many people edit, a multiplayer game. They also cost more to run. Proxies and load balancers (they sit in front of your servers and pass traffic on) need extra setup. If the connection drops, you must reconnect it yourself.

11. Which One to Pick

Here are the three options for sending an answer as it is written:

And as a decision:

Most chat apps end up with plain chunked text or SSE, not a WebSocket. The code for this note offers both plain text (/api/chat/stream) and SSE (/api/chat/sse) so you can compare them.

Streaming is data arriving bit by bit. SSE is one format for doing it. They are not the same thing.

12. Streaming Must Survive the Whole Pipeline

Here is the rule that trips up most first attempts. Streaming is not only a feature of the model. It has to work along the whole path from the model to the screen. Every step must pass each piece on right away. If one step waits to collect everything first, the user sees everything at once, and all the other steps streamed for nothing.

Each dotted line is a place where streaming quietly dies. The answer still arrives. It just arrives all at once.

A reverse proxy is a server that sits in front of your app and passes requests on to it. Common places where a buffer hides:

The servers in this note send Cache-Control: no-cache and X-Accel-Buffering: no on both streaming routes for exactly this reason.

Think of a relay race. Four runners can be fast, but if the third one stops to tie a shoe, the team is slow. Or a water pipe with a tank in the middle. Water flows in, but nothing comes out of your tap until the tank is full.

One buffering step anywhere, and the user sees the whole answer at once.

13. Spring AI: .call() vs .stream()

In Spring AI, the normal way to ask the model looks like this:

public String chat(String message) {
    return chatClient.prompt()
            .user(message)   // add the user's message
            .call()          // send it and wait for the end
            .content();      // the reply as one String
}

prompt() starts building a request. .user(message) adds the user's text. .call() sends it and waits. .content() gives back the text. The return type is String: one finished value.

For streaming you change one word, and the return type changes with it:

public Flux<String> stream(String message) {
    return chatClient.prompt()
            .user(message)
            .stream()        // instead of call(): do not wait for the end
            .content();      // now a Flux<String>, one element per piece
}

Here are the two shapes next to each other:

The interesting change is not call to stream. It is String to Flux<String>. Everything that uses this method now deals with "many values over time" instead of "one value later".

14. What Flux<String> Means

A String means: "later, you will get one piece of text".

A Flux<String> means: "over time, you will get zero, one or many pieces of text, and then a signal that says it is complete, or that it failed".

time ---------------------------------------------->
"Your"   " refund"   " is on"   " its way."   (complete)

Think of a String as a box. Someone hands you the box when it is packed. Think of a Flux<String> as a conveyor belt at a sushi restaurant. Plates come past one by one. You take each as it arrives. At some point the kitchen says "that is all".

Or a phone contact list. Downloading it as one file is a String. Your phone showing contacts as they sync, one by one, is a Flux.

A Flux can be in a few states:

Two things to notice. A Flux always ends in exactly one way: completed or failed. And nothing starts until someone subscribes, which means signs up to receive the values. A Flux is a plan for a stream. When you return it from a controller, Spring subscribes, and only then does the request to the model go out.

Other languages have the same idea under other names:

The syntax changes. The idea stays: values that become available over time.

15. Returning a Flux From the Controller

The controller barely changes. It returns the Flux instead of a String:

@PostMapping(value = "/chat/stream", produces = MediaType.TEXT_PLAIN_VALUE)
public Flux<String> stream(@RequestBody(required = false) String message) {
    String text = requireText(message);           // 400 if empty

    // Spring MVC writes each piece as soon as the Flux emits it,
    // and ends the response when the Flux completes.
    return chatService.stream(text)
            .doOnError(e -> log.error("Streaming chat failed", e));
}

What happens after return is the part worth seeing:

The controller method finishes almost at once. It returns a plan. Spring MVC then subscribes, and writes and flushes each piece to the response as it comes. "Flush" means push it out to the network now instead of keeping it in a local buffer.

One thing to know. The 200 OK status goes out with the first piece. If the provider fails after that, you cannot switch to an error status. The response can only end early. Section 18 comes back to this.

16. text/plain or text/event-stream

Both routes in the code stream. They differ in what goes on the wire.

produces = MediaType.TEXT_PLAIN_VALUE sends raw text, piece after piece:

Your refund is on its way.

The browser just sees text growing. Simple, and enough for a basic chat.

produces = MediaType.TEXT_EVENT_STREAM_VALUE sends SSE events. In Spring you return a Flux<ServerSentEvent<...>>:

@PostMapping(value = "/chat/sse", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<Map<String, String>>> sse(@RequestBody(required = false) String message) {
    return chatService.stream(requireText(message))
            .map(piece -> ServerSentEvent.builder(Map.of("delta", piece)).build())
            // a named event at the end, so the client knows it is complete
            .concatWith(Mono.just(ServerSentEvent.builder(Map.<String, String>of()).event("done").build()))
            // SSE can carry an error inside the stream; plain text cannot
            .onErrorResume(e -> Mono.just(
                    ServerSentEvent.builder(Map.of("message", "The model provider stopped.")).event("error").build()));
}

The browser's EventSource object can read SSE, but it only sends GET requests, and our question goes in a POST body. So we read SSE with fetch() too. One extra detail: a read can end in the middle of an event. So keep a small text buffer, and handle only complete events (each ends with a blank line):

let pending = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  pending += decoder.decode(value, { stream: true });

  const events = pending.split("\n\n");
  pending = events.pop(); // the last part may be incomplete; keep it for next time

  for (const event of events) {
    const lines = event.split("\n");
    const name = lines.find((l) => l.startsWith("event:"))?.slice(6).trim() || "message";
    const data = JSON.parse(lines.find((l) => l.startsWith("data:")).slice(5));
    if (name === "done") return;
    if (name === "error") throw new Error(data.message);
    bubble.textContent += data.delta;
  }
}

This is the same idea as section 7, one level up. Network chunks do not line up with events, just as they do not line up with tokens.

17. Saving the History When the Reply Comes in Pieces

Note 4's chatbot kept a history so the model could see the conversation. Without streaming, saving the reply is easy, because the reply is one finished String:

String reply = chatClient.prompt()...call().content();
history.addTurn(message, reply);   // one question, one reply

With streaming there is no finished String. There are pieces. The tempting mistake is to save each piece as it passes:

The left side has three assistant messages. The model wrote one. Those pieces are just how the reply travelled. Next turn, the model would see a strange history where it "said" three tiny things in a row.

A waiter writes your whole order on one slip, not one slip per word you said. A bank statement shows "Paid 250 rupees", not one line per coin.

So each piece does two jobs: it goes to the client now, and it goes into a buffer. When the stream completes, the buffer becomes one message.

The client got every piece the moment it existed. The history got one clean message at the end. You get both.

In code, all three versions do the same thing:

public Flux<String> stream(String message) {
    StringBuilder buffer = new StringBuilder();      // new buffer for every call

    return chatClient.prompt()
            .system(SYSTEM_PROMPT)
            .messages(contextFor(message))
            .stream()
            .content()
            .doOnNext(buffer::append)                // each piece: copy into the buffer
            .doOnComplete(() ->                      // once, only on success
                    history.addTurn(message, buffer.toString()));
}

async *stream(message) {
  const events = await client.responses.create({ ...request(message), stream: true });
  const pieces = []; // the buffer for this one reply

  for await (const event of events) {
    if (event.type === "response.output_text.delta") {
      pieces.push(event.delta); // into the buffer
      yield event.delta;        // and out to the caller, now
    } else if (event.type === "error" || event.type === "response.failed") {
      throw new Error("The model stream failed"); // leaves before the save
    }
  }

  history.addTurn(message, pieces.join("")); // only reached on success
}

async def stream(self, message: str) -> AsyncGenerator[str, None]:
    events = await self._client.responses.create(**self._request(message), stream=True)
    pieces: list[str] = []  # the buffer for this one reply

    async with events:  # closes the provider stream however we leave
        async for event in events:
            if event.type == "response.output_text.delta":
                pieces.append(event.delta)  # into the buffer
                yield event.delta           # and out to the caller, now
            elif event.type in ("error", "response.failed"):
                raise RuntimeError("The model stream failed")  # leaves before the save

    self.history.add_turn(message, "".join(pieces))  # only reached on success

doOnNext runs for each piece as it passes, and lets the piece carry on down the stream. doOnComplete runs once, only when the stream finished normally. In Node and Python, yield hands one piece to the caller and pauses. The function carries on when the caller asks for the next piece. So the line after the loop runs only if the loop ends normally. It plays the same role as doOnComplete.

The buffer is created inside the method. That matters. If it were one variable shared by all requests, two users streaming at the same time would write into the same buffer and get each other's words saved.

18. What If the Stream Breaks?

A stream can end in three ways:

Only one of the three saves anything.

If the provider fails halfway, the buffer holds half a sentence. Saving it would put a broken reply into the history, and the model would see it next turn as something it said. So nothing is saved, and the user can ask again.

If the client hangs up (closes the tab, loses signal), the server stops reading. In Spring the Flux is cancelled. In Node the loop does break. In Python the generator is closed. In all three the save line never runs. As a bonus, the model stream is closed too, so you stop paying for tokens nobody will read.

Some apps prefer to keep the partial text with a "stopped" flag. That is a fair choice. The important part is to decide on purpose, not by accident.

And remember the status code from section 15. Once the first piece is sent, the 200 is gone. The plain text route can only end early. The SSE route can say more: it sends an event: error before it ends. The Node and Python versions also wait for the first piece before they send anything, so a provider that fails right away still gets a clean 502.

19. The Browser Half: the Common Mistake

At this point the backend streams. Pieces leave the server as they are written. Now the browser has to read them the same way.

Here is the line that undoes all of it:

const response = await fetch("/api/chat/stream", { method: "POST", body: message });
const reply = await response.text(); // waits for the whole body
bubble.textContent = reply;

response.text() means: "read the complete body, join it, turn it into text, and give it to me when it has finished". For a normal API that is what you want. For a chat stream it puts the whole answer back into one block.

Same response, same bytes, two very different experiences. The whole pipeline in section 12 was streaming. One line in the browser turned it off.

It is like a live match on TV that you record and only watch after the final whistle. The broadcast was live. You chose not to watch it live.

20. What await fetch() Actually Waits For

A common belief is that await fetch(...) waits until the whole response has downloaded. It does not. It waits until the status and headers arrive. The body may still be on its way.

So after await fetch() you hold a response with:

response
 |- status      200, already known
 |- headers     already known
 |- body        a ReadableStream, still filling up

A parcel tracking page is a good picture. "Out for delivery" means the parcel is on its way. It does not mean it is at your door. fetch() resolving is "out for delivery". The body arriving is the doorbell.

21. Reading the Body: getReader() and read()

The live body is response.body, a ReadableStream. To read it, you ask for a reader and then call read() in a loop:

const reader = response.body.getReader();

while (true) {
  const { value, done } = await reader.read();
  if (done) break;   // the server ended the response
  // value holds the bytes that arrived since the last read
}

Each read() waits until some bytes are there, then gives you an object:

{ value: <some bytes>, done: false }
{ value: <some bytes>, done: false }
{ value: undefined,    done: true  }

Here is the loop as a state machine:

Notice that the loop goes round once per network chunk, not once per token or word. That is section 7 again. We never assume what one read() holds.

Think of a library book drop. You check the box again and again. Sometimes there is one book, sometimes three. When the library closes (done), you stop checking.

22. Bytes, Not Text: Uint8Array and TextDecoder

value is not a string. HTTP moves bytes, so value is a Uint8Array, a plain list of numbers from 0 to 255.

The word "Spring" in UTF-8 is six bytes, one per letter:

"Spring"  ->  83 112 114 105 110 103

To show it, you turn bytes back into text with a TextDecoder. Create one decoder for the whole reply and pass { stream: true } on every call:

const decoder = new TextDecoder(); // once, before the loop
const text = decoder.decode(value, { stream: true }); // inside the loop

Why { stream: true }? Because not every character is one byte. English letters are one byte each. Many others are not:

A network chunk can end in the middle of a character. Say the reply is "Refund of ₹250". The first chunk might end with the bytes 226 130, and the next chunk might start with 185.

With { stream: true } the decoder knows more bytes are coming. It keeps the half character and finishes it when the rest arrives. Without it, each chunk is decoded alone, and the user sees a broken mark where the rupee sign should be.

A Hindi reply like "नमस्ते" is 18 bytes. Almost any chunk boundary can land inside a letter. An English only test may never show this bug. The first Indian user will.

When the loop ends, call decoder.decode() once with no arguments. That tells the decoder the stream is over and gives you anything it was still holding.

It is like a ticket torn in two at the turnstile. Half a ticket is useless. Keep it until the other half comes, then read the whole thing.

23. One Bubble, Updated in Place

The response arrives as "Spring", " Boot", " is", " great". Those are four pieces of one assistant message, not four messages. So the chat should show one bubble that grows, not four bubbles.

This is the same rule as the history in section 17, on the screen this time. The backend saves one message. The browser shows one bubble.

Here is the whole browser function. It assumes a small addMessage(role, text) helper that adds a chat row and returns it:

async function sendMessage(message) {
  addMessage("user", message);
  const response = await fetch("/api/chat/stream", {
    method: "POST",
    headers: { "Content-Type": "text/plain" },
    body: message,
  });
  if (!response.ok) throw new Error(`Request failed: ${response.status}`);

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  // One bubble for the whole reply: created once, filled as text arrives.
  const bubble = addMessage("assistant", "").querySelector(".message");

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    bubble.textContent += decoder.decode(value, { stream: true });
    scrollToBottom(); // keep the newest line in view
  }
  bubble.textContent += decoder.decode(); // flush the last bytes
}

A few details are worth a closer look:

• textContent, not innerHTML. The reply comes from a model, and a model can be talked into writing HTML or a <script> tag. textContent always shows it as text.
• scrollToBottom() after each piece. The bubble gets taller as text is added. Without scrolling, new lines slide below the visible area. A polished app stops auto scrolling if the user scrolls up to reread.
• A real app also shows a "typing" indicator until the first piece, turns the Send button off while waiting, and removes the empty bubble if the request fails.

The same reading loop also runs outside a browser. client/stream-client.mjs in the code folder uses the exact same fetch, getReader() and TextDecoder calls in Node, and prints each piece to the terminal. Run it with --wait to see the response.text() way for comparison.

24. The Whole Thing, End to End

Here is every layer in one picture:

Notice the split after the Flux. One path goes to the user right away. The other path goes to the history, and only at the end.

Read it top to bottom once more:

1. The model writes tokens one at a time. That never changes.
2. The provider sends them out as deltas.
3. The backend receives them as a stream (Flux, async generator, async iterator) and passes each piece on.
4. At the same time it buffers them, and saves one message on success.
5. HTTP carries the pieces as chunks of one response.
6. The browser reads bytes with a reader, decodes them with a TextDecoder, and grows one bubble.

Break any step and the user sees the whole answer at once, or a broken history, or broken characters.

25. Building It: the Streaming Chat Service

Time to turn the ideas into a service. It is the Tomato support chat from note 4, cut down to what this note is about. It has five endpoints:

The same five routes as a map:

The three POST routes share one service and one history. Only the shape on the wire differs. GET /api/history is there so you can check, with your own eyes, that a streamed reply was saved once and whole.

The decisions worth copying:

26. The Code, Side by Side

The full projects are on the code page:

Browse the code for this note, or open src/FDE/code/06-streaming-chat/ on GitHub. The README there has the run and test commands for each version.

26.1 Waiting for the first piece

Sections 15 and 18 said the status goes out with the first byte. Node and Python use that. They ask for the first piece before they write anything. If the provider has already failed, they can still send a clean 502.

const pieces = chat.stream(message);
let first;
try {
  first = await pieces.next(); // throws if the provider failed right away
} catch (error) {
  response.status(502).type("text/plain").send("The model provider did not reply.");
  return;
}

response.status(200).set(STREAM_HEADERS).type(format.contentType);
if (!first.done) response.write(format.piece(first.value));
for await (const text of pieces) {
  if (clientGone) break; // closes the generator: nothing half done is saved
  response.write(format.piece(text)); // one write, one chunk
}
response.end(format.done());

async def open_stream(request: Request) -> tuple[AsyncGenerator[str, None], str | None]:
    message = await read_message(request)
    pieces = get_chat().stream(message)
    try:
        return pieces, await anext(pieces)  # raises if the provider failed right away
    except StopAsyncIteration:
        return pieces, None
    except Exception:
        raise HTTPException(status_code=502, detail="The model provider did not reply.")

# Then FastAPI sends each yielded string as its own chunk:
return StreamingResponse(body(pieces, first, render=lambda text: text),
                         media_type="text/plain", headers=STREAM_HEADERS)

In Node, pieces.next() pulls one piece, and clientGone turns true when the connection closes early. In Python, anext(pieces) also pulls one piece, and StopAsyncIteration means the stream was empty.

Spring MVC starts the response as soon as the Flux starts, so the Spring version cannot give a 502 after that point. Its SSE route turns a failure into an event: error instead.

26.2 Spring AI vs Node.js and Python

26.3 Where to look in the code

The browser style reader lives in client/stream-client.mjs. It prints the time to first text and the total time, so you can see section 5 with real numbers. They will differ from the 0.8 s and 8 s above, which are illustrative.

The tests check the things this note cares about: every piece reaches the caller in order, the history ends up with one assistant message equal to the joined pieces, and a stream that fails or is abandoned halfway stores nothing.

26.4 What this demo leaves out on purpose

Key Takeaways

• The model generates one token at a time. Streaming only changes delivery, not generation.
• Streaming lowers time to first token. Total time stays about the same. Users feel the first number much more than the second.
• The model reads all its input before the first output token. Longer context, longer wait for the first word.
• Token, provider delta, network chunk and read() result are four different things. Never depend on where a piece starts or ends.
• A stream is still one HTTP response. The body just arrives in parts (chunked transfer in HTTP/1.1).
• SSE is a one way event format on top of HTTP, good for chat. WebSockets are two way and only worth it when both sides really talk.
• Streaming is end to end. One buffering step, in the backend, a proxy or response.text(), turns it off for the user.
• In Spring AI, .stream() instead of .call() turns String into Flux<String>. Return the Flux and Spring MVC writes each piece as it comes.
• Send each piece to the client and into a per request buffer. Save one assistant message on complete. Never one per piece, never on error or hang up.
• await fetch() resolves at the headers. Read the body with getReader() and read() in a loop.
• read() gives bytes. Use one TextDecoder with { stream: true }, or a rupee sign or Hindi letter split across chunks will break.
• Grow one bubble with textContent, scroll after each piece, and never put model output into innerHTML.