Building a Chatbot: Context, Memory, Streaming and Hallucinations

ChatGPT feels like one long conversation. You say something, it answers, you say "yes", and it knows what you mean. Underneath, the model works very differently. This note explains what really happens, and the application code that makes a chat feel like one conversation.

Note 3 ended with a problem: our summariser mixed the application's instruction and the user's ticket into one message. This note starts there, with roles and the system prompt. Then it covers memory, the context window, why a prompt does not train the model, streaming, and hallucination. Along the way we build a small support chatbot for a food ordering app called Tomato, in Spring Boot, Node.js and Python. All three projects are in this repo under code/04-tomato-support-chat/.

Keep these words apart. They are easy to blur and the whole note depends on them:

1. A Model Call Is Not a Conversation

Take this exchange:

User:      My food hasn't arrived.
Assistant: I am sorry for the trouble. Would you like to wait or request a refund?
User:      I want a refund.

To the user this is one conversation. To the model it can be three separate requests with nothing linking them:

Each API call stands on its own. The model does not keep anything from request 1 when request 2 arrives. If the conversation seems continuous, it is because the application made it that way.

The LLM generates. The application creates the experience around it.

2. Keeping the Conversation History

The chatbot receives:

My food hasn't arrived.

The model answers:

I am sorry for the trouble. Would you like to wait or request a refund?

Now the user types:

Yes.

If the next request contains only "Yes.", the model has no idea what the user is agreeing to. Yes to waiting? Yes to a refund? So the application sends the earlier messages again, every time:

USER:      My food hasn't arrived.
ASSISTANT: I am sorry for the trouble. Would you like to wait or request a refund?
USER:      I want a refund.

The application stores the history and replays the relevant part on each call. That is all "the chatbot remembers what I said" means.

3. Messages and Roles

The history is not one block of text. It is a list of messages, and each message has a role. The role tells the model what kind of message it is looking at. The two roles you see in every chat are user and assistant.

User is what the person typed: a question, a complaint, some data.

USER: I ordered three hours ago. The tracking page hasn't updated for an hour.

Assistant is what the model generated earlier.

ASSISTANT: Your order appears delayed. Would you like to wait another
           10 minutes or cancel it?

Why send the model's own old replies back to it? Because the next user message often depends on them:

USER: Yes, I can wait, but if its quality is affected, I want my money back.

What is "its"? What does "yes" answer? Both only make sense next to the assistant message before them. So both sides of the conversation are part of the context.

4. Why Rules Do Not Belong in the User Message

Say the Tomato chatbot should:

• stay professional
• only answer food ordering questions
• never guess
• focus on support problems

We could glue these rules onto every user message. That is exactly what the note 3 summariser did with its "Summarize this support ticket" line. It works, but it mixes two things that come from different places:

The rules are ours and stay the same on every call. The customer's text changes every time and could say anything. They should sit in different slots. That is what the system role is for.

5. The System Role

A system message tells the model how to behave for the whole conversation:

SYSTEM:
You are a customer support executive for Tomato, a food ordering app.
Answer questions about food orders, tracking, payments, refunds and
company policy.
Be professional and empathetic.
Do not invent missing information.
If the user asks about something unrelated to food ordering,
politely say you cannot help with that.

Think of it as the application's standing instructions. The customer never sees it and never types it. It rides along with every request.

6. How System and User Messages Split the Work

The user works inside the behaviour the application set. For example:

SYSTEM: You are a programming tutor. Guide students with hints
        instead of giving complete solutions straight away.

USER:   Give me the complete solution.

The system prompt pushes the model to keep tutoring with hints even though the user asked for something else. The rules come from the app, the request comes from the user, and the model sees both.

7. A System Prompt Does Not Retrain the Model

Writing "You are a customer support assistant" does not turn the model into a trained support model. None of its weights (the numbers it learned in training) change. What happens is simpler:

The system prompt is just more text in the context. It shapes this answer. Leave it out of the next request and its effect is gone.

8. What a System Prompt Can Define

A system prompt is easier to read and change when it has clear parts:

The code for this note writes its system prompt in exactly these four sections.

9. A Prompt Is Not a Security Boundary

System instructions carry a lot of weight, but the model works by probability, not by fixed rules. It usually follows them, but not always. A user can argue with it, trick it or confuse it. So a production app does not rely on the prompt alone. It also uses:

• application logic that decides what is allowed before the model is called
• permissions checked in code, not in the prompt
• input and output validation
• structured outputs that code can check
• tools that do real work in a controlled way
• guardrails and external verification

A prompt shapes behaviour. Engineering enforces it. If refunds over a certain amount need a manager, that rule lives in code, not in a sentence the model might ignore.

10. Memory Is an Illusion

Yesterday the user said:

I teach Java.

Today they ask:

What do I teach?

The app answers "Java." It feels like the model remembered. What the app is more likely doing:

The memory belongs to the application. The model saw "I teach Java" in today's request because the app put it there.

11. The Context Window

Now the conversation looks like this:

SYSTEM:    You are a programming tutor.
USER:      Explain Docker.
ASSISTANT: Docker lets us package an application...
USER:      Explain it with an analogy.

Can we keep adding messages forever? No. A model can only take in a limited amount for one request. That limit is the context window.

Picture a single sheet of paper. On it are the instructions, the earlier conversation, some documents, a few examples and the current question. Someone hands it to you and says: "Answer the last question using only what is on this sheet." You can work with what fits on the page and nothing else. For an LLM, that sheet is the context window.

The size is measured in tokens, the same tokens from note 2.

Context window: how many tokens the model can process in one request.

12. What Goes Into the Context

A single request can hold much more than the user's last sentence:

Retrieved documents are files the app looks up for this question. Tool results are answers from other systems, such as an order status. All of it affects what comes out. The model is not answering the last line. It is answering everything it was given.

13. Context Is Not Model Knowledge

Keep two things apart.

Model parameters (also called weights, the numbers inside the model) are learned during training. They hold the patterns and skills the model picked up from its training data.

Context is what you supply for this interaction.

Context can change on every call without anyone retraining anything.

14. Input and Output Share the Window

The model also needs room for its answer. The input and the generated output both count against the same limit.

A small example. Say a model has a 16,000 token window. If the app packs 15,500 tokens of history and documents into the request, only about 500 tokens are left for the reply. Ask for a detailed answer and it will be cut off. If you fill almost all the space with input, you cannot expect a long output.

15. Conversation History vs Context Window

These are related, but not the same.

Conversation history is every exchange the application has saved: user, assistant, user, assistant, and so on. It can live in memory, a database, Redis (a fast in-memory store), the browser, or anywhere else the app chooses.

Context window is what actually goes to the model for this one reply.

The application can store far more than it sends. The code for this note does exactly that: it keeps every message and sends only the most recent ones.

16. Requests Grow Every Turn

If you send the full history each time, every request is bigger than the one before:

Turn 1: [A]
Turn 2: [A B]
Turn 3: [A B C]
Turn 4: [A B C D]

Put numbers on it. Say the system prompt is 100 tokens, each earlier turn is about 200 tokens, and the new question is 50 tokens.

The user's new message is still only 50 tokens. The request is more than ten times bigger because of what came before. From note 3 you know what that means: more input tokens, more cost, more latency.

A longer conversation means more input tokens on every request.

17. Context Management

A production app should not send everything forever. It decides what is worth sending. Common ways to do that:

Choosing what reaches the model is one of the main jobs in building a GenAI app. The Tomato code uses the first strategy, last N messages, because it is the easiest one to follow in code.

18. A Bigger Window Does Not Mean Send Everything

Some models accept a very large context. That does not mean you should fill it.

Imagine asking a colleague, "What did Rohit say about Java in yesterday's meeting?" and handing them a stack of a hundred thousand pages to find it in. They might find it. They might latch onto the wrong paragraph. It will certainly take longer. Handing them the one page with the meeting notes would have been better.

More information is not better information. Unrelated text costs tokens, adds latency and gives the model more chances to pick up the wrong thing.

The job is not to give the model more context. It is to give it the right context.

This idea comes back again and again in FDE work.

19. Same Question, Different Context

Take one question:

Explain Docker.

Change only what surrounds it.

Context A: a complete beginner

SYSTEM: You are a system architect. The user is a complete beginner.
        Avoid technical jargon. Use a simple real world analogy.
USER:   Explain Docker.

The answer will probably start with something like shipping containers.

Context B: an experienced backend engineer

SYSTEM: You are a system architect. The user is an experienced backend
        engineer who knows Linux, process isolation and deployment.
        Explain Docker technically and briefly.
USER:   Explain Docker.

Now the answer talks about images, containers, namespaces, OS level isolation and deployment.

The question did not change and the model was not retrained. The only thing that changed was the context.

20. Next Token Prediction Depends on Context

From note 2, a model repeatedly picks the next token. At a high level it is working out:

P(next token | context)

In words: given everything the model has received so far, which token is likely to come next?

Different context gives different probabilities, and different probabilities give a different answer. The model never answers the last sentence on its own. It answers the whole context.

21. Does the Model Learn From My Prompt?

Try this:

USER:      From now on, when I say "Tomato" I mean our food ordering app.
USER:      What does Tomato mean?
ASSISTANT: Our food ordering app.

It feels like you taught the model something. You did not, at least not in the machine learning sense.

Training is a loop that changes the model:

At the end of it you have a trained model: an architecture (the model's design) plus learned parameters. Learning means the parameters changed.

A normal prompt changes nothing:

Say you tell the model "My company's internal project is called GateKeeper", then ask "What is my project called?". If the first message is still in the context, it answers "GateKeeper". But nothing like this happened in between. Backpropagation is the step that works out how to adjust each parameter after an error:

Start a new chat without that message in the context and the model has never heard of GateKeeper.

Using information from the context is not the same as being trained on it.

22. Application Memory Is Not Training

When an AI app remembers you across chats, it is usually doing what we saw in section 10: look up saved facts, put them in the context, call the model.

This difference matters a lot once you build personalised apps. "The AI knows this customer" almost always means "our code fetched this customer's data and put it in the prompt".

23. Building It: the Tomato Support Chat

Time to turn the ideas into a service. The Tomato support chat has three endpoints:

On every message, the service does the same three steps: send the system prompt in its own slot, send the last N messages plus the new one, and save the turn after the reply. The HTTP layer checks for an empty message first:

Four decisions are worth copying:

24. The Code, Side by Side

The full projects are on the code page:

Browse the code for this note, or open src/FDE/code/04-tomato-support-chat/ on GitHub. The README there has the run and test commands for each version.

24.1 The system prompt

The same prompt in all three, written in the four parts from section 8:

ROLE:
You are a customer support executive for Tomato, a food ordering app.

TASK:
Find the customer's main problem and how urgent it is, then help with it.

BEHAVIOUR:
Be professional and empathetic. If the customer has a problem,
acknowledge it first, for example "I understand your frustration".

CONSTRAINTS:
Only answer questions about food orders, order tracking, payments,
refunds and company policy. For anything else, politely say you
cannot help with that.
Do not invent order details, times or amounts you were not given.

24.2 One chat turn

This is the core of the service: build the context, call the model, save the turn. If the reply is empty, the code throws before saving, so the history never holds a question with no answer.

public String chat(String message) {
    // Recent history plus the new USER message.
    List<Message> context = history.recent(maxContextMessages);
    context.add(new UserMessage(message));

    String reply = chatClient.prompt()
            .system(SYSTEM_PROMPT)   // SYSTEM role: our rules, in their own slot
            .messages(context)       // USER and ASSISTANT messages, oldest first
            .call()
            .content();

    if (reply == null || reply.isBlank()) {
        throw new IllegalStateException("The model returned an empty reply");
    }

    // Save only now, so a failed call leaves no half turn behind.
    history.addTurn(message, reply);
    return reply;
}

async reply(message) {
  const response = await client.responses.create({
    model,
    instructions: SYSTEM_PROMPT, // SYSTEM role: our rules, in their own slot
    input: [
      ...recentMessages(history, maxContextMessages), // earlier user and assistant turns
      { role: "user", content: message },             // the new message
    ],
    store: false, // we keep the conversation ourselves
  });

  if (!response.output_text) {
    throw new Error("The model returned an empty reply");
  }

  // Save only now, so a failed call leaves no half turn behind.
  saveTurn(message, response.output_text);
  return response.output_text;
}

async def reply(self, message: str) -> str:
    response = await self._client.responses.create(
        model=self._model,
        instructions=SYSTEM_PROMPT,  # SYSTEM role: our rules, in their own slot
        input=[
            *recent_messages(self._history, self._max),  # earlier turns
            {"role": "user", "content": message},        # the new message
        ],
        store=False,  # we keep the conversation ourselves
    )

    if not response.output_text:
        raise RuntimeError("The model returned an empty reply")

    # Save only now, so a failed call leaves no half turn behind.
    self._save_turn(message, response.output_text)
    return response.output_text

24.3 Keeping only the recent messages

Last N messages. The window starts at the list length minus N, never below 0. There is one small rule: never start the window on an assistant reply, because a reply without its question is an answer to nothing.

public synchronized List<Message> recent(int max) {
    int from = Math.max(0, messages.size() - max);

    // Do not start on an ASSISTANT message that lost its question.
    if (from < messages.size() && messages.get(from).getMessageType() == MessageType.ASSISTANT) {
        from++;
    }
    return new ArrayList<>(messages.subList(from, messages.size()));
}

export function recentMessages(history, max) {
  let from = Math.max(0, history.length - max);

  // Do not start on an assistant message that lost its question.
  if (history[from]?.role === "assistant") {
    from += 1;
  }
  return history.slice(from);
}

def recent_messages(history: list[Message], max_messages: int) -> list[Message]:
    start = max(0, len(history) - max_messages)

    # Do not start on an assistant message that lost its question.
    if start < len(history) and history[start]["role"] == "assistant":
        start += 1
    return history[start:]

The Java version is synchronized because Spring handles many requests at the same time on different threads, and synchronized lets only one thread use the list at once. Node and Python run one event loop (one thread that switches between tasks). Each turn is saved in one step with no await in the middle, so two requests cannot mix up the list.

24.4 Spring AI vs Node.js

The same ideas, mapped line by line:

Python follows the Node.js column almost exactly: instructions=, input=, await client.responses.create(...), and a fake client in the tests.

24.5 Where to look in the code

The tests check the things this note cares about: the system prompt goes separately, old turns are replayed, only the last N are sent while all are stored, and a failed call saves nothing.

24.6 What this demo leaves out on purpose

25. Streaming

Once you know the model writes one token at a time, streaming is easy to understand. Say the answer is:

Docker is a containerization platform.

The model does not produce that in one go. It builds it up:

Docker
Docker is
Docker is a
Docker is a containerization
Docker is a containerization platform.

If the tokens already come out one by one, the app does not have to wait for the last one before showing the first. Sending them on as they arrive is streaming.

26. Without Streaming vs With Streaming

Without streaming, the client waits for the whole answer:

With streaming, pieces flow through as soon as they exist:

The user starts reading while the model is still writing. Streaming makes the app feel faster. The model does not finish any sooner. Total generation time is about the same. What drops is the time until the first word appears, and for a chat UI that is the number users notice.

27. Streaming in Code

The only change from section 24.2 is how we ask for the reply and what we do with it. A Flux<String> is Spring's stream of text pieces that arrive over time. In Node and Python, yield hands one piece to the caller, then carries on with the next. In all three, the turn is saved only when the stream ends normally, so a stream that fails midway saves nothing.

public Flux<String> stream(String message) {
    StringBuilder reply = new StringBuilder();

    return chatClient.prompt()
            .system(SYSTEM_PROMPT)
            .messages(contextFor(message))
            .stream()                 // instead of call()
            .content()                // Flux<String>: one element per piece
            .doOnNext(reply::append)  // collect the pieces as they pass
            .doOnComplete(() -> history.addTurn(message, reply.toString()));
}

// In the controller, returning Flux<String> makes Spring MVC write each
// piece to the response as soon as it arrives.
@PostMapping(value = "/chat/stream", produces = MediaType.TEXT_PLAIN_VALUE)
public Flux<String> stream(@RequestBody(required = false) String message) {
    return chatService.stream(requireText(message));
}

// chat.js: ask for a stream of events instead of one response.
async *stream(message) {
  const events = await client.responses.create({ ...request(message), stream: true });
  let reply = "";

  for await (const event of events) {
    if (event.type === "response.output_text.delta") {
      reply += event.delta;
      yield event.delta; // pass each text piece on straight away
    }
  }
  saveTurn(message, reply); // only after the last piece
}

// app.js: write each piece to the HTTP response as it arrives.
for await (const piece of chat.stream(message)) {
  response.write(piece);
}
response.end();

# chat.py: ask for a stream of events instead of one response.
async def stream(self, message: str) -> AsyncIterator[str]:
    events = await self._client.responses.create(**self._request(message), stream=True)
    reply = ""

    async for event in events:
        if event.type == "response.output_text.delta":
            reply += event.delta
            yield event.delta  # pass each text piece on straight away

    self._save_turn(message, reply)  # only after the last piece


# main.py: FastAPI sends each yielded piece as a chunk.
return StreamingResponse(body(), media_type="text/plain")

One thing to watch with streaming: the HTTP status goes out with the first piece. If the provider fails after that, the status is already 200, so you cannot switch to a 502 (the "provider failed" status). You can only end the stream early. In Python the code waits for the first piece before it replies. In Node nothing is sent until the first write(), because the headers go out with it. In both, a provider that fails straight away still gets a clean 502.

To see it work, run any version and use curl -N, which stops curl from buffering:

curl -N -X POST http://localhost:8080/api/chat/stream \
  -H "Content-Type: text/plain" -d "How long do refunds take?"

28. Hallucination

One of the biggest limits of an LLM is hallucination: an answer that sounds right but is false.

Go back to what the model does:

At a high level, P(next token | previous context). Now notice what that loop does not do on its own:

• look up a database of guaranteed facts
• check each sentence against the real world
• refuse to output anything it has not confirmed

It produces a likely continuation, based on patterns it learned and the context it was given. Likely and true usually overlap. They are not the same thing, and hallucination lives in the gap.

29. Fluent Is Not the Same as True

Ask:

Who won the 2038 Cricket World Cup?

2038 has not happened, so the right answer is that there is no winner yet. A poorly grounded answer might say:

India defeated Australia in the final...

Why does that sound convincing? Because "X defeated Y in the final by Z runs" is a very common pattern in cricket writing. The sentence is well formed. It just describes something that never happened.

An answer can pass every check a reader instinctively runs and still be wrong:

People tend to assume that if it sounds intelligent it must be correct. That assumption is false. Fluency and truth are separate things.

Why doesn't the model just say "I don't know"? Because it does not feel certain or unsure the way a person does. It generates tokens. A phrase like "The answer is definitely..." is also just generated tokens. Modern models are trained to admit uncertainty more often, and they do, but you cannot rely on the wording.

Confident wording is not evidence that the answer is true.

30. What Causes It, and Why the Believable Ones Are Dangerous

Hallucinations tend to show up when:

An obviously wrong answer is not the dangerous kind:

Java was created by banana spaceship blue.

Nobody believes that. This one is dangerous:

Java was created in 1992 by James Gosling at IBM as part of Project Oak.

Most of it is right. James Gosling and Project Oak are real. But the sentence names the wrong company: Java came out of Sun Microsystems, not IBM. The public release was in 1995, and "created" and "released" are different events, so the date needs a check too. It reads like a fact, so it gets copied into a document, and nobody checks.

A hallucination is usually a believable fabrication, not obvious nonsense.

A useful experiment is to ask about something made up that sounds real:

Explain the Law of Constant-Time Sorting, proposed in 2018.

There is no such law. The question contains a false assumption. A model that accepts it may invent a definition, a history, examples, some maths and applications. A better behaved model will say it cannot find such a law. Either way you learn the same thing: you cannot assume answers are reliable. You have to design for it and test it.

31. How Applications Reduce Hallucination

A model on its own often lacks reliable, current or private information. The application can ground it by giving it real facts to answer from:

These cut hallucinations down. None of them gets the number to exactly zero. That is why a production GenAI system is a lot more than a model call.

32. The Bigger Picture

Building an AI app is not this:

A real one looks more like this:

The LLM is one box. The application around it decides:

• what the model sees
• what it remembers
• which instructions it follows
• what outside information it can reach
• how its answers are checked
• how reliable the final experience is

The Tomato chat covers the first four boxes: logic, system instructions, history, and a simple form of context management. Retrieval, tools and validation come later in the series.

33. Key Takeaways

1. A model call is not a conversation. The application stores the history and sends it again on every request.
2. Roles give messages meaning. System sets the rules, user is the request, assistant is what the model said before.
3. Put application rules in the system message, not glued onto the user's text. A role, task, behaviour and constraints layout keeps it readable.
4. A prompt shapes behaviour but does not enforce it. Permissions, validation and business rules belong in code.
5. Memory belongs to the application. The model only "remembers" what the app puts back into the context.
6. The context window is the model's working space for one request, counted in tokens. Input and output share it.
7. Conversation history is what you store. The context window is what you send. You can store a lot and send a little.
8. Requests grow every turn if you send everything. Manage the context: last N, summary plus recent, or only what is relevant.
9. Relevant context beats maximum context, even when the window is huge.
10. Information in a prompt does not train the model. No weights change.
11. Streaming sends tokens as they are generated. It makes the app feel faster. It does not make the model faster.
12. Fluent is not true. Hallucinations are usually believable, which is what makes them dangerous.
13. Grounding with retrieval, tools, validation and evaluations reduces hallucination. It never removes it, so reliability has to be designed.

The goal is not just to connect an app to an LLM. It is to build the right system around it.

Next: that is the end of the notes so far. Go back to the FDE notes index for the full list and the runnable code.