In Topic 03, you learned how to call an LLM through an API. Now we'll solve the next problem: How do we turn separate model calls into a chatbot that feels like one continuous conversation?
We'll use the course's example of Tomato, a food-ordering app with an AI customer-support chatbot.
1. A model call is not a conversation
Imagine this exchange:
User: My food hasn't arrived.
Assistant: Would you like to wait or request a refund?
User: I want a refund.As a user, you see one continuous conversation. But the model may receive three separate API requests.
The application must send the relevant earlier messages again with the new request.
Remember: The LLM generates the answer. The application creates the conversation experience.
2. How does a chatbot remember the conversation?
Suppose the user says, "Yes."
What does "Yes" mean? It depends on what was asked before.
The application stores the conversation history and sends the relevant messages with the next request.
For example, the next request might contain:
USER: My food hasn't arrived.
ASSISTANT: Would you like to wait or request a refund?
USER: I want a refund.Now the model has the context needed to understand the request.
The stored history could live in a database, Redis, or another storage system. For a simple demo, it might live in an array in application memory.
3. Understand message roles
Chat messages have roles that tell the model what kind of information it is receiving.
| Role | Meaning | Example |
|---|---|---|
system | Instructions set by the application | "You are a helpful support assistant." |
user | The user's message | "Where is my order?" |
assistant | A previous model reply | "Your order appears delayed." |
The distinction matters because application rules and user requests have different purposes.
Why use a system message?
Imagine Tomato's chatbot should only help with food orders, payments, tracking, and refunds.
Instead of repeating those rules inside every user message, the application supplies them separately:
SYSTEM:
You are a support assistant for Tomato.
Help with food orders, tracking, payments and refunds.
Be professional and empathetic.
Do not invent order details.
USER:
My food hasn't arrived. Can I get a refund?This makes the application easier to understand and maintain.
Memory trick: System = rules. User = request. Assistant = previous reply.
4. A system prompt does not retrain the model
Suppose you tell the chatbot:
You are a customer-support assistant.
Does that permanently change the model into a customer-support model? No.
The system prompt provides instructions for the current request. It does not normally change the model's trained parameters, also called weights.
Think of the model as an employee who already has general skills. The system prompt is the job description for the task at hand, not a complete retraining process.
A good system prompt has four parts:
- Role: Who is the assistant?
- Task: What should it do?
- Behaviour: How should it respond?
- Constraints: What must it avoid?
One important warning: a system prompt shapes behaviour, but it is not a security boundary. Permissions, refund limits, and other business rules must be enforced by backend code.
5. Memory vs. context window
These two concepts are easy to confuse.
- Application memory is information your application saves for future use.
- Context window is the information the model can process in a single request, measured in tokens.
Imagine you told an application yesterday, "I teach Java." Today you ask, "What do I teach?"
The application could retrieve the saved fact and send it to the model.
It feels like the model remembered, but the application supplied the information.
| Application memory | Model training |
|---|---|
| Stores facts outside the model | Changes model parameters |
| Can be updated or deleted by the application | Requires a training or fine-tuning process |
| Facts are retrieved and added to context | Learned patterns are stored in the model's weights |
Remember: Saving information is not the same as training the model.
6. What is a context window?
Think of the context window as a sheet of paper given to the model for one answer. The model can work with the information on that sheet, but it cannot process an unlimited amount at once.
The context may contain:
- System instructions
- Earlier user messages
- Earlier assistant replies
- Current user message
- Retrieved documents
- Tool results
All of this consumes part of the context window.
The input and generated output both count against the model's context limit. If the input uses almost all available space, there may be little room left for the answer.
Conversation history is not the context window
- Conversation history: Everything the application has stored.
- Context window: The subset of information actually sent to the model for one request.
For example, an application might store 1,000 messages but send only the latest 20 relevant messages.
This is important because sending every old message on every request can increase cost and latency.
7. How do we manage a long conversation?
There are several strategies.
| Strategy | How it works | Main trade-off |
|---|---|---|
| Last N messages | Send only the most recent messages | Older facts may be lost |
| First few + recent | Keep the opening context and recent messages | Important middle details may be lost |
| Summary + recent | Summarize earlier conversation and send recent messages | The summary may omit details |
| Relevant history only | Retrieve facts related to the current question | Requires retrieval or search logic |
The key is not to send the most information possible. It is to send the most relevant information.
For example, if a customer asks about order #1234, keeping that order number may matter more than retaining 50 unrelated messages about other topics.
8. Does chatting teach the model new information?
Suppose you tell the model:
My internal project is called GateKeeper.
Then ask:
What is my project called?
It can answer "GateKeeper" if the earlier message is still in the context.
But if you start a new chat without that information and the application doesn't retrieve it, the model may not know.
Why? Because a normal prompt changes the context, not the model's trained weights.
Lifetime memory trick:
- Context = information available now.
- Application memory = information saved for later.
- Training = changing the model's learned parameters.
9. What is streaming?
Without streaming, a chatbot waits for the entire answer before showing anything.
With streaming, it displays the response as pieces arrive.
Suppose the answer is:
Docker is a containerization platform.
The model may generate it in pieces:
Docker
Docker is
Docker is a
Docker is a containerization
Docker is a containerization platform.| Without streaming | With streaming |
|---|---|
| User waits until the complete answer is ready. | First piece → next piece → next piece. The user sees text as it arrives. |
Streaming usually makes the app feel faster because the first text appears earlier. It doesn't necessarily make the model finish generating the whole answer sooner.
In a Node.js application using the OpenAI SDK, the basic idea looks like this:
const stream = await client.responses.create({
model: "gpt-5-mini",
input: "Explain Docker simply.",
stream: true,
});
for await (const event of stream) {
if (event.type === "response.output_text.delta") {
process.stdout.write(event.delta);
}
}Each text delta is a piece of the answer. A real web application would send those pieces to the frontend through an appropriate streaming response.
One production detail: once a streaming HTTP response has started, you generally cannot change its status code to 502 if the model provider fails halfway through. You need to handle mid-stream errors separately.
10. What is an LLM hallucination?
A hallucination happens when a model generates an answer that sounds convincing but is false or unsupported.
For example:
Who won the 2038 Cricket World Cup?
Since 2038 is in the future as of October 2026, there is no winner yet. A poorly grounded model might invent a winner and describe a convincing final.
Why does this happen? An LLM generates likely continuations based on learned patterns and the context it receives. It doesn't automatically verify every claim against a trusted database.
An answer can be:
- Fluent
- Grammatically correct
- Detailed
- Confident-sounding
And still be factually wrong.
Hallucinations are more likely when information is missing, outdated, ambiguous, or misleading.
Remember: Fluency is not truth. Confidence is not proof.
11. How do we reduce hallucinations?
You cannot guarantee zero hallucinations, but you can reduce the risk.
| Technique | How it helps |
|---|---|
| RAG / retrieval | Supplies relevant documents to answer from |
| Search | Retrieves current information |
| Tools and APIs | Gets real data from systems such as an order database |
| Structured outputs | Makes the response easier for code to validate |
| Validation | Checks responses against known rules or facts |
| Evaluations | Measures quality against test cases |
| Guardrails | Helps enforce safety and scope requirements |
For Tomato's chatbot, the application should retrieve the customer's real order status rather than let the LLM invent an arrival time.
A prompt saying "Never make up an order status" helps guide the model, but the stronger solution is to fetch the actual status and validate what the application tells the customer.
12. Interview revision
"How does a chatbot remember earlier messages?"
The application stores conversation history and sends relevant messages with each request.
"What is a system prompt?"
Application-level instructions that guide the model's behaviour.
"Does a system prompt retrain the model?"
No. It changes the context, not the trained weights.
"Memory vs. context window?"
Memory is stored information; context is what the model receives for one request.
"How can you control context growth?"
Send recent messages, summaries, or only relevant history.
"What is streaming?"
Sending generated text to the client as it becomes available.
"Does streaming make the model generate faster?"
Not necessarily; it mainly reduces the wait before the first text appears.
"What is a hallucination?"
A generated answer that sounds plausible but is false or unsupported.
"How do you reduce hallucinations?"
Use trusted data, retrieval, tools, validation, and evaluations.
"Is a system prompt a security boundary?"
No. Enforce permissions and business rules in backend code.
13. The complete memory map
The big takeaway from Topic 04: Building a reliable chatbot is not just about calling an LLM. It's about managing instructions, history, context, delivery, and correctness around the model.
Quick quiz
- Who usually manages conversation history between API calls?
- The LLM automatically
- The application
- The model's weights
- What does the context window represent?
- All messages ever stored
- The information the model can process for one request
- The database where memory lives
- Does a system prompt retrain the model?
- Yes, after every message
- No, it shapes the current request
- Only when streaming is enabled
- What is the main benefit of streaming?
- It guarantees zero hallucinations
- It removes the need for an API
- The user sees text before the full answer is ready
- What is the safest way to get a customer's real order status?
- Ask the LLM to guess
- Retrieve it from the order system
- Use the system prompt alone