Talking to an LLM from Your Application

The last note was about what happens inside the model. This one is about everything around it. You will learn why ChatGPT is not "an LLM", what you send to a model provider, and what comes back. Then you build a small Spring Boot service that summarises support tickets. It uses Spring AI, a Java library for calling AI models. The same service is built again in Node.js and Python, so you can see the shape is the same in every language. All three projects are in this repo under code/03-ticket-summariser/, and every file can be browsed from this page. We end on a problem the next note picks up: instructions and user data are not the same thing.

1. ChatGPT Is More Than an LLM

ChatGPT and an LLM are not the same thing. When you open ChatGPT and type "Explain Docker to me like I am a beginner", you are not talking straight to a Transformer (the neural network inside the model). A whole software application sits between you and the model. ChatGPT, Claude, Gemini and DeepSeek are all applications of this kind.

An LLM is a capability. A GenAI application is software built around that capability.

Think of the LLM as an engine. An engine produces power, but an engine alone is not a car. A car also needs steering, brakes, transmission, a dashboard, sensors, navigation and safety systems. In the same way, the LLM is the intelligence engine inside a much larger application.

The LLM writes the language. The application decides what information reaches the model and what happens to the reply. For an FDE this matters because most GenAI work is about controlling that flow of information.

2. ChatGPT vs a Raw LLM

Ask: "What is the weather in Delhi right now?"

A raw LLM can only answer from what is in the request plus what it learned during training. It has no live weather data.

A GenAI application with tools can add a step:

The natural language answer comes from the LLM. The current weather comes from the application and its external tools. A tool is an ordinary program that does one fixed job and gives the same answer for the same input (this is what "deterministic" means). Examples: a weather tool, a calculator, a database query, web search, an email service, a payment service. Model plus tools can do far more than the model alone.

3. Why Does an LLM Need Tools?

An LLM generates tokens. It is not a calculator, a database, a search engine or an operating system. It is often right on a simple sum, because it saw many similar sums in training. When you need reliable computation, give it a real calculator or a code execution tool.

That leaves a gap. People write plain language ("Calculate how many * characters are in this text"). A program expects structured input, such as exact fields and values. The LLM can be the bridge:

So the LLM is useful not just for writing the final answer but for understanding what the user wants and preparing the right input for another tool.

4. The Most Important Principle

The LLM only knows what reaches the LLM.

This one sentence explains a large part of GenAI application engineering.

The model cannot see your database, your files or yesterday's chat unless your code sends them.

5. Can We Talk Directly to an LLM?

Yes. We do not need ChatGPT to talk to an LLM provider. Our own application can call the provider through an API. It is ordinary client-server communication, like POST /payments with some JSON.

There is no special connection just because AI is involved. It is an API request, like any other.

6. What Does an LLM API Request Need?

Four things.

6.1 Endpoint

Where the request goes: the URL of the service we want to use.

6.2 Authentication

Running AI models costs the provider a lot of computing power. So the provider must know who is calling and whether they are allowed to. We prove it with an API key (a secret string), sent as Authorization: <credential>.

API keys must never be exposed publicly or committed to GitHub.

6.3 Model

A provider offers several models, tuned for different needs: cost, speed, reasoning ability, context size (how much text fits in one request) and multimodality (handling more than text). So the request says which model to use, for example {"model": "Model X"}.

6.4 Input

The text for the model to work on, such as Summarize this support ticket: "I was charged twice...". That text is split into tokens and goes through everything from the previous note:

7. Our First FDE Problem

Imagine you are placed with the support team of a software company. A customer submits:

Hi team,
Our production deployment has failed three times since yesterday.
It looks like the payment-service container keeps restarting.
Customers are occasionally getting 502 errors.
We already tried restarting the deployment manually but the issue
returned after approximately twenty minutes.
Can someone investigate urgently?

Instead of making someone read every long ticket, we want an LLM to summarise it. The instruction: "Summarize this support ticket in 2 lines." A possible response:

Production payment-service is repeatedly restarting and causing
intermittent 502 errors. Manual restarts only temporarily resolve
the issue.

We build this first without Spring or Java, to show it is only an API call.

8. Talking to the Model Using Postman

Step 1: create an API key from the LLM provider. This key authenticates our application. Keep it private.

Step 2: create an HTTP request. Postman is a tool for sending HTTP requests by hand. In it: New, HTTP Request, method POST. For the OpenAI Responses API the URL is:

https://api.openai.com/v1/responses

9. Add Authentication

In Postman go to Authorization, choose type Bearer Token, and paste the API key. The request now carries:

Authorization: Bearer <API_KEY>

The server reads this header to see who is calling and whether they are allowed.

10. Specify the Content Type

The body we send is JSON, and this header tells the server so:

Content-Type: application/json

The request so far:

POST /v1/responses
Authorization: Bearer <API_KEY>
Content-Type: application/json

We are authenticated, but we have not told the model what to process yet.

11. Create the Request Body

In Postman choose Body, then raw, then JSON, and paste:

{
  "model": "gpt-5-mini",
  "input": "Explain Docker in two simple sentences."
}

Every important part answers one simple question:

12. What Happens After We Click Send?

We have now talked to an LLM without using ChatGPT.

13. Understanding the API Response

The response holds more than the generated text. This is a simplified version:

{
  "id": "resp_...",
  "object": "response",
  "status": "completed",
  "model": "gpt-5-mini",
  "output": [
    {
      "type": "message",
      "role": "assistant",
      "content": [
        { "type": "output_text", "text": "Docker packages an application..." }
      ]
    }
  ],
  "usage": {
    "input_tokens": 15,
    "output_tokens": 30,
    "total_tokens": 45
  }
}

It has two parts: the generated content, and metadata (facts about the call).

The user usually sees only the generated content. We use the metadata to track and manage each call.

14. Token Usage Becomes an Engineering Concern

Say a response reports input tokens 150, output tokens 40, total 190. Why care? Because tokens measure how much work the model did, and the bill follows from that. So the token theory from the previous note now affects real decisions:

Latency is how long the user waits. Context size is how much text fits in one request. Both depend on tokens, so tokens matter even when you are "just integrating" an existing model.

15. Input Tokens vs Output Tokens

Say you send a 10 page incident report and ask "Summarize this incident report". The model first reads the text you supplied. Those are input tokens. If it writes a 10 line summary, the new tokens it generates are output tokens.

In plain terms: input tokens are what the model reads, output tokens are what it writes.

16. Moving from Postman to Our Application

Postman showed that talking to an LLM is just an API call. But the goal is for our application to make that call, not a person clicking Send. For this series we use Spring AI.

Spring AI does not provide the intelligence. The provider owns the model. Spring AI gives you ready-made Java classes and methods that hide the HTTP details, so models are easier to plug into a Java application.

17. Create the Spring Boot Project

For the first example we only need normal web support and the OpenAI model integration:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

The web starter lets the app run an HTTP server. The OpenAI starter is the part that knows how to talk to OpenAI models.

18. Configure the API Key

In Postman we set the Bearer Token by hand. The Spring app needs the same key. We do not build the header ourselves. We write the key in configuration and Spring AI builds the header:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}

The ${OPENAI_API_KEY} placeholder tells Spring to read the value from an environment variable (a setting that lives on your machine, outside the code). Set it before you start the app:

export OPENAI_API_KEY="your-api-key"

Do not do this:

api-key: sk-xxxxxxxx

Hard coding means writing the key into the file. Anyone who can read the repo can then steal it, and it is easy to push to GitHub by mistake.

19. Configure the Model

The model also goes in application.yml:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-5-mini

Compare with Postman: {"model": "gpt-5-mini"} in the JSON body versus chat.options.model: gpt-5-mini in YAML. Same requirement, different place to write it down.

20. What Have We Replaced So Far?

One row is left: the input. It comes from our Java code, through ChatClient.

21. Creating a ChatClient

@Service
public class SummarizerService {

    private final ChatClient chatClient;

    public SummarizerService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
}

Spring passes in a ChatClient.Builder that is already set up with the key and model from application.yml. builder.build() turns it into a client, an object that can talk to the LLM provider. That is all you need for now. Ignore the Spring internals.

22. Sending Our First Prompt

The Postman body was:

{
  "model": "gpt-5-mini",
  "input": "Summarize this support ticket: Payment was deducted but my order is still processing."
}

With Spring AI:

String response = chatClient.prompt()
        .user("""
                Summarize this support ticket:
                Payment was deducted but my order is still processing.
                """)
        .call()
        .content();

Do not memorise this as syntax. Read it as steps. (The """ block is a Java text block, a string that can span several lines.)

It sends exactly the same request we sent by hand in Postman, just through Java methods.

23. Building the Summarizer Service

Now let the ticket change on each call. The method takes the ticket and adds it after the instruction:

@Service
public class SummarizerService {

    private final ChatClient chatClient;

    public SummarizerService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String summarize(String ticket) {
        return chatClient.prompt()
                .user("Summarize this support ticket in two sentences:\n\n" + ticket)
                .call()
                .content();
    }
}

The \n\n puts a blank line between the instruction and the ticket. The method now accepts any support ticket and sends it to the configured model. This is the simplified version. The repo's service builds the prompt in buildPrompt() and returns a SummarizeResponse (summary, model, usage).

24. Expose the Summarizer as an API

The request body is JSON with one field. A record is enough to receive it:

public record SummarizeRequest(String text) {
}

The controller takes that record and hands the text to the service:

@RestController
@RequestMapping("/api")
public class SummarizerController {

    private final SummarizerService summarizerService;

    public SummarizerController(SummarizerService summarizerService) {
        this.summarizerService = summarizerService;
    }

    @PostMapping("/summarize")
    public String summarize(@RequestBody SummarizeRequest request) {
        return summarizerService.summarize(request.text());
    }
}

@RequestBody tells Spring to turn the JSON body into a SummarizeRequest. The repo's controller also returns the SummarizeResponse, answers 400 for an empty text and 502 when the provider fails. The application now exposes POST http://localhost:8080/api/summarize. Example request:

{
  "text": "Payment was deducted three days ago but the order still shows Payment Processing. I contacted support twice and need the product tomorrow for my daughter's birthday."
}

25. Complete Request Flow

The controller receives request.text(), passes it to summarizerService.summarize(...), and the service runs chatClient.prompt().user(...).call().content().

That is our first real GenAI powered API.

26. Why Use Spring AI Instead of Calling the HTTP API Directly?

Technically we do not need Spring AI. We could use RestClient (Spring's HTTP client) and build the same JSON request we sent from Postman:

{ "model": "...", "input": "..." }

That works. The benefit shows up as the application grows. Later we will need multiple messages, conversation memory, streaming, structured outputs, tool calling, RAG, multiple model providers and observability. If we hand write the HTTP requests and response parsing for each of these, our code gets tied to one provider's API format. Switching providers would then mean rewriting it.

For a tiny app Spring AI saves a little code. For a larger GenAI application it saves a lot.

27. The Same Summariser in Node.js and Python

Nothing about this is Java specific. The same four ingredients (endpoint, key, model, input) look almost identical in Node.js and Python. The service is built three times in this repo, and the code lives on its own page so this note stays about the ideas:

Browse the code for this note, or open src/FDE/code/03-ticket-summariser/ on GitHub. The README there has the run and test commands for each version.

27.1 Plain fetch, exactly the Postman request

Before any SDK (a ready-made library for an API), this is the Postman request written as code. These two snippets are not in the repo. The real Node and Python code uses the openai SDK with store: false:

export async function summarize(ticket) {
  const res = await fetch("https://api.openai.com/v1/responses", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-5-mini",
      input: "Summarize this support ticket in two sentences:\n\n" + ticket,
    }),
  });

  const data = await res.json();
  // data.output[0].content[0].text holds the generated summary
  // data.usage holds input_tokens, output_tokens, total_tokens
  return data.output[0].content[0].text;
}

import os
import httpx


def summarize(ticket: str) -> str:
    res = httpx.post(
        "https://api.openai.com/v1/responses",
        headers={
            "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
            "Content-Type": "application/json",
        },
        json={
            "model": "gpt-5-mini",
            "input": "Summarize this support ticket in two sentences:\n\n" + ticket,
        },
    )

    data = res.json()
    # data["output"][0]["content"][0]["text"] holds the generated summary
    # data["usage"] holds input_tokens, output_tokens, total_tokens
    return data["output"][0]["content"][0]["text"]

Both read the key from an environment variable. In the Node version, await waits for the reply before the next line runs. Every line maps to something we set by hand in Postman: the URL is the endpoint, the Authorization header is the API key, model and input are the body. The SDKs build the URL and the header for us. We still write the model and the input.

27.2 One shape, three languages

Each project has the same three parts. The names change, the shape does not.

The HTTP layer never imports the OpenAI SDK, and the service never sees an HTTP request. This split makes testing easy. A test gives the HTTP layer a fake service, and gives the service a fake client. So mvn test, npm test and pytest all run without a key.

All three answer the same request with the same response:

{ "text": "Payment was deducted but the order still shows processing." }

{
  "summary": "The customer was charged but the order is stuck in processing.\nThey want it resolved before tomorrow.",
  "model": "gpt-5-mini",
  "usage": { "inputTokens": 61, "outputTokens": 38, "totalTokens": 99 }
}

Three decisions worth copying:

27.3 Spring AI, Node and Python side by side

Same request, same response, same token bill. The framework only changes where you write things down. That is the point of doing it in Postman first.

27.4 Where to look in the code

Start with the service in the language you know best, then read the same file in the other two.

28. Getting More Than Just the Generated Text

So far we used .content() because we only wanted the text. Sometimes we need the richer information that came with the response:

ChatResponse response = chatClient.prompt()
        .user(prompt)
        .call()
        .chatResponse();

chatResponse() returns the whole reply, not only the text. That gives us the metadata and token usage we saw in the raw API response. The three projects in code/03-ticket-summariser/ already do this: every answer carries a usage object, read from chatResponse() in Spring and from response.usage in Node and Python.

29. The Next Problem: Instructions vs User Data

Look at what our summariser does:

.user("Summarize this support ticket in two sentences:\n\n" + ticket)

Two different kinds of information are mixed into one user message:

The instruction comes from our application. The data comes from the end user. They have different purposes and different levels of trust. We wrote the instruction, so we trust it. The ticket comes from outside, so we cannot be sure what it contains. Should the model treat both exactly the same way?

That question is where the next note starts, with the idea of separating system instructions from user messages.

30. Key Takeaways

1. An LLM is a capability. ChatGPT, Claude and Gemini are applications built around one.
2. The LLM only knows what reaches the LLM. History, data, live information and actions all have to be supplied or executed by the application.
3. Tools are plain deterministic programs. The LLM turns natural language into structured tool input and turns tool output back into natural language.
4. Calling a model is a normal HTTP request with four parts: endpoint, API key, model, input.
5. The response has generated content plus metadata. Token usage in that metadata drives cost, latency and context limits.
6. Input tokens are what the model reads. Output tokens are what it writes.
7. Never hard code API keys. Use an environment variable and read it from configuration.
8. Spring AI is an abstraction over the same API call. prompt(), user(), call(), content() map directly to the Postman request, and so do client.responses.create() and output_text in Node and Python.
9. Keep the HTTP layer and the model call in separate units, and inject the client. That is what makes the service testable without an API key.
10. Mixing application instructions and user data in one message is a warning sign in the design. The fix is the topic of the next note.

Next: Building a Chatbot: Context, Memory, Streaming and Hallucinations, where the instruction moves into a system message and the summariser becomes a chatbot that remembers the conversation.