How Large Language Models Work

From tokens to Transformers. The previous note ended with "GPT style models generate text by repeated next token prediction". This note opens that sentence up and follows one piece of text through the whole pipeline. You do not need the maths to build on an LLM. You do need to know the shape of what happens inside, because tokens, context and sampling show up later as cost, latency and bugs.

1. The Basic Idea Behind an LLM

At the lowest level an LLM does one simple thing:

Given everything written so far, predict what should come next.

I drink coffee every ____

Likely completions: morning, day, evening. Add context:

I drink coffee every morning before going to ____

Now: work, office, college, gym. Add more:

I work remotely as a software engineer. Every morning I drink coffee before
going to my ____

Now "desk" becomes very likely. The next piece of text is not decided by the last word alone. It depends on all the context the model can see.

2. LLMs Predict the Next Token

Input: The capital of India is. Inside, the model produces something like:

(The numbers are made up, the shape is real. Sections 19 and 20 show where they come from.) One token is picked and added to the end:

The capital of India is  ->  The capital of India is Delhi

Now the model predicts what comes after "Delhi", and so on. This is the first mental model:

One correction: an LLM does not predict the next word. It predicts the next token. A token is a small piece of text. The next sections explain why.

3. Why Text Must Become Numbers

A computer can only do maths: add, multiply, and combine numbers. So "I love programming" has to become numbers before the model can touch it.

Imagine a tiny vocabulary: cat, dog, apple, car. Give each an id:

cat -> 1    dog -> 2    apple -> 3    car -> 4

Now the computer can tell the words apart. But these numbers carry no meaning. car = 4 and cat = 1 does not mean a car is bigger than a cat, or that a car is four cats. Think of student roll numbers. Rohit is 104 and Aditya is 237. That does not make Aditya "more student" than Rohit.

Token id ≠ meaning of the token. The id only says which token we are talking about.

4. What Exactly Is a Token?

Beginners often assume one word = one token. Not always. Depending on the tokenizer (the tool that cuts text into tokens), a short phrase can become several tokens, and one word like "Elephants" may be split into more than one.

A token can be a whole word, part of a word, punctuation, a symbol, a space, or any other piece of text that shows up often.

You will also hear "1 token ≈ 4 characters". That is a rough guess for English, handy for sizing. It is not the definition of a token.

5. Why Not Simply Use Words?

Say every complete word got its own entry in the vocabulary: Automatically -> 43. Now think about Automatically, Automatic, Automate and Automation. Each needs its own entry. People also keep inventing new terms, names, abbreviations and technical words. The vocabulary would grow huge and still miss words.

6. Why Not Use Individual Characters?

Go to the other extreme: one token per character.

programming  ->  p r o g r a m m i n g

The vocabulary is tiny, but every text becomes very long and slow to process. Modern tokenizers sit in the middle:

Individual characters  <---  Tokens  --->  Whole words

For example, "unbelievable" might become un + believ + able. The exact split depends on the tokenizer. Different models use different tokenizers, so:

Same sentence + different tokenizer = potentially different tokens

Common text is stored in few tokens. Unusual text needs more pieces. This is also why API providers bill in tokens rather than words.

7. From Text to Token IDs

A simplified tokenizer vocabulary:

So there are three different things: the original text, the tokens, and the token ids. When the model generates, the lookup runs in reverse. A predicted id turns back into readable text: 4381 -> programming.

8. Why Word Order Matters

Dog bites man.      vs      Man bites dog.
Rohit teaches Aditya.  vs   Aditya teaches Rohit.

Same words, different meaning. So the model cannot treat a sentence as a loose bag {Rohit, teaches, Aditya}. It must know where each token sits:

So the input to the Transformer has to carry two things:

Token representation + Position information -> Position aware representation

Position alone is still not enough, because meaning also depends on context.

9. Context Changes Meaning

I deposited money at the bank.      -> a financial institution
We sat on the bank of the river.    -> the side of a river

One fixed representation of "bank" cannot fit both. In the first sentence "deposited" and "money" give the clue. In the second, "river" and "sat" do. So a token cannot be understood alone. Its representation must change with the words around it. Embeddings and attention exist to do this.

10. Embeddings: Meaningful Numbers for Tokens

A token id only says which token we have. The model needs something richer to compute with. Take king, queen, banana, laptop. King and queen are close in meaning. Banana and laptop are not. Imagine each word scored on a few features:

King and Queen have almost the same row: similar meaning, similar numbers.

These are not real embedding dimensions. Real models do not have clean axes like "Royalty" or "Food". The model learns its own features during training. So instead of carrying king -> 781 around, the token gets a list of numbers called a vector:

king -> [0.41, -0.82, 1.34, 0.15, ...]

That vector is an embedding. The pipeline so far:

11. Why Embeddings Alone Are Not Enough

The starting embedding of a token is general. It is the same wherever the token appears. But the meaning needed for this sentence depends on the other tokens. "bank" starts with the same embedding in both sentences above. After the model looks at the context, the two versions should end up different.

Initial representation of "bank"
+ information from other tokens
-> context specific representation

Attention is the step that does this.

12. What Attention Does

Rohit gave Aditya his laptop because his computer was broken.

To understand "his", you look at other words and ignore the rest. Attention lets a neural network do the same. Each token asks:

Which other tokens matter to me, and how much should they change my representation?

The animal didn't cross the street because it was too tired.   -> "it" = animal
The animal didn't cross the street because it was too wide.    -> "it" = street

Change one word and what "it" points to changes. Attention is dynamic. The model has not memorised "it" -> animal. The link is worked out fresh for each sentence.

13. Query, Key and Value

These are the three famous words in attention. A search engine is a good analogy. You search for "best Java course":

• Your search text is the Query.
• The engine compares it with a short description of each page. These descriptions are like Keys.
• From the pages that match, it takes the useful content. That content is like the Values.

In a real Transformer these are not English questions. They are vectors.

14. Every Token Produces Q, K and V

For "The cat sat on the mat", every token gets three learned vectors:

Why three? The same information is useful in different ways, depending on the question. Think about one person. For "Who knows Java?" their technical skills matter. For "Who lives closest to me?" their location matters. For "Who should teach this class?" you care about subject knowledge, communication and availability. Same person, different facts for each question.

Q, K and V give attention a separate learned vector for asking, matching and contributing information.

15. The Core Self-Attention Process

Say the model is working on the token "it". It takes Query("it") and compares it with the Key of every other token:

Query("it")
  vs Key("animal")   -> high relevance
  vs Key("street")   -> medium relevance
  vs Key("because")  -> low relevance

The relevance scores become weights, and the matching Values are mixed together in those proportions:

New representation of "it"
  ≈ 0.60 × Value(animal)
  + 0.20 × Value(street)
  + 0.05 × Value(cross)
  + ...

The result is a new vector for "it" that now holds information from its context. This is self-attention: tokens in one sequence attend to other tokens in the same sequence.

16. What Is a Transformer?

A common mistake: Transformer = attention. Not quite. Attention is one important part inside a Transformer. A Transformer is a stack of the same kind of block, repeated:

GPT style models have many such layers. Each block holds several smaller neural network steps, not just one.

17. Why Multiple Transformer Layers?

Some links in language are simple and some are abstract.

The developer couldn't deploy the application because the server ran out of memory.

A direct link is server <-> memory. A more abstract idea is "deployment failed because the server lacked resources". The model cannot see every useful link in one step. So it refines the representations again and again:

Layer 1: basic relationships
Layer 2: richer combinations
Layer 3: more contextual relationships
...
Final contextual representation

Think of photo editing. You start with the raw image, then fix exposure, contrast, colour, masks and details. Each pass improves the last one. Transformer layers do the same to token representations.

18. What Travels Through the Layers?

After each self-attention step, every token has an updated vector:

Token 1 -> updated vector
Token 2 -> updated vector
Token 3 -> updated vector

These vectors move on into the next blocks and are changed again. By the final layers, each token's vector holds a lot: the token itself, its position, the words around it, and how it relates to them.

19. From the Final Representation to the Next Token

Input: The capital of India is. After all the layers, the last position has a contextual vector, say H_final = [0.4, -1.2, 0.8, ...].

Now the model must answer: out of every token in the vocabulary, which comes next? The vocabulary is the full list of tokens the model knows. If it has 100,000 tokens, the model gives each one a score:

Delhi   ->  12.8
Mumbai  ->   9.8
Kolkata ->   9.3
Punjab  ->   8.8
banana  ->  -4.1
...

These raw scores are called logits. They are not probabilities yet. A higher score only means a better fit, and a score can be negative.

20. Softmax: Turning Logits into Probabilities

Scores like 12.8 and -4.1 are hard to choose from. We want percentages that are never negative and add up to 100%. Softmax is the function that does this. The highest score gets the biggest share. Low scores like "banana" get a tiny one:

These come from running softmax on the five logits above. The ranking follows the scores. A real vocabulary has many more tokens, which would share a tiny remainder. This is one of the most important ideas behind an autoregressive LLM (one that generates a token at a time):

Given the preceding tokens, produce a probability distribution for the next token.

The model is not directly "thinking" Write an essay or Explain Java. It only repeats given context -> predict next token distribution. A long answer comes from doing this many times, plus what the model learned in training.

21. Autoregressive Generation

Prompt: The capital of India is. The model produces probabilities, "Delhi" is selected, and the text becomes The capital of India is Delhi. Run the prediction again:

"."      -> 75%
","      ->  8%
"and"    ->  5%
"which"  ->  4%

"." is selected, giving The capital of India is Delhi. and the process repeats. So the model does not write a whole answer in one go. It builds it one token at a time:

This is autoregressive generation. Auto means the model feeds its own earlier output back in as input. Regressive means each new prediction depends on what came before.

22. Why ChatGPT Can Stream Text

In a chat interface the text appears bit by bit:

Java is
Java is a
Java is a programming
Java is a programming language

This works because the model really does produce text one token at a time. An application can send each token, or a small chunk, to the screen as soon as it arrives. Generation happens one step at a time, and the interface can show each step as it happens.

23. Which Token Gets Selected?

Suppose the model outputs:

The model only gives these numbers. Something else must choose one token, and there are several ways to do it. They belong to the decoding step, not the model.

24. Greedy Decoding

The simplest way: always pick the highest probability token. Java at 40% wins every time. It is "greedy" because at each step it grabs whatever looks best right now.

The problem: language often has many good continuations. Always taking the top one makes text predictable, repetitive and dull.

25. Sampling

Instead of always taking the top token, treat the probabilities as a weighted lottery. Use the table from section 23. Imagine 100 tickets: 40 say Java, 35 say Python, 10 say C++, 8 say JavaScript, 4 say Rust, 3 say Other. Draw one.

Java is the most likely winner but not a sure one. Python could come out. C++, JavaScript or Rust will win now and then. This is sampling:

Higher probability = more likely to be chosen, but not guaranteed.

This is why the same prompt can give different answers on different runs.

26. Temperature

Temperature is a setting that makes the distribution sharper or flatter before sampling. Say the logits behind the section 23 table are:

Java -> 3.7    Python -> 3.6    C++ -> 2.3    JavaScript -> 2.1    Rust -> 1.4    Other -> 1.1

Adjusted logit = original logit / temperature

Softmax is then applied to the adjusted values.

26.1 Temperature = 1

Nothing changes. Temperature 1 is the baseline. Softmax gives Java about 40% and Python about 36%, close to the section 23 table.

26.2 Low temperature (0.5)

3.7/0.5, 3.6/0.5, ... = 7.4, 7.2, 4.6, 4.2, 2.8, 2.2. The gaps between logits grow. After softmax the top two take almost everything: Java rises to about 52% and Python to about 42%, while Rust drops to about 0.5%. Lower temperature sharpens the distribution. Output is more predictable and consistent, with less variety.

26.3 High temperature (2)

3.7/2, 3.6/2, ... = 1.85, 1.8, 1.15, 1.05, 0.7, 0.55. The logits move closer together, so lower ranked tokens get a bigger share: Java falls to about 29% and Rust rises to about 9%. Higher temperature flattens the distribution and gives unlikely tokens a better chance.

27. Model vs Decoding Strategy

The model says "here is the probability distribution for the next token". The decoder says "given this distribution, how do we pick one?"

Greedy, sampling and temperature all belong to the second stage. They change how we choose from the model's predictions, not the predictions themselves.

28. The Complete LLM Generation Pipeline

Additional context: the loop ends when the model emits a special end-of-text token, or when a maximum length is reached.

29. Summary

1. Text becomes tokens. Models work on tokens, not words or sentences.
2. Tokens get numbers. Token ids say which token it is. Embeddings are the useful learned vectors.
3. Position matters. "Dog bites man" is not "Man bites dog".
4. Context changes meaning. The same token means different things next to different tokens.
5. Attention connects relevant information. Each token collects information from other tokens, based on relevance.
6. Transformers refine again and again. Many layers build richer contextual representations step by step.
7. The final representation becomes vocabulary scores. Those raw scores are logits.
8. Softmax makes probabilities. Every possible next token gets one.
9. A decoding strategy picks one token. Greedy or sampling, shaped by temperature.
10. The chosen token is appended and the loop repeats. This repeated next token prediction is autoregressive generation.

In one line: an LLM takes the tokens so far, builds context aware numbers for them through Transformer layers, predicts a probability for every possible next token, picks one, appends it, and repeats.

Next: Talking to an LLM from Your Application, where all of this turns into an HTTP request, a token bill and a Spring Boot service.