How Large Language Models Work

The goal is to understand what happens inside an LLM when you type a message and receive an answer: from converting text into numbers to selecting the next token.

You do not need to become a machine-learning researcher. You need a clear mental model so you can build, debug, and optimize AI applications.

The one idea to remember for life

An LLM repeatedly predicts the next token using the context available to it.

Imagine you type:

I drink coffee every morning before going to my ...

The model considers the whole available context and assigns probabilities to possible next tokens. Maybe office, work, or desk are candidates.

It selects a token, adds it to the text, and predicts again.

Repeat until the response ends or a generation limit is reached.

This is the central mechanism behind autoregressive text generation. Everything else in this lesson helps explain how the model makes each prediction.

The complete journey

We'll follow one sentence through the model:

The capital of India is Delhi.

Memory trick: Text → Tokens → IDs → Embeddings → Context → Probabilities → Choice → Repeat.

We'll understand each stage with simple examples.

Part 1: Tokens, how the model breaks text apart

1. Why can't the model just read English?

A computer performs calculations on numbers. Before a language model can process your text, the text must be converted into numerical representations.

The first step is tokenization.

A tokenizer splits text into smaller units called tokens. A token may be a whole word, part of a word, punctuation, or another piece of text.

For example, a simplified tokenizer might represent:

This is only an illustration. The exact tokens depend on the model's tokenizer; programming might be one token or several.

2. Why not use one token for every complete word?

Imagine the vocabulary needs separate entries for:

• Automatic
• Automatically
• Automation
• Automate
• Automated

Then consider all possible names, technical terms, misspellings, abbreviations, and newly invented words. The vocabulary would become enormous and still wouldn't cover everything.

3. Why not use one token for every character?

Take the word programming.

If every character is a separate token, the model must process many pieces:

p → r → o → g → r → a → m → m → i → n → g

That makes sequences much longer.

Modern tokenizers use a middle ground between complete words and individual characters.

Tokenization is a compromise

A useful rule of thumb is that common text often uses fewer tokens than unusual text. The frequently quoted estimate of four English characters per token is only a rough estimate, not a definition.

Why should a developer care? API cost, context-window usage, and often latency depend on the number of tokens processed or generated.

4. Token IDs: giving each token a number

After tokenization, each token is mapped to an ID from the model's vocabulary.

For example, using a made-up vocabulary:

The pipeline now looks like this:

Here's the important distinction:

• A token is a piece of text.
• A token ID is the number assigned to that token.
• An ID does not itself tell us the token's meaning.

Think of employee IDs. Employee 104 isn't more important than employee 237 just because the number is smaller.

5. Check your understanding

If cat = 10 and dog = 20, does that mean a dog is twice as large as a cat?

• Yes, because 20 is twice 10.
• No, IDs only identify tokens.

Part 2: Embeddings, turning IDs into useful numbers

6. Why aren't token IDs enough?

Suppose the model sees these token IDs:

king   → 781
queen  → 942
banana → 205
laptop → 617

These numbers identify tokens, but they don't directly express useful relationships.

We want the model to learn that king and queen are related in some contexts, while banana and laptop are usually less related.

That's where embeddings come in.

An embedding is a vector: a list of numbers that the model can process mathematically.

For an easy illustration, imagine words represented using four features:

These are made-up features for teaching, not actual dimensions used by a modern LLM.

In a real model, embeddings may contain hundreds or thousands of learned numerical dimensions. We usually can't assign simple human labels such as "royalty" to individual dimensions.

The key idea is that the model learns numerical representations that help it process patterns and relationships.

7. Remember this analogy

Imagine a library:

• Token: The title of a book.
• Token ID: The book's catalog number.
• Embedding: A numerical representation of the book's learned characteristics and relationships.

The catalog number identifies the book. The representation helps a computer work with its properties.

Memory trick: ID tells the model which token. Embedding gives it a learned numerical representation.

8. Why embeddings alone aren't enough

Consider the word bank:

• "I deposited money at the bank."
• "We sat on the bank of the river."

The token bank starts from the same token identity and initial embedding in both sentences. But its intended meaning differs.

The model must use the surrounding words to build a context-aware representation.

This is one of the key jobs of attention.

Part 3: Position and context

9. Why does word order matter?

Compare these sentences:

Dog bites man.
Man bites dog.

Both contain the same words, but the meaning changes because the order changes.

The model therefore needs information about where tokens appear in the sequence.

A simplified input representation is:

Token embedding + Position information

Position information allows the model to distinguish a token at the beginning of a sentence from the same token appearing later. Different Transformer architectures encode positional information in different ways.

10. Context changes the meaning of a word

Let's use bank again:

The model uses the surrounding context to update the representation of bank.

Memory trick: An embedding gives a token its starting representation; context helps the model work out what that token means here.

Part 4: Attention, how tokens find useful context

11. What is attention?

Imagine you're reading this sentence:

The animal didn't cross the street because it was too tired.

When you see it, you naturally connect it to animal, not street.

Now change the sentence:

The animal didn't cross the street because it was too wide.

This time, it could refer to the street.

The surrounding words change the interpretation. Attention is a mechanism that lets a model combine information from relevant tokens in the context.

A useful way to remember it:

Attention asks: "Which other tokens should I pay attention to in order to understand this token?"

12. Visualize attention

Sentence: "The animal was tired because it ran."

Focus token: it

Illustration only: these are not measured attention weights from a real model.

The model does not follow a manually written rule like "if the word is it, look for an animal." It learns patterns from training data and computes context-dependent attention scores.

13. Query, Key, and Value: the most important attention analogy

These three names sound complicated until you compare them to a search engine.

Imagine searching for a Java course.

• Query (Q): What information am I looking for?
• Key (K): What information does each result indicate it contains?
• Value (V): What useful content does each result provide?

In a Transformer, Query, Key, and Value are numerical vectors calculated from the token representations using learned transformations. They are not literal English questions.

Memory trick: Q asks, K matches, V contributes.

14. Why does every token get Q, K, and V?

Consider:

The cat sat on the mat.

The model creates Query, Key, and Value vectors for the token representations. A token's Query is compared with the Keys of other tokens to calculate how relevant they are to it.

Why three vectors? Because finding relevant information, deciding how well it matches, and providing the information are different jobs.

The same token can be relevant in different ways depending on what another token needs.

15. Self-attention, step by step

Let's follow the token it in:

The animal didn't cross the street because it was tired.

The simplified process is:

This process is called self-attention because tokens attend to other tokens in the same sequence.

The important result is that a token's representation becomes more context-aware. In a decoder-only model such as GPT, a causal attention mask prevents a position from using future tokens that haven't been generated yet.

Part 5: Transformers, processing context in layers

16. Is a Transformer just attention?

Not quite.

Attention is one important component of a Transformer. A Transformer contains repeated blocks with attention and other neural-network operations, including feed-forward layers and normalization.

17. Why are there multiple layers?

Consider this sentence:

The developer couldn't deploy the application because the server ran out of memory.

The model needs to relate server to memory, recognize that the server ran out of that resource, and connect the event to the deployment failure.

A simplified way to picture the layers is:

• Earlier processing: basic token and local relationships.
• Intermediate processing: richer combinations and grammatical relationships.
• Later processing: increasingly useful contextual features for predicting the next token.

This is a conceptual explanation, not a strict rule that every layer has one particular job.

Think of editing a photograph. You might first adjust exposure, then contrast, then color and details. Each stage refines the result. Transformer blocks repeatedly transform the representations in a similar broad sense.

18. What travels through the layers?

Each token has a numerical vector. Attention and other operations update those vectors, which are passed through the network.

The model does not need to convert each intermediate vector back into English. The internal representations remain numerical until the generation process selects output tokens.

By the final layers, the representations reflect the token, its position, and contextual relationships learned by the network.

Part 6: Logits and softmax, predicting the next token

Now the model has processed the prompt:

The capital of India is

What happens next?

19. Logits: raw scores for possible tokens

The final representation is used to calculate a score for every token in the model's vocabulary.

A simplified example:

These scores are called logits.

Logits are not probabilities. They can be positive or negative, and they don't have to add up to 1 or 100%.

A higher logit generally means the token is preferred by the model's next-token scoring step, before decoding adjustments.

20. Softmax: turning scores into probabilities

Raw scores aren't as intuitive as probabilities. Softmax converts the logits into a probability distribution.

For logits z₁, ..., zₙ, softmax calculates:

Pᵢ = e^zᵢ / (e^z₁ + e^z₂ + ... + e^zₙ)

You don't need to memorize the formula yet. Remember what it does:

• Every probability is non-negative.
• The probabilities add up to 1.
• Higher logits receive higher probabilities.
• Tokens with very low logits receive very small probabilities.

For our example, imagine the distribution is:

These percentages are illustrative, not actual output from a model.

Logits vs. probabilities

Memory trick: Logits are scores. Softmax turns scores into probabilities.

21. Does the model generate the whole answer at once?

Not in standard autoregressive generation. The model produces a distribution for the next token, selects one token, and repeats.

For example:

Prompt:
The capital of India is

Step 1: Select "Delhi"
Text: The capital of India is Delhi

Step 2: Predict the next token
Perhaps "." is selected.

Text: The capital of India is Delhi.

Step 3: Continue until generation ends.

A long answer is built through repeated next-token prediction.

Part 7: Decoding, choosing which token to use

The model calculates the next-token distribution. A decoding strategy determines how to select a token from it.

22. Greedy decoding

Greedy decoding always selects the token with the highest probability at each step.

Suppose the model predicts:

Greedy decoding selects Java.

It is predictable, but repeatedly choosing the top token can produce less varied or repetitive text. Greedy decoding is not guaranteed to produce the best overall answer.

23. Sampling

Sampling treats the probabilities as a weighted lottery.

Imagine 100 tickets:

• 40 tickets say Java.
• 35 say Python.
• 10 say C++.
• 8 say JavaScript.
• 4 say Rust.
• 3 say Other.

Draw a ticket at random. Java is the most likely result, but Python can be selected too.

This is why the same prompt can produce different responses when sampling is enabled.

Try the idea: choose a decoding strategy

• Greedy: always choose the highest-probability token.
• Sampling: allow other tokens according to their probabilities.

Selected token: Java. Greedy always chooses the 40% option in this example.

24. Temperature: controlling randomness

Temperature adjusts the logits before softmax is applied.

Conceptually:

new zᵢ = zᵢ / T

Here, T is the temperature.

• Low temperature: Makes the probability distribution sharper, favoring higher-scoring tokens.
• Temperature of 1: Leaves the logits unchanged before softmax.
• High temperature: Flattens the distribution, giving lower-scoring tokens more chance.

The temperature scale illustrates the concept; it does not run a real model or calculate its actual token probabilities.

Temperature doesn't make the model more knowledgeable. It changes the distribution used for token selection.

For many factual tasks, a lower temperature can help make output more consistent, but it cannot guarantee correctness.

25. Model vs. decoder: don't mix these up

This distinction is useful when debugging output. The model's learned behavior and the strategy used to select its tokens are related but different.

Part 8: Streaming, why the answer appears word by word

26. How does ChatGPT show text before the full answer is ready?

You've probably seen an answer appear gradually:

Java
Java is
Java is a
Java is a programming
Java is a programming language.

This works because generation happens incrementally. The application can send generated text to the interface as tokens or chunks arrive, instead of waiting for the entire answer.

Remember that model generation and UI streaming are two related but separate things. The model generates incrementally; the application transports and displays that output.

This is why streaming can improve perceived responsiveness: the user sees the beginning of an answer before generation finishes.

Part 9: The entire LLM pipeline in one diagram

This is the diagram worth revisiting until you can explain it without looking at your notes.

In a simplified decoder-only Transformer, the newly selected token is added to the context, and the model predicts again. Real implementations use optimizations such as a key-value cache to avoid recomputing everything from scratch.

Part 10: How to remember this for life

Use this everyday analogy: writing a message with a smart autocomplete system.

The 8-step memory chant

Repeat this in your own words:

1. Split the text into tokens.
2. Number the tokens with IDs.
3. Represent them as embeddings.
4. Locate them using position information.
5. Connect relevant context through attention.
6. Score possible next tokens with logits and softmax.
7. Select one token through decoding.
8. Repeat until the response ends.

If you remember this sequence, you have the high-level mechanics of an LLM.

Part 11: Interview questions and answers

"What is a token?"

A token is a piece of text processed by a language model. It can be a complete word, part of a word, punctuation, or another text unit.

"What is the difference between a token ID and an embedding?"

A token ID identifies a token. An embedding is a learned vector of numbers used by the model to represent it.

"Why is positional information important?"

Because word order changes meaning. "Dog bites man" and "Man bites dog" contain the same words but describe different events.

"What is attention?"

Attention allows the model to calculate which other tokens are relevant to a token's current representation and combine useful information from them.

"Explain Query, Key, and Value."

Query represents what a token is looking for, Key is used to calculate relevance, and Value carries the information that attention combines.

"What is the difference between a Transformer and attention?"

Attention is a mechanism inside a Transformer. A Transformer consists of repeated blocks containing attention and other neural-network operations.

"What are logits?"

Logits are raw scores assigned to possible next tokens before conversion into probabilities.

"What does softmax do?"

Softmax converts scores into a probability distribution whose values are non-negative and sum to one.

"What is autoregressive generation?"

The model predicts a token, appends it to the context, and predicts another token. This repeats until generation stops.

"What is the difference between greedy decoding and sampling?"

Greedy decoding selects the highest-probability token every time. Sampling selects a token probabilistically, allowing alternatives to be chosen.

"What does temperature do?"

Temperature changes the sharpness of the probability distribution before sampling. Lower temperature favors high-scoring tokens more strongly; higher temperature makes the distribution flatter.

"Why does streaming make an AI chat interface feel faster?"

Because the application displays generated text as chunks arrive rather than waiting for the complete response.

Part 12: Test your understanding

Topic 02 quiz

1. What does a token ID tell the model?
   • The token's exact meaning
   • Which token it represents
   • How important the token is
   • The token's probability
2. What is an embedding?
   • A learned numerical vector
   • The final English response
   • A sampling algorithm
   • A database query
3. What is the primary purpose of attention?
   • Guarantee factual answers
   • Convert logits to probabilities
   • Combine relevant contextual information
   • Choose the final answer's length
4. What is the correct order?
   • Softmax → tokens → attention → tokenizer
   • Tokenizer → IDs/embeddings → Transformer processing → logits/probabilities → decoding
   • Decoding → tokenizer → training → output
   • Embeddings → tokenizer → IDs → softmax
5. What is the difference between logits and probabilities?
   • They are always identical
   • Logits are raw scores; softmax converts them to probabilities
   • Probabilities can be negative
   • Logits are generated by the frontend
6. What does greedy decoding do?
   • Always picks the highest-probability token
   • Randomly picks any token equally
   • Always picks the lowest score
   • Retrieves a document from a vector database
7. What happens in autoregressive generation?
   • The model permanently retrains after each word
   • The entire answer must be generated in one operation
   • A selected token is appended and the model predicts again
   • The frontend predicts tokens without the model
8. What generally happens when temperature increases?
   • The model gains new factual knowledge
   • The probability distribution becomes flatter
   • All tokens get exactly equal probability
   • The model's learned weights are retrained

Final takeaway

Your course's second topic explains what happens inside an LLM between receiving your prompt and producing the next piece of text.

The single most important sentence to remember is:

An LLM converts tokens into numerical representations, uses Transformer layers to understand context, calculates probabilities for the next token, selects one, and repeats.

Once this pipeline is clear, topics such as API costs, context windows, streaming, temperature, embeddings, and AI application debugging become much easier to understand.