Vector Embeddings from First Principles

Type "Batman" into a search box. You think of a hero, Gotham, a dark city, a rich man in a cape. The computer thinks of six letters. It can store them perfectly and still have no idea what they mean. This note is about closing that gap: how we turn a word, a movie or a sentence into numbers that carry some of its meaning.

Note 6 was about how an answer travels from the model to the screen. This note steps away from chat and builds one idea from the ground up. We start with a single number per movie, grow it into a list of numbers, learn three ways to compare those lists, see why hand picked numbers stop working, and end with embeddings that a machine learns by itself. Note 2 showed embeddings inside an LLM in one section. Here they get the whole note, because notes 8, 9 and 10 are built on top of them. The code is a small lab in Java, Node.js and Python under code/07-vectors-and-similarity/. It needs no API key.

Here are the words used in this note. Each one is explained again the first time it shows up, but this table is the short version:

1. A Computer Stores Words, Not Meaning

Give a person the word Batman and a cloud of ideas shows up at once: superhero, Gotham, Bruce Wayne, dark, rich, crime, night. Give a computer the same word and it gets this:

B  a  t  m  a  n
66 97 116 109 97 110

Those are the character codes. The computer can save them, copy them and print them back without a single mistake. None of that tells it that Batman is a hero and not a type of fruit.

Your phone does the same thing. It stores a contact called "Mom" and a contact called "Amma". To you, they might be the same person. To the phone they are two different strings. A shop till works the same way: it knows item 8901234 costs 40 rupees, but it has no idea that item 8901234 is "bread" and that bread goes well with butter.

The diagram puts the two views side by side:

Same input, two very different results. The person gets meaning. The computer gets an identity: a way to tell this string apart from others.

Storing a symbol is not the same as understanding what it means.

2. Same or Different Is Not Enough

Normal code is very good at one question: are these two things exactly the same?

"Batman".equals("Batman");    // true
"Batman".equals("Superman");  // false

That answer is correct and also not very useful. It says Superman is just as different from Batman as "banana" is. It cannot say "these two are related".

Real users run into this every day. A customer on a food delivery app types "my food has not come yet". Another types "where is my order". A support team would put both in the same pile. An exact match puts them in different piles, because the letters differ. Same with "I forgot my password" and "How do I get back into my account?". As strings they share almost nothing. In meaning, they ask for the same thing.

Here is what exact matching can and cannot tell you:

The only two outcomes are true and false. Everything in between, the "almost the same" and "a bit related", falls into false. So the real question of this note is: how do we turn meaning into something maths can work with?

3. Why We Need Numbers

Computers, neural networks and machine learning all work on numbers. There is not much maths you can do on the word "Pizza". Numbers you can add, multiply, measure distances and angles between, and process by the million.

So the first job is to turn real things into numbers. Those "things" can be almost anything: words, sentences, documents, movies, users, songs, products, images, even protein molecules.

The path looks like this:

The middle step is the hard part. The rest of this note is about doing it well.

One easy idea does not work: giving every item a random id. Your train ticket has a PNR number. Your library book has a call number. Your cricket team gives each player a shirt number. Those numbers identify things, but shirt 7 is not "more similar" to shirt 8 than to shirt 45. An id is a name written in digits. It carries no meaning, so the distance between two ids means nothing.

The numbers only help if their positions say something about meaning.

4. First Try: One Number per Thing

Forget language for a moment. Say we are building a movie recommender. We give each movie one score: how much action it has, from 0 to 10.

For the first time, one property of a movie is a number. Now we can measure "how close" two movies are. Is Batman closer to Avengers or to Titanic?

|9 - 10| = 1     Batman to Avengers
|9 - 2|  = 7     Batman to Titanic

The smaller gap wins. In terms of action, Batman is closer to Avengers. The computer worked that out with a subtraction.

What we built is a number line. Every movie has a spot on it:

This is a one dimensional space. Things near each other on the line are similar in the one property we measured.

The same trick works anywhere you can name one property. A food app could rate each dish by spice level: dal 2, butter chicken 4, vindaloo 9. A cricket app could rate players by batting average. Each gives you a line, and each line lets you ask "who is near whom".

Where one number breaks

Look at Titanic and Hera Pheri. Both have an action score of 2. The computer says they are identical. They are not. One is a sad love story on a sinking ship. The other is a comedy about three broke men. Both just happen to have little action.

The spice line has the same problem. A plain rice dish and a sweet kheer can both have spice level 0. Nobody would call them the same dish.

One number can hold one property. Real things have many.

5. Two Numbers Are Better Than One

So add a second property: comedy.

Batman is no longer just 9. It is the pair [9, 1]. Avengers is [10, 7], Titanic is [2, 1] and Hera Pheri is [2, 10].

Now Titanic and Hera Pheri are far apart, because their comedy numbers are far apart. Each movie is a point on a flat map, with action going across and comedy going up. The groups that form on that map look like this:

Inside "low comedy", Batman and Interstellar sit near each other on the action axis, and Titanic sits at the far end. Hera Pheri is now nowhere near Titanic, which fixes the problem from section 4.

6. What a Vector Is

That ordered list of numbers has a name. A vector is an ordered list of numbers that describes one thing. In Batman = [9, 1], position 1 is action and position 2 is comedy. Nothing stops us from adding more:

Batman = [9, 1, 2, 9, 8, 7]   // action, comedy, romance, darkness, violence, mystery

The parts of a vector, as a picture:

Each position has one fixed job. That job is the same for every movie in the list.

Order matters

Take a cricket app that stores each player as [batting, bowling], both out of 10.

[9, 2]   strong batter, weak bowler
[2, 9]   weak batter, strong bowler

Same two numbers, opposite players. Swap the order and you describe a different person. That is why every vector in one collection must use the same order. A vector is not a bag of numbers. It is a row in a table where every column has a fixed meaning.

7. What a Dimension Is

Each position in a vector is one dimension. [9] has one, [9, 2] has two, [9, 2, 3] has three. A real text embedding has 768, 1536 or more.

Adding a dimension adds one more direction things can differ in:

You can picture the first three. Nobody can picture 768. That is fine. The maths for distance and angles works the same way with 2 numbers or 2,000. You just add more terms to the sum.

A delivery address is a good everyday example of dimensions. Building number, street, area, city, pin code. Each is one dimension of "where". You need all of them together to land at the right door.

High dimensional spaces are hard for people to picture, not hard for computers to calculate with.

8. Vector Space: From "Same?" to "How Close?"

Put all our movie vectors together and you get a vector space: a set of points where every point is a vector and you can measure between any two.

This changes the kind of question we can ask:

Exact matching gives two answers. A vector space gives a score on a sliding scale. That score is what lets a search engine say "these results are close, and this one is the closest". Every idea in notes 8, 9 and 10 rests on this switch.

9. Three Ways to Compare Vectors

Once things are points, we need a way to say how close two points are. There are three common ones:

You do not need to remember formulas. You need a feel for what each one counts as "close", because they can give different answers on the same data. The next sections take them one at a time, with numbers you can check on paper.

10. Euclidean Distance

Euclidean distance is the normal straight line distance. It is what a ruler would measure on the map from section 5. Take Batman [9, 1], Avengers [10, 7] and Titanic [2, 1]. Batman to Avengers:

differences:  9 - 10 = -1     1 - 7 = -6
squares:      1               36
add:          37
square root:  6.08

Batman to Titanic is simpler. The differences are 7 and 0, so the distance is exactly 7.00. 6.08 is smaller, so Batman is closer to Avengers.

The same four steps, as a flow:

Squaring makes every difference positive, so a gap of -6 counts as much as a gap of 6. The square root undoes the squaring, so the answer is a plain gap again, like 6.08.

A second example, with food. Rate dishes as [spicy, sweet]: pani puri [8, 3], bhel puri [6, 3], gulab jamun [0, 10]. Pani puri to bhel puri is 2.00. Pani puri to gulab jamun is about 10.63. The two street snacks sit close. The dessert is far away. That matches what anyone at a chaat stall would tell you.

11. A Bad Representation Gives Bad Distances

Distance is not magic. It is only as good as the numbers you feed it.

Say we described movies as [letters in the title, release month] instead. Batman has 6 letters. Titanic has 7. If both came out in a similar month, they land right next to each other. The distance maths is correct. The answer is useless, because those two properties have nothing to do with what the movies are like.

The maths in both boxes is the same. Only the features changed, and the features decided whether "close" means anything.

A library has the same problem. Shelve books by the colour of the cover and you can measure "closeness" perfectly well. It will put a cookbook next to a physics book because both are red.

Similarity is only as good as the representation behind it.

12. Why Direction Can Matter More Than Size

Here is a case where straight line distance gives the wrong idea.

A restaurant counts two kinds of words in each review: praise words and complaint words. That makes each review a vector [praise, complaint].

Short review A  = [2, 1]      "Tasty food. Bit slow."
Long review B   = [20, 10]    a full page saying the same thing, ten times over
Grumpy review C = [1, 2]      "Okay food. Slow and cold."

A and B have the same opinion: twice as much praise as complaint. B just uses more words. C has a different opinion: more complaint than praise.

Now measure:

Euclidean distance says the short happy review is closest to the grumpy one. That is backwards. The problem is length. B is long because the writer typed more, not because the opinion is different.

Draw each review as an arrow from zero to its point and the picture is clear:

A and B point the same way. C points somewhere else. If what you care about is the pattern, you want a measure that looks at the angle and ignores how long the arrow is.

The same thing happens with shopping. A small family buys 2 kg of rice and 1 kg of dal each week. A hostel kitchen buys 20 kg of rice and 10 kg of dal. Their baskets have the same mix. A shop that wants to recommend products should treat them alike, even though one basket is ten times bigger.

13. Dot Product

The dot product is the simplest way to combine two vectors into one number. Multiply the numbers in matching positions, then add the results.

dot([3, 4], [6, 8]) = (3 x 6) + (4 x 8) = 18 + 32 = 50

The steps:

That is all the arithmetic. What the number means is more interesting.

Two things push the dot product up: pointing the same way, and being long. The last row shows the catch. [30, 40] is not "more similar" to [3, 4] than [6, 8] is. It is just bigger, and the dot product rewards that.

In a song app that can be a feature or a bug. If a song's vector is long because it is very popular, dot product will push popular songs up the list. Sometimes that is what you want. Sometimes it buries the small song that fits the listener best.

14. Cosine Similarity

Cosine similarity is the dot product with the length taken out. Divide the dot product by the two lengths, and only the direction is left.

cosine(a, b) = dot(a, b) / (length(a) x length(b))

The length of a vector is the straight line distance from zero to its point. For [3, 4] that is 5 (the 3-4-5 triangle from school). For [6, 8] it is 10.

cosine([3, 4], [6, 8]) = 50 / (5 x 10) = 1.00
cosine([3, 4], [4, 3]) = 24 / (5 x 5)  = 0.96
cosine([3, 4], [4, -3]) = 0 / (5 x 5)  = 0.00
cosine([3, 4], [-3, -4]) = -25 / (5 x 5) = -1.00

The steps, in order:

The result always lands between -1 and 1. 1 means same direction, 0 means a right angle (nothing in common), and -1 means opposite directions.

Back to the reviews from section 12. Cosine of A and B is 1.00: the same opinion, whatever the length. Cosine of A and C is lower, 0.80, because their opinions differ. Now A matches B best, which is what we wanted.

The whole function fits in a few lines of Python. zip pairs up matching positions, so the dot line is the dot product from section 13:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    length_a = math.sqrt(sum(x * x for x in a))
    length_b = math.sqrt(sum(y * y for y in b))
    return dot / (length_a * length_b)

print(cosine([2, 1], [20, 10]))  # 1.0  same opinion, longer review
print(cosine([2, 1], [1, 2]))    # 0.8  different opinion

The lab version adds two checks this one skips: both vectors must have the same length, and neither can be all zeros. A zero vector has no direction, so "same direction" has no answer.

Cosine similarity asks one thing: do these two point the same way?

15. Recommendations With Nothing but Cosine

You already have enough to build a simple recommender. No LLM needed.

Describe a user's taste as [action, romance] and do the same for each movie. The user is [8, 2]. Movie A is [9, 2], movie B is [5, 5], movie C is [1, 9]. Score every movie against the user with cosine:

Sort by score, show the top few. That is the core of many real recommenders. The hard part is getting good vectors for users and movies, which is where the rest of this note is going.

16. Which Metric to Use

All three in one table:

There is no single right answer. It depends on how the vectors were made and what you want to measure. This chart is a reasonable starting point:

The dotted note matters in practice. Many embedding models return vectors that are already scaled to length 1. For those, cosine and dot product give the same number, and Euclidean distance gives the same ranking. The choice only really matters when lengths differ.

17. When the Three Metrics Disagree

The lab for this note has a small example where all three metrics pick a different winner. The lab has six movies, each described as [action, comedy, romance]. The table shows five, because the sixth is the query movie. Junior Hero is made up for this example: a gentle kids' superhero film whose numbers are exactly half of The Dark Knight's. Same mix, half the volume.

Ask "which movie is most like The Dark Knight [9, 1, 2]?":

Each metric picks a different winner from the same table:

Each answer makes sense once you know the question behind it. Interstellar is the nearest point. Avengers has big numbers everywhere, so it wins the dot product. Junior Hero has exactly the same mix, so cosine calls it a perfect match. When someone says "the most similar item", always ask: similar by which measure?

18. Who Picks the Features?

Every vector so far had numbers that we chose. We decided the first number means action and the second means comedy. We decided Batman gets a 9.

Choosing useful properties by hand is called feature engineering. A lot of machine learning worked this way for years, and it can work very well. A bank checking loan risk uses income, existing loans and missed payments. A house price model uses area, number of rooms and distance to the station.

The pipeline looks like this:

Look at the second box. Every useful idea has to come from a person. That is fine for ten loan properties. It falls apart fast for movies, and it falls apart completely for language. Three problems show up.

The next three sections take them one by one.

19. Problem 1: How Many Features?

Try to describe Interstellar. Space, science, family, a father and his daughter, time, love, survival, physics, sadness, hope, sacrifice, being alone, huge visuals. Is that enough? Should "has a robot" be a feature? "Makes you cry"? "Long running time"?

Ask five friends and you get five different lists. There could be thousands of useful properties, and someone has to pick which ones count.

A grocery app hits the same wall. Is a mango described by sweetness, colour, price, season, region, ripeness, how messy it is to eat? Every new feature means another column someone has to fill in for every product.

20. Problem 2: Features Depend on the Task

The same movie needs different features for different jobs:

One object, three different vectors. A list built for recommendations is useless for parental controls. There is no single correct feature list.

A cricket player is the same. A fantasy league app cares about recent form and points per match. A selector cares about technique against fast bowling. A sponsor cares about followers on social media. Same player, three vectors.

21. Problem 3: Language Is Too Messy for a Feature List

Now try words. For "king" you might list: human, male, royal, strong, adult, leader, rich, old fashioned. "Queen" shares most of those. Fine so far.

Then comes "apple". Now you need fruit, food, sweet, red, round. You also need company, phone, technology, because "Apple" is a brand.

Then "Java". Programming language, island in Indonesia, a word for coffee.

Every new word adds features, and many words need several unrelated sets. The feature list grows without end and nobody can keep it consistent.

22. Context Changes Meaning

It gets worse. The same word changes meaning with the words around it.

"I deposited money at the bank."
"The fisherman sat on the river bank."

Same spelling. Different meaning. The words nearby decide which one:

You see this all the time in cricket talk. "He hit a cover drive" is a cricket shot. "Take cover" means hide. "Bat" is a wooden blade in one sentence and a flying animal in another.

Language also has tone, sarcasm, grammar, and phrases where the meaning is not the sum of the words. "Break a leg" is a kind wish. A fixed list of hand made features cannot hope to cover all of that.

Hand made features work for a small, fixed problem. They do not scale to language.

23. The Goal: Let the Machine Learn the Space

Vectors are clearly useful. What we want is a vector for everything: Batman, Superman, Titanic, king, queen, dog. And we want the space to behave sensibly, without anyone writing the rules by hand:

That is the wish list. The trick that gets us there is to stop asking "which features should a person design?" and start asking "what task can a machine practise on, so that good vectors appear by themselves?"

24. What an Embedding Is

An embedding is a vector whose numbers were learned by a machine, so that similar things end up close together. If Batman becomes vector A, Superman vector B and Titanic vector C, we want A and B to score as more similar than A and C.

Here is the part that surprises people. We do not care what each number means on its own. We care that the space as a whole behaves well: that related things are near and unrelated things are far.

The model does the job a person did in section 18. Nobody fills in the features. The model finds its own.

25. The Numbers Start Out Random

So where do numbers like [0.23, -0.81, 0.48] come from? Not from a person typing scores. They are learned during training, and training starts from random numbers. Give king, queen, banana and apple two random numbers each, and king might well land closer to banana than to queen. That is expected. Training has not happened yet.

The lab for this note shows the same thing with real numbers. With its default seed, "tea" starts out closer to "train" (cosine about 0.30) than to "coffee" (cosine about -0.15). Random start, random neighbours.

The useful layout is not there at the start. It appears as training goes on.

Embeddings do not start meaningful. The meaning shows up during training.

26. You Know a Word by Its Neighbours

Training needs a task. You cannot just tell a model "understand language". You give it something concrete to practise on, and useful vectors appear as a side effect.

The task comes from a simple observation. Look at these sentences:

The king ruled the kingdom.       The queen ruled the kingdom.
The king lived in the palace.     The queen lived in the palace.
The king wore the crown.          The queen wore the crown.

I ate a banana.                   I ate an apple.
The banana is a fruit.            The apple is a fruit.

King and queen keep showing up next to the same words: ruled, palace, crown. Banana and apple show up next to ate and fruit. An old idea from linguistics says it in one line: you shall know a word by the company it keeps.

The lab uses its own tiny example. Tea and coffee share neighbours like drink, hot, cup, milk and morning. Train and bus share left, station, ticket and late.

Tea and coffee never need to be told they are both drinks. They hang out with the same words, so a model that learns from neighbours will treat them alike. Same for train and bus. And no line connects the drink group to the travel group.

Think of a school. If two students always sit at the same lunch table, join the same clubs and catch the same bus, you can guess they have a lot in common without ever talking to them. Words are the same.

27. A Toy Training Task

Here is a simple task a model can practise on: given one word, guess which words appear near it. In "The king wears a golden crown", the input is king and the right guesses are wears, golden and crown. The lab drops small words like "the" and "a" first.

Now take "The queen wears a golden crown". Queen has almost the same neighbours. The model only sees each word's vector. So if king and queen must make almost the same guesses, the easiest fix is almost the same vector.

The training loop that does this:

Round and round, millions of times. The model guesses "king goes with banana", gets a big error, and the vectors move a little. Later it guesses "king goes with crown", gets a small error, and the vectors barely move.

Nobody typed "king is similar to queen" anywhere. The relationship comes out of the patterns in the text.

28. The Embedding Table

Where do those vectors live? In an embedding table: one row per word, one column per dimension.

Look at the pairs. King and queen have nearly the same row. So do dog and cat, and banana and apple. These are the model's parameters: before training, random numbers; after training, useful ones.

When the model meets a word, it looks up its row. That step is called an embedding lookup:

In code, a lookup is just reading from a map. This is the whole idea in a few lines of JavaScript:

// A tiny embedding table: one learned row of numbers per word.
const table = new Map([
  ["tea", [0.21, 0.71, 0.45]],
  ["coffee", [0.19, 0.68, 0.49]],
  ["train", [0.88, -0.3, 0.1]],
]);

// The embedding lookup: word in, vector out.
function embed(word) {
  const row = table.get(word);
  if (!row) throw new Error(`not in the vocabulary: ${word}`);
  return row;
}

embed("coffee"); // [0.19, 0.68, 0.49]

It is like a phone's contact list. You give a name, it gives back the number. The clever part is not the lookup. It is how the numbers in the table got there.

29. How Training Nudges the Vectors

One nudge starts with two words that shared a sentence, say tea and coffee. The model guesses how likely they are to appear together. The right answer is yes, so both vectors move a small step to make the next guess better. Then it picks a random word that was not in the sentence, like "ticket", and pushes the pair apart: tea and ticket both move a small step away from each other. As pseudo code:

guess  = sigmoid(dot(tea, coffee))   # 0 to 1: how sure "these go together"
error  = target - guess              # target is 1 here, 0 for the random word
step   = learning_rate * error       # small, so one sentence cannot wreck the table

tea    = tea    + step * coffee      # tea leans towards coffee
coffee = coffee + step * old_tea     # coffee leans towards tea

Sigmoid is a small function that squashes any number into the range 0 to

1. Here it turns a dot product into "how sure are we that these go together". The learning rate is how big each step is. Small steps, many times, is safer than big jumps.

A worked nudge with tiny numbers:

tea    = [0.5, 0.0]       coffee = [0.0, 0.5]
dot = 0, sigmoid(0) = 0.5, target = 1, error = 0.5
learning rate 0.2  ->  step = 0.1

tea    = [0.5, 0.0] + 0.1 x [0.0, 0.5] = [0.5, 0.05]
coffee = [0.0, 0.5] + 0.1 x [0.5, 0.0] = [0.05, 0.5]

Before the nudge, the cosine was 0 (a right angle). After one nudge, it is about 0.20. One step closed the angle a little. Hundreds of steps close it a lot.

A pair made with a random word is called a negative sample. Without it, every nudge would pull words together and the whole table would slowly squash into one point. The push away keeps unrelated groups apart.

Here is the effect on the whole space:

On the left, neighbours are random. On the right, words that share sentences sit together, and the two groups sit apart.

In the lab, after 300 passes over its 12 sentences, the numbers come out roughly like this:

Tea and coffee went from unrelated to almost the same direction. Tea and train went from "a bit similar" to opposite. The code never said what tea is.

30. Why Close Vectors Help the Model

Why would a model "want" king and queen to have similar vectors? Because it saves work.

Say the model has to finish these:

The king lives in the ...
The queen lives in the ...

Good guesses for both are palace, castle, kingdom. If king and queen had totally unrelated vectors, the model would have to learn that fact twice, once for each word. If their vectors are nearly the same, whatever it learned for king already mostly works for queen.

The model is not thinking "king and queen are related". It just found that similar inputs need similar outputs, and similar vectors are the cheapest way to get that.

A new cashier learns the same way. Once they learn that a 500 ml milk packet goes in the cold bag, they do the same for a 500 ml curd packet without being told. Similar items, similar handling.

31. Latent Dimensions: The Numbers Have No Labels

With hand made features, every position had a name, like [royalty, gender, authority]. A learned embedding looks more like king = [0.31, -0.84, 0.17, 0.66, ...], with queen at [0.29, -0.78, 0.21, 0.61, ...].

It is tempting to say "dimension 1 is royalty". Usually that is wrong. Royalty is spread over many dimensions at once, say 4, 18, 91 and 207. Each of those dimensions also helps describe many other ideas. The meaning lives in the whole vector, not in one clean slot.

That is what latent means: hidden. Not observed, not named by anyone. You will hear "latent space", "latent factors" and "latent representation". All of them mean the learned, unlabelled space the vectors live in.

The top row is readable but limited. The bottom row has no labels, but it can hold far more than a person could ever list.

Map coordinates are a good way to think about it. Mumbai is roughly [19.08, 72.88] (latitude, longitude). Neither number means "Mumbai". 72.88 does not mean "big port city". But together they put Mumbai at one spot, and that spot is close to Pune at [18.52, 73.86] and far from Delhi at [28.61, 77.21]. The location is useful because of where it is compared with other locations. Embeddings work the same way, just with hundreds of coordinates instead of two.

32. Vector vs Embedding

These two words get mixed up all the time. Every embedding is a vector. Not every vector is an embedding. [height, weight, age] = [172, 68, 30] is a vector. A person chose what each number means, so it is usually called a feature vector. An embedding is a vector whose numbers were learned, so that it places something in a useful space.

The vector is the container. The embedding is one kind of thing you can put in it. A plastic box is a box whether it holds lunch or screws. "Lunch box" tells you what is inside and why.

33. Almost Anything Can Be Embedded

Embeddings are not only for words. If you can find a task to train on, you can learn a useful space for almost anything:

A music app can embed songs from what people play one after another. A shop can embed products from what people buy in the same basket. A photo app can embed images so that two photos of the same beach end up close.

Each of these is a separate model with its own space. Vectors from two different models cannot be compared. That is like comparing a GPS coordinate with a pin code.

The question is always the same: can we learn a space where useful relationships between these things turn into distances and angles?

34. The Embedding Model Is a Function

From the outside, an embedding model is a function. Something goes in, a fixed size list of numbers comes out.

The output length is fixed by the model. A model with 768 dimensions always gives 768 numbers, whether you send one word or a whole page. OpenAI's text-embedding-3-small, which note 8 uses, gives 1536.

In maths notation this is f(text) -> R^1536: text in, a list of 1536 real numbers (R) out.

Two practical rules come out of this. Same model in, same space out, so you can compare. Change the model and you have to embed everything again, because the new numbers live in a different space.

35. What You Can Do With Embeddings

Once things are vectors, many jobs become "find the nearest vectors":

Some everyday cases:

• A support desk groups tickets by meaning, so "refund not received" and "where is my money back" land in one queue.
• A food app search finds "paneer tikka" when you type "cottage cheese starter".
• A forum spots that a new question is a duplicate of an old one with different wording.
• A chatbot fetches the three policy paragraphs closest to the user's question and puts them in the context, the "only relevant" strategy from note 4.

Note 8 builds the first of these: a movie search that works on meaning, not keywords.

36. Back to Note 2: Embeddings Inside an LLM

Note 2 showed an embedding step inside every LLM. Now you can see what it was doing:

The token id is like the shirt number from section 3: it says which token, nothing more. The embedding table turns that id into a learned vector, the same lookup as section 28. Then the Transformer layers adjust those vectors using the words around them. That is how "bank" ends up with one meaning near "money" and another near "river".

The embedding models you call from an API in note 8 are built from the same parts. They are trained so that a whole sentence or paragraph comes out as one vector.

37. The Whole Journey

Here is everything in this note on one page:

The turning point is the move from box C to box F. We stopped asking "which features should a person write down?" and started asking "what task should the machine practise?". Training plus data then does the rest.

An embedding is not the meaning itself. It is a position in a learned space where related meanings end up close together.

38. Building It: the Vectors Lab

The code for this note is a lab, not a service. There is no web server and no model call, so it needs no API key. It runs the ideas from this note with numbers small enough to check by hand.

Each language has the same three modules and one demo:

How the demo runs:

Decisions worth copying:

39. The Code, Side by Side

The full projects are on the code page:

Browse the code for this note, or open src/FDE/code/07-vectors-and-similarity/ on GitHub. The README there has the run and test commands for each version.

39.1 One nudge

The heart of the toy training loop. target is 1 for two words from the same sentence and 0 for the random negative sample. The code guesses with sigmoid(dot), turns the error into a step, then moves both vectors.

private void nudge(String a, String b, double target, double learningRate) {
    double[] va = table.get(a);
    double[] vb = table.get(b);
    double guess = sigmoid(VectorMath.dot(va, vb));
    double step = learningRate * (target - guess);
    for (int i = 0; i < va.length; i++) {
        double oldA = va[i]; // vb must move using va from before this step
        va[i] += step * vb[i];
        vb[i] += step * oldA;
    }
}

function nudge(a, b, target, learningRate) {
  const va = table.get(a);
  const vb = table.get(b);
  const step = learningRate * (target - sigmoid(dot(va, vb)));
  for (let i = 0; i < va.length; i += 1) {
    const oldA = va[i]; // vb must move using va from before this step
    va[i] += step * vb[i];
    vb[i] += step * oldA;
  }
}

def _nudge(self, a: str, b: str, target: float, learning_rate: float) -> None:
    va = self._table[a]
    vb = self._table[b]
    step = learning_rate * (target - sigmoid(dot(va, vb)))
    for i in range(len(va)):
        old_a = va[i]  # vb must move using va from before this step
        va[i] += step * vb[i]
        vb[i] += step * old_a

The oldA line is easy to miss. Without it, vb would move using the already updated va, and the two words would not be treated the same way.

39.2 Where to look in the code

The tests check the things this note claims: the 3-4-5 numbers, cosine ignoring length while distance does not, the three metrics picking three different movies, and training pulling tea towards coffee and away from train.

39.3 What this demo leaves out on purpose

Key Takeaways

• A computer stores the letters of a word, not its meaning. An id says which thing it is, not what it is like.
• To do maths on meaning, turn things into vectors: ordered lists of numbers where each position has one fixed job.
• One number captures one property. More dimensions capture more, and the maths works the same in 2 dimensions or 1,536.
• A vector space lets you ask "how close?" instead of "same or different?".
• Euclidean distance measures the gap between points. Dot product rewards direction and size. Cosine similarity looks at direction only.
• The metrics can disagree. Always ask "similar by which measure?", and remember that for vectors scaled to length 1 they rank the same way.
• Similarity is only as good as the vectors. Bad features give correct maths and useless answers.
• Hand made features break down: nobody knows how many to use, the right ones change with the task, and language has too many meanings to list.
• An embedding is a vector whose numbers were learned so that related things end up close. Every embedding is a vector; not every vector is an embedding.
• Training starts from random numbers. Words that keep the same company get nudged towards each other, and random other words get pushed away.
• The dimensions are latent. No single number means "royalty". Meaning lives in the position of the whole vector, like a city on a map.
• An embedding model is a function with a fixed output size. Only compare vectors that came from the same model.
• Search, recommendations, clustering, duplicate finding and retrieval for LLMs all come down to finding the nearest vectors.