Vector Embeddings from First Principles

Imagine you type “Batman” into a search box. You know it's a superhero associated with Gotham, crime, and a dark city. But a computer initially sees only characters and their codes.

How can we teach a computer to work with meaning rather than just matching letters?

The answer begins with vectors and leads to embeddings. These concepts are the foundation of semantic search, recommendations, and RAG systems.

1. Why do we need embeddings?

A computer can store words perfectly without understanding how they relate to one another.

For example:

"Batman" === "Batman";   // true
"Batman" === "Superman"; // false

This works for exact matching, but it cannot tell us whether Batman and Superman are more related than Batman and a banana.

Consider customer support:

• “My food has not come yet.”
• “Where is my order?”

The wording is different, but the meaning is similar. Exact string matching may treat them as unrelated.

The solution is to represent things as numbers so we can compare them mathematically.

The numbers are useful only if their relationships represent something meaningful. Giving every word a random numeric ID does not solve the problem.

2. Start simple: represent a movie with numbers

Suppose we're building a movie recommendation system.

Initially, we assign each movie one score: its action level, from 0 to 10.

Now we can calculate how close two movies are based on action.

• Batman to Avengers: |9 - 10| = 1
• Batman to Titanic: |9 - 2| = 7

A smaller gap means closer on this feature, so Batman is closer to Avengers.

But there's a problem: Titanic and Hera Pheri both have an action score of 2. That doesn't mean they are similar movies. One is a tragic romance; the other is a comedy.

One number captures only one property. Real things have many properties.

3. What is a vector?

Add another property: comedy.

A vector is an ordered list of numbers describing something.

For Batman:

Batman = [9, 1]
          │  │
          │  └── Comedy score
          └───── Action score

Each position has a fixed meaning. Every movie must use the same order.

For example, in a cricket app:

• [9, 2] means strong batter, weak bowler.
• [2, 9] means weak batter, strong bowler.

The numbers are the same, but their order changes the meaning.

What is a dimension?

A dimension is one position in a vector.

• [9] has 1 dimension.
• [9, 2] has 2 dimensions.
• [9, 2, 3] has 3 dimensions.
• A text embedding might have 768 or 1,536 dimensions.

A collection of vectors forms a vector space. We can measure distances or compare directions between its points.

That changes our question from “Are these strings identical?” to “How similar are these items?”

4. Three ways to compare vectors

Once we have vectors, we need ways to compare them. The three metrics in your notes are Euclidean distance, dot product, and cosine similarity.

A. Euclidean distance: how far apart?

This is the ordinary straight-line distance between two points.

For vectors A = [9, 1] and B = [10, 7]:

d(A, B) = √((9 - 10)² + (1 - 7)²)
        = √(1 + 36) = √37 ≈ 6.08

The steps are:

1. Subtract corresponding numbers.
2. Square each difference.
3. Add the squares.
4. Take the square root.

Memory trick: Euclidean = ruler. Smaller distance means closer points.

B. Dot product: how aligned and how large?

Multiply corresponding numbers and add the results.

For A = [3, 4] and B = [6, 8]:

A · B = (3 × 6) + (4 × 8) = 50

The dot product grows when vectors point in the same direction, but it also grows when vectors are larger.

For example:

A = [3, 4]
B = [6, 8]      dot product = 50
C = [30, 40]    dot product = 250

B and C point in the same direction as A, but C is much larger, so its dot product is larger.

Memory trick: Dot product = alignment plus size.

C. Cosine similarity: do they point in the same direction?

Cosine similarity compares direction while ignoring vector length.

cos(A, B) = (A · B) / (|A| × |B|)

Its value ranges from -1 to 1.

Consider two customer reviews:

Review A = [2, 1]      Short review
Review B = [20, 10]    Longer review, same pattern
Review C = [1, 2]      Different pattern

A and B have exactly the same direction. Their cosine similarity is 1.0, even though B is ten times larger.

A and C have cosine similarity 0.8, so A matches B better by direction.

Memory trick: Cosine = compare the angle, not the length.

5. Which metric should we use?

There is no universally best metric. It depends on what your application considers similar.

For a simple movie recommendation example, the user likes action movies:

User preference = [8, 2]   // action, romance

Movie A = [9, 2]
Movie B = [5, 5]
Movie C = [1, 9]

Compare the user's vector with each movie using cosine similarity. Rank the movies by score and recommend the closest matches.

One important detail from the notes: if all vectors are normalized to length 1, dot product and cosine similarity give the same score, and Euclidean distance gives the same ranking.

6. Why hand-picked features don't scale

So far, we decided what each number means. For example:

Batman = [action: 9, comedy: 1, romance: 2]

This is called a feature vector. A person chooses the features and assigns the values.

This works for simple problems. For example, a house-price model might use area, number of rooms, and distance from a station.

But it becomes difficult for language.

Problem 1: Too many features

How would you describe a movie like Interstellar? Space, science, time, family, sacrifice, hope, sadness, physics, and many more.

How many features are enough?

Problem 2: Features depend on the task

The same movie needs different features depending on the application:

• Movie recommendations: genre, tone, pace.
• Box-office prediction: budget, marketing, release date.
• Parental controls: violence, language, adult scenes.

There is no single hand-picked feature list that works perfectly for every task.

Problem 3: Language depends on context

Consider the word bank:

• “I deposited money at the bank.”
• “The fisherman sat on the river bank.”

The spelling is the same, but the meaning differs because of the surrounding words.

A person cannot manually create a complete, consistent feature list for every word, phrase, and context.

That leads to the central idea of this topic: instead of manually choosing all the features, let a model learn useful representations from data.

7. What is an embedding?

An embedding is a vector whose numbers were learned through training, rather than manually assigned by a person.

For example, a model might represent text like this:

Batman  → [0.23, -0.81, 0.48, ...]
Superman → [0.21, -0.76, 0.51, ...]
Titanic → [-0.62, 0.17, 0.83, ...]

These are illustrative vectors, not actual model outputs.

The goal is for related items to have useful relationships in the learned vector space. Batman and Superman should generally be more related than Batman and Titanic for a suitable semantic task.

The individual numbers usually don't have simple human-readable labels. Their value comes from how the entire vector behaves when compared with other vectors.

8. How does a model learn embeddings?

The notes introduce a simple training idea: you can learn about a word by looking at the words that appear around it.

Consider:

The king wore the crown.
The queen wore the crown.

I ate a banana.
I ate an apple.

King and queen appear in similar contexts. Banana and apple also appear in similar contexts.

A model can learn these relationships without a person explicitly writing “king is similar to queen.”

The training loop

The initial vectors are random, so they do not begin with meaningful relationships.

During training, the model adjusts them to improve its predictions. Over many examples, words used in similar contexts can develop similar vectors.

The notes also explain negative sampling: the model uses examples of words that did not appear together to help keep unrelated words apart. Otherwise, repeatedly pulling words together could cause everything to collapse into one group.

The exact training process used in modern embedding models can be more complex; this is a simplified learning example.

9. What is an embedding table?

An embedding table stores one learned vector per word or token.

These example values show how related words might have similar rows.

When the model receives a token ID, it can look up that token's vector in the table. This is called an embedding lookup.

A tiny JavaScript example:

const table = new Map([
  ["tea", [0.21, 0.71, 0.45]],
  ["coffee", [0.19, 0.68, 0.49]],
  ["train", [0.88, -0.3, 0.1]],
]);

function embed(word) {
  const vector = table.get(word);

  if (!vector) {
    throw new Error(`Word not found: ${word}`);
  }

  return vector;
}

console.log(embed("coffee"));
// [0.19, 0.68, 0.49]

This is a toy lookup table, not a trained embedding model. The important part is that the lookup itself is simple. The learning that produced the numbers is the clever part.

10. What does “latent dimension” mean?

In a manually designed vector, each position has a label:

[action, comedy, romance]

In a learned embedding, the dimensions might look like:

[0.31, -0.84, 0.17, 0.66, ...]

You generally cannot say that dimension 1 means “royalty” or dimension 2 means “gender.” Meaning can be distributed across many dimensions, and each dimension can contribute to multiple patterns.

This is why the notes use the term latent, meaning hidden or not explicitly labelled.

Remember this distinction:

• Feature vector: a person defines what the numbers represent.
• Embedding: training learns the numbers and their useful relationships.
• Latent space: the learned space containing these representations.

Every embedding is a vector, but not every vector is an embedding.

11. How embeddings are used in GenAI applications

Embeddings are not limited to individual words. Models can create vectors for sentences, documents, images, products, songs, and more.

Common applications include:

• Semantic search: Find information by meaning, not just exact keywords.
• Recommendations: Find products or movies similar to a user's interests.
• Clustering: Group similar documents or support tickets.
• Duplicate detection: Find questions that mean the same thing despite different wording.
• RAG: Retrieve relevant documents to provide context to an LLM.

For example, a customer asks:

“Where is my money back?”

Your database contains a support article titled:

“How to check the status of a refund.”

A keyword search might miss the relationship. An embedding-based search can identify their similar meaning and retrieve the article.

One important rule from the notes: vectors from different embedding models generally cannot be directly compared, because their coordinates belong to different learned spaces. If you change embedding models, you generally need to regenerate the vectors used by your search system.

12. How embeddings fit inside an LLM

The notes connect this concept back to how large language models work.

A token ID identifies a token. The embedding table converts that ID into a learned vector. Transformer layers then process the vectors using context.

The embedding API you use for semantic search serves a related purpose: it converts an input text into a fixed-length vector. For example, the notes identify OpenAI's text-embedding-3-small as producing 1,536 dimensions by default.

13. Quick interview revision

"What is a vector?"

An ordered list of numbers describing an item.

"What is a dimension?"

One position in a vector.

"What is a vector space?"

A space of vectors where relationships can be measured.

"What is Euclidean distance?"

Straight-line distance between two vectors.

"What is dot product?"

Multiply corresponding values and add them.

"What is cosine similarity?"

A measure of directional similarity, ignoring vector length.

"What is a feature vector?"

A vector whose features were chosen by a person.

"What is an embedding?"

A vector learned through training.

"What is an embedding table?"

A table storing learned vectors for tokens or words.

"What does latent mean?"

Hidden or not explicitly labelled.

"Why use embeddings in RAG?"

To retrieve documents that are semantically relevant to a query.

Quick quiz

1. What is an embedding?

• A random ID assigned to a word
• A vector whose numbers were learned through training
• A list of keywords only

2. Which metric compares direction while ignoring vector length?

• Euclidean distance
• Dot product
• Cosine similarity

3. Why can hand-picked features be difficult for language?

• Words have context-dependent meanings and too many possible features
• Computers cannot store numbers
• Every word has exactly one meaning

4. What is an embedding table?

• A table of database passwords
• A lookup table containing learned vectors
• A list of all possible sentences

5. What is the purpose of embeddings in semantic search?

• Match only identical strings
• Retrieve items with similar meaning
• Generate a random result

The big picture: A vector gives us numbers we can compare. An embedding is a vector learned from data so those comparisons can capture useful relationships. That is what makes semantic search, recommendations, and retrieval for RAG possible.