Deep Learning for Natural Language: Word Embeddings

A word ID tells a model which token it has encountered, but says nothing about what that token means. Word embeddings replace isolated token indicators with learned dense vectors. This lecture explains their geometry, how Word2Vec and GloVe learn from context, and how to use trainable or pretrained embeddings in a Keras classifier.

Learning goals

You should finish able to calculate cosine similarity, explain an embedding lookup, interpret GloVe’s objective, distinguish static and contextual representations, align pretrained vectors with your vocabulary, and pool token vectors without averaging padding.

1. Why move beyond one-hot vectors?

With vocabulary size V, a one-hot token vector has V coordinates and exactly one nonzero entry. For distinct tokens i and j, eᵢᵀeⱼ=0, each vector has norm one, and ‖eᵢ−eⱼ‖=√2. Thus great and awesome are as far apart as great and volcano in that representation.

An embedding table E∈ℝ^(V×d) assigns token i the row E[i], where d is much smaller than V. Dimensions such as 50–300 are common in classic pretrained word vectors; they are examples, not a universal range for modern language models.

For V=50,000 and d=100, the table has 5,000,000 parameters, requiring about 20 MB for float32 values before optimizer state. Each token’s dense representation has 100 values rather than 50,000. Integer IDs already provide compact storage, however: embeddings do not eliminate the need to store vocabulary or model parameters, and efficient implementations never need to materialize every token’s one-hot vector.

An embedding lookup is equivalent to eᵢᵀE, but directly selects a row. The representation learns from an objective: a classifier-trained table reflects its classification task, while a language-pretrained table reflects its source corpus and pretraining task. Dense vectors alone do not guarantee useful semantics.

2. Geometry: similarity, direction, and analogies

For nonzero vectors u and v, cosine similarity is:

cos(u,v)=(uᵀv)/(‖u‖‖v‖).

It measures alignment rather than magnitude. Euclidean distance also depends on vector lengths. For unit-normalized vectors, ‖u−v‖²=2−2cos(u,v), so the two rankings are equivalent under that normalization. Cosine similarity is undefined for a zero vector; handle missing or zero representations explicitly.

Worked similarity example

Let movie=[1,1], film=[2,1], and volcano=[−1,1]. Then:

• movie·film=3, so cosine similarity is 3/√10≈0.9487.
• movie·volcano=0, so their cosine similarity is 0.
• The movie–film Euclidean distance is 1.

These invented values illustrate a desired geometry; training does not promise exactly this arrangement. Related words often occur in related contexts, but antonyms such as hot and cold can also have similar contexts. Distributional similarity is broader than synonymy.

Analogy arithmetic

A common test asks whether:

v(brother)−v(man)+v(woman)≈v(sister).

The result is a query vector, not a token. A nearest-neighbor search compares it with vocabulary vectors, usually excluding the input words. Whether sister is nearest depends on the checkpoint, vocabulary, normalization, and scoring rule. Analogies are approximate empirical patterns, not logical identities or a reliable general reasoning mechanism.

For a deliberately constructed example, let man=[1,0], woman=[1,1], brother=[2,0], and sister=[2,1]. The arithmetic gives [2,0]−[1,0]+[1,1]=[2,1] exactly. We built that relationship into the example; a learned model must be tested rather than assumed to reproduce it.

Useful information may appear in directions, but embedding coordinates are not named concepts. A shared orthogonal rotation preserves dot products and distances while changing individual coordinates. Do not assume that coordinate 17 universally means age or that a single direction cleanly captures a complex social concept.

3. Word2Vec: learn from local prediction tasks

Word2Vec is a family of efficient approaches introduced by Mikolov and colleagues. Two key formulations are:

• Continuous bag-of-words (CBOW): predict a center word from surrounding context words.
• Skip-gram: predict surrounding words from a center word.

For the invented sentence “small birds cross blue skies”, a context window around cross might contain birds and blue, or more neighbors with a wider window. Training collects many such examples. Input and context embedding tables learn parameters useful for the prediction objective.

With a full softmax, a skip-gram conditional probability can be written:

P(context=j | center=i)=exp(uⱼᵀvᵢ)/Σₖ exp(uₖᵀvᵢ).

Here vᵢ is a center-word vector and uⱼ is a context-word vector. Computing the denominator over a large vocabulary is costly; negative sampling is one common alternative that trains observed pairs against sampled alternatives. It has a different objective from directly computing that full softmax.

The transferable product is the learned representation, not necessarily a useful document classifier by itself. Context window choices, corpus size, sampling, and which table is exported all affect the result.

4. GloVe: learn from aggregated co-occurrences

GloVe stands for Global Vectors. It uses word-context co-occurrence statistics collected across a corpus. Let Xᵢⱼ measure how often context word j occurs near target word i under the chosen window and weighting scheme.

A tiny co-occurrence calculation

Use two invented sentences, a window of one neighboring word, no cross-sentence windows, and unweighted directed counts:

cats eat fish
dogs eat fish

Selected rows of X are:

Target / context	cats	dogs	eat	fish
cats	0	0	1	0
dogs	0	0	1	0
eat	1	1	0	2
fish	0	0	2	0

Cats and dogs have identical context rows in this tiny corpus. That suggests similar distributional roles, but does not establish that they are synonyms or biologically identical. Real corpora produce noisy, frequency-dependent patterns rather than universally identical rows for related words.

The objective

A standard GloVe objective is:

J=Σ_(i,j:Xᵢⱼ>0) f(Xᵢⱼ)[wᵢᵀw̃ⱼ+bᵢ+b̃ⱼ−log(Xᵢⱼ)]².

The target vectors wᵢ, context vectors w̃ⱼ, and biases are learned. The weighting function reduces the influence of unreliable rare counts and limits the dominance of frequent pairs. A common form is f(x)=min((x/x_max)^α,1) for positive x, with α=3/4 used in the original work. The weighting exponent α is unrelated to a learning rate.

This is a weighted log-bilinear least-squares objective. Fixing one vector table makes parts of the problem regression-like, but learning both tables jointly is not ordinary linear regression with a closed-form OLS solution. Gradient-based optimization adjusts the parameters. Zero entries are excluded from this expression, avoiding log(0).

For one positive pair with Xᵢⱼ=4, predicted score s=wᵢᵀw̃ⱼ+bᵢ+b̃ⱼ=1, and an illustrative weight f(4)=0.5, the contribution is 0.5(1−log 4)²≈0.07461. Its derivative with respect to s is 2×0.5×(1−log 4)≈−0.38629. A small gradient-descent step increases s toward log 4≈1.38629.

The final exported word representation can combine the target and context tables. Use the downloaded file’s documented convention rather than assuming every GloVe file is interchangeable.

5. Limitations: static meaning, coverage, and bias

Classic Word2Vec and GloVe give a word type one vector. The same bank lookup is used for “the bank approved a loan” and “we walked along the river bank.” It may combine evidence from several senses, but it is not literally guaranteed to be their arithmetic average.

Static word vocabularies also struggle with unseen words, spelling variants, and domain-specific vocabulary. Lowercasing or punctuation removal can reduce matching quality if it differs from the pretrained corpus’s tokenization. Measure vocabulary coverage instead of assuming downloaded vectors cover your application.

Learned associations can encode social stereotypes and other corpus biases. A classifier can reproduce or amplify such associations through its own training and deployment choices. Geometry is evidence about learned distributional patterns, not a statement of normative truth. Evaluate relevant subgroups and task errors; removing a single direction does not establish that all harmful associations are gone.

Contextual representations

A contextual model computes a token representation from its surroundings, so the two uses of bank can receive different hidden states. Transformers are a major approach, but recurrent language models such as ELMo also produce contextualized representations. Contextualization does not guarantee correct word-sense interpretation or remove dataset bias.

Many contextual models start from static token lookup embeddings and then transform them through context-dependent layers. The distinction is between the initial lookup and the final contextual representation, not between “all lookup tables” and “all Transformers.”

6. STIE in Keras: indexing, then embedding

For this workflow:

1. TextVectorization(output_mode="int") performs standardization, tokenization, and indexing.
2. Embedding(input_dim=V, output_dim=d) looks up a vector for each ID.
3. A sequence encoder or pooling layer combines token representations for prediction.

With maximum length L, a batch moves through these shapes:

Stage	Shape
Raw strings	(B,)
Integer token sequences	(B,L)
Embedding output	(B,L,d)
Masked global average	(B,d)
Three-class softmax	(B,3)

output_sequence_length=L pads short sequences and truncates long ones. This trades memory and computation against lost context; inspect the training length distribution before choosing L. Keras integer output reserves 0 for padding and 1 for unknown tokens. Set input_dim to the actual vocabulary length, including both reserved entries, so every emitted ID has a valid row.

7. Padding masks and average pooling

For token IDs t₁,…,t_L, define mₖ=1 when tₖ≠0 and mₖ=0 otherwise. A masked average is:

h=Σₖ mₖE[tₖ] / Σₖmₖ,

assuming at least one nonpadding token. With token vectors [2,0] and [0,2], followed by two padding positions, the correct result is [1,1]. Averaging two additional zero padding vectors would instead give [0.5,0.5].

Use Embedding(mask_zero=True) followed directly by a mask-aware GlobalAveragePooling1D. Setting mask_zero creates a mask; it does not by itself force the stored padding row to zero or make every possible later layer respect that mask.

An empty or punctuation-only text can become all padding. Its masked-average denominator is zero, so choose an explicit policy: reject it, filter it with reported counts, or use a defined empty-text representation. Do not assume pooling will invent a sensible output. The example filters such documents during dataset preparation and checks new inputs before inference.

Average pooling still loses order. “dogs chase cats” and “cats chase dogs” have the same average of static embeddings. Repetition affects the average’s weighting, but merely repeating the entire sequence leaves its average unchanged. Embeddings plus mean pooling form a useful baseline, not a sequence-understanding model.

8. A trainable embedding classifier

Use a separate environment with compatible Keras 3, TensorFlow, and NumPy. The following blocks form one workflow. Prepare the same genre_text/train, genre_text/validation, and genre_text/test folders described in the previous lesson, each containing hip-hop, pop, and rock subfolders with one authorized UTF-8 document per .txt file.

Keep duplicate songs and related versions together, and use an artist-grouped split when evaluating unseen artists. No model accuracy is claimed here; this code requires your prepared corpus.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"

from pathlib import Path
import numpy as np
import tensorflow as tf
import keras

keras.utils.set_random_seed(42)
class_names = ["hip-hop", "pop", "rock"]
root = Path("genre_text")

def load_split(name, shuffle):
    return keras.utils.text_dataset_from_directory(
        root / name, class_names=class_names,
        label_mode="int", batch_size=64,
        shuffle=shuffle, seed=42,
    )

raw_train = load_split("train", True)
raw_val = load_split("validation", False)
raw_test = load_split("test", False)
max_length = 300
vectorizer = keras.layers.TextVectorization(
    max_tokens=20000, output_mode="int",
    output_sequence_length=max_length,
)
vectorizer.adapt(raw_train.map(lambda text, label: text))
vocabulary = vectorizer.get_vocabulary()
vocab_size = len(vocabulary)
assert vocab_size > 2

def keep_nonempty(dataset, name):
    rows = dataset.unbatch()
    before = sum(1 for _ in rows)
    retained = rows.filter(
        lambda text, label: tf.reduce_any(
            vectorizer(tf.expand_dims(text, 0)) != 0
        )
    )
    after = sum(1 for _ in retained)
    print(name, "excluded empty documents:", before - after)
    if after == 0:
        raise ValueError("No nonempty documents in " + name)
    return retained.batch(64).prefetch(tf.data.AUTOTUNE)

train_ds = keep_nonempty(raw_train, "train")
val_ds = keep_nonempty(raw_val, "validation")
test_ds = keep_nonempty(raw_test, "test")

The vocabulary is adapted only on training text. The empty-text rule is fixed in advance and reused on held-out splits; report exclusions because evaluation now applies to the retained nonempty documents. It does not use validation labels to learn vocabulary. For large corpora, perform the audit efficiently rather than repeatedly reading files as this transparent teaching implementation does.

def build_classifier(embedding_layer):
    inputs = keras.Input(shape=(), dtype="string", name="document")
    ids = vectorizer(inputs)
    x = embedding_layer(ids)
    x = keras.layers.GlobalAveragePooling1D()(x)
    x = keras.layers.Dense(32, activation="relu")(x)
    outputs = keras.layers.Dense(len(class_names), activation="softmax")(x)
    model = keras.Model(inputs, outputs)
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=1e-3),
        loss=keras.losses.SparseCategoricalCrossentropy(),
        metrics=[keras.metrics.SparseCategoricalAccuracy(name="accuracy")],
    )
    return model

scratch_embedding = keras.layers.Embedding(
    input_dim=vocab_size, output_dim=100,
    mask_zero=True, trainable=True, name="learned_embedding",
)
scratch_model = build_classifier(scratch_embedding)
scratch_history = scratch_model.fit(
    train_ds, validation_data=val_ds, epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", patience=3, restore_best_weights=True,
    )], verbose=2,
)

The embedding table starts randomly and receives task-loss gradients. It is not learning the GloVe objective. If V=20,000 and d=100, the table contains 2,000,000 parameters, while the hidden layer has 100×32+32=3,232 and the output has 32×3+3=99. Total stored parameters: 2,003,331. Actual V may be smaller because max_tokens is a cap.

Only looked-up rows receive ordinary lookup-loss gradient contributions on a given batch, although regularization and optimizer behavior can affect update details. Rare words receive little direct evidence, making task-trained embeddings sensitive to dataset size.

9. Initialize a table from pretrained GloVe

Obtain a documented pretrained release from the official Stanford GloVe page. For this example, place the extracted 100-dimensional glove.6B.100d.txt file locally. Record its source, release, and usage terms. Do not confuse corpus size, vocabulary size, and vector dimension.

Row alignment is essential: row i must correspond to vectorizer.get_vocabulary()[i]. The GloVe file’s line order has no relationship to your vectorizer’s token IDs. The code streams the file and retains only matched rows, avoiding a dictionary containing the entire pretrained vocabulary.

embedding_dim = 100
glove_path = Path("glove.6B.100d.txt")
word_index = {word: i for i, word in enumerate(vocabulary)}
rng = np.random.default_rng(42)
embedding_matrix = rng.normal(
    0, 0.05, size=(vocab_size, embedding_dim)
).astype("float32")
embedding_matrix[0] = 0.0
matched = set()

with glove_path.open(encoding="utf-8") as source:
    for line in source:
        fields = line.rstrip().split()
        if not fields:
            continue
        index = word_index.get(fields[0])
        if index is None or index < 2:
            continue
        if len(fields) != embedding_dim + 1:
            raise ValueError("Unexpected vector dimension for " + fields[0])
        vector = np.asarray(fields[1:], dtype="float32")
        if not np.isfinite(vector).all():
            raise ValueError("Non-finite vector for " + fields[0])
        embedding_matrix[index] = vector
        matched.add(index)

if not matched:
    raise ValueError("No vocabulary matches; check tokenization and vector file")
embedding_matrix[1] = embedding_matrix[sorted(matched)].mean(axis=0)
print("Known-token type coverage:", len(matched) / (vocab_size - 2))

pretrained_embedding = keras.layers.Embedding(
    input_dim=vocab_size, output_dim=embedding_dim,
    embeddings_initializer=keras.initializers.Constant(embedding_matrix),
    mask_zero=True, trainable=False, name="glove_embedding",
)
glove_model = build_classifier(pretrained_embedding)
glove_history = glove_model.fit(
    train_ds, validation_data=val_ds, epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", patience=3, restore_best_weights=True,
    )], verbose=2,
)

The constant initializer loads the table; supplying embedding weights through the layer’s supported weights interface is another route. Matching words use pretrained vectors, the padding row is zero, unknown uses the mean of matched vectors, and unmatched known words retain deterministic random initializations. When the table is frozen, those unmatched rows stay random. They have no pretrained meaning; measure coverage and reconsider this strategy if it is poor.

Type coverage weights every vocabulary entry equally. Token-weighted coverage measures how much actual text is covered and can tell a different story. Neither requires adapting the vocabulary on test data. Record the coverage policy, preprocessing, and tokenizer with your model.

Frozen vectors reduce the number of trainable parameters but still occupy memory. To fine-tune, change the embedding layer’s trainability and recompile with an appropriate, often smaller, learning rate. Compare frozen, fine-tuned, and scratch models on validation data; pretrained initialization is not guaranteed to win.

10. Evaluate and preserve the input contract

After choosing a model and settings using validation data, evaluate the chosen model on the same retained test examples as your baseline. Use majority-class and bag-of-words comparisons, per-class results, and uncertainty rather than assuming dense semantic features must improve accuracy.

# Set this choice from validation evidence before final test evaluation.
chosen_model = glove_model
print(chosen_model.evaluate(test_ds, return_dict=True, verbose=0))

def predict_checked(texts):
    texts = tf.convert_to_tensor(texts, dtype=tf.string)
    if texts.shape.rank != 1:
        raise ValueError("Pass a one-dimensional batch of document strings")
    nonempty = tf.reduce_any(vectorizer(texts) != 0, axis=1)
    if not bool(tf.reduce_all(nonempty).numpy()):
        raise ValueError("At least one document is empty after preprocessing")
    return chosen_model(texts, training=False).numpy()

print(predict_checked(["A quiet lantern shines beside the road."]))
chosen_model.save("genre_embeddings.keras")

The model file includes the vectorizer and embedding table; preserve class-name ordering and the external empty-input check as part of deployment. Calling the saved model directly does not automatically apply predict_checked. A production service needs that rule at its input boundary or a defined safe-empty representation inside the model.

Sequence truncation and mean pooling can both discard information. Better static vectors do not fix missing order, long-distance dependencies, or word-sense ambiguity. Those limitations motivate sequence encoders and contextual models in later lessons.

11. Exercises with answers

1. One-hot geometry: What is the Euclidean distance between distinct unit one-hot vectors? √2; their dot product and cosine similarity are zero.
2. Lookup shape: For B=16, L=200, and d=50, what shape follows the embedding layer? (16,200,50); masked global average pooling gives (16,50).
3. Table size: How many parameters are in a 10,000×64 embedding table? 640,000, even when frozen; freezing changes trainable status, not storage size.
4. Cosine: For u=[1,0] and v=[1,1], what is cosine similarity? 1/√2≈0.7071.
5. GloVe target: For Xᵢⱼ=e², what log-count target is used? 2. A zero count is excluded from the displayed objective.
6. Pooling: For vectors [3,0] and [0,3] followed by four zero padding rows, what is the masked mean? [1.5,1.5]. The unmasked six-row mean would be [0.5,0.5].
7. Alignment: Can you load pretrained vectors into your table in file-line order? No. Match each pretrained token to your vocabulary’s index.
8. Unknown versus padding: Which index is excluded by mask_zero=True? 0. Index 1 is unknown and remains a valid, unmasked token.
9. Static ambiguity: Why does GloVe give bank the same initial vector in financial and river contexts? Its lookup is indexed by word type, not by the surrounding sentence.
10. Order: Does averaging static embeddings distinguish “cats chase dogs” from “dogs chase cats”? No. The multiset and average are identical.
11. Contextual models: Are Transformers the only way to compute context-dependent word representations? No. Recurrent models such as ELMo also contextualize tokens.

Further reading and sources

• Mikolov et al., Efficient Estimation of Word Representations in Vector Space: Word2Vec architectures and vector relationships.
• Pennington, Socher, and Manning, GloVe: Global Vectors for Word Representation: the co-occurrence objective and its motivation.
• Stanford GloVe project: documented code and pretrained vector releases.
• Keras TextVectorization, Embedding, and GlobalAveragePooling1D: sequence indices, vector lookup, and masking.
• Peters et al., Deep Contextualized Word Representations: ELMo and context-dependent representations.
• Bolukbasi et al., Debiasing Word Embeddings: evidence of stereotyped associations in learned word geometry and a proposed intervention.

The numerical examples are constructed for this lecture. The model templates require a prepared corpus and, for the pretrained experiment, a compatible local GloVe file. They do not establish a measured accuracy or a verified analogy result.

Feynman study lab

Explain it simply

An embedding represents a token by a learned list of numbers whose geometry reflects training patterns.

Concrete example. Vectors (1, 0) and (2, 0) have cosine similarity 1 because they point in the same direction.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Does cosine similarity 1 guarantee that two words mean the same thing?

Check your explanation after you try

No. Similarity depends on representation and training context. Static embeddings can mix senses and encode biases.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.