Deep Learning for Natural Language: The Basics

Text carries explanations, opinions, stories, instructions, and conversations. Natural language processing, or NLP, turns this unstructured material into representations that computational models can use. This lesson starts with a simple question: how can a neural network accept a sentence when it expects numbers?

Learning goals

You should finish able to explain the standardize–tokenize–index–encode pipeline, calculate count and multi-hot document vectors, construct n-grams, identify bag-of-words limitations, and train a Keras text classifier without fitting its vocabulary on held-out data.

1. Why natural language processing?

Large amounts of human communication and recorded knowledge exist as text, from scientific articles and support tickets to books and messages. NLP tasks include sentiment classification, topic discovery, information extraction, search, translation, and text generation.

These tasks are different. Predicting the genre of a text is supervised classification; generating the next token is a different prediction problem. Success at a narrow benchmark does not establish human-like understanding, factual reliability, or causal reasoning.

Unlike a spreadsheet column containing age, a sentence has variable length, context-dependent meanings, and a structure that matters. The same word can have different meanings, while different words can express similar ideas. Our first representation deliberately simplifies these difficulties so we can understand the complete learning pipeline.

2. STIE: from strings to numerical representations

STIE is a useful teaching mnemonic: standardize, tokenize, index, encode. It describes common stages, not a universal mandatory recipe. Modern NLP systems can use different tokenizers and representations.

S — Standardize

Standardization makes selected variations consistent. For example, lowercasing and removing punctuation transforms the invented text “The CAT, the cat!” into “the cat the cat”.

But cleaning is a modeling decision, not automatically an improvement:

• Lowercasing merges “Apple” and “apple,” which can remove useful distinctions.
• Punctuation carries sentiment, sentence structure, and contractions. Removing the apostrophe in “don’t” can produce “dont,” not “do not.”
• Accent removal can merge distinct words or names. Unicode normalization and accent stripping are different operations.
• Stop-word removal can accidentally remove important negation, as when “not good” loses “not.”
• Stemming applies rules to shorten forms and can produce nonwords; lemmatization attempts to identify dictionary forms using linguistic analysis. Neither is required for every task.

Keras’s default TextVectorization standardization lowercases and strips punctuation. It does not automatically strip accents, remove stop words, stem, or lemmatize. If you customize preprocessing, apply the same rule during training and prediction.

T — Tokenize

Tokenization divides text into units. A whitespace tokenizer gives [“the”, “cat”, “the”, “cat”]. This is easy to inspect, but whitespace is not a universal word-boundary rule: languages such as Chinese often require different segmentation, and spelling variants can expand the vocabulary.

Tokens can also be characters, subwords, or punctuation-aware units. Subword tokenizers can represent unfamiliar words through known pieces. That is a later step beyond this word-based baseline.

I — Index

Build a vocabulary from training text and assign tokens distinct integer IDs. An ID is a lookup key, not a semantic measurement: token 8 is not “twice as meaningful” as token 4.

Unseen or excluded tokens map to an out-of-vocabulary marker, often written [UNK]. This lets a model accept new text, but different unknown words then collapse to the same representation. Rare tokens omitted by a vocabulary cap also become unknown; they are not necessarily silently deleted.

Special indices depend on the implementation and output mode. For example, Keras integer-sequence output reserves 0 for padding/masking and 1 for unknown tokens. Its bag-of-words output uses an unknown bucket without that sequence-padding slot. Inspect the actual vocabulary rather than assuming all modes share identical IDs.

E — Encode

An encoding turns IDs into model inputs. One-hot encoding represents a token by a basis vector: one entry is 1 and all others are 0. Integer sequences can instead feed an embedding lookup, which learns dense vectors; document-level encodings can aggregate tokens immediately.

You normally do not need to construct a giant one-hot matrix for every document. Direct indexed counting produces the same count vector more efficiently.

3. A complete one-hot and bag-of-words example

For a hand calculation, use this fixed conceptual vocabulary, including one unknown bucket and no padding slot:

Index	Token	One-hot vector
0	[UNK]	[1,0,0,0,0,0]
1	the	[0,1,0,0,0,0]
2	cat	[0,0,1,0,0,0]
3	sat	[0,0,0,1,0,0]
4	on	[0,0,0,0,1,0]
5	mat	[0,0,0,0,0,1]

For “the cat cat”, the token IDs are [1,2,2]. Each token has a six-dimensional one-hot vector, so the conceptual token matrix has shape 3×6. Another document can have a different number of rows, making direct use by a fixed-width dense layer inconvenient without additional handling.

Count encoding

Sum the one-hot vectors across token positions:

c(d)=Σₜ e(tokenₜ).

The document becomes [0,1,2,0,0,0]. The cat coordinate records two occurrences, while the the coordinate records one. The output width depends on the vocabulary, not document length.

Multi-hot encoding

Record only whether each token occurs:

mⱼ(d)=1 if cⱼ(d)>0, otherwise 0.

The same document becomes [0,1,1,0,0,0]. Elementwise logical OR of one-hot token vectors gives the same result. “The cat” and “the cat cat” now have identical vectors, so repetition information is lost.

For “the dog”, dog is unknown, giving count and multi-hot vectors [1,1,0,0,0,0] in this vocabulary. Two unknown words would increase the unknown count to two while leaving its multi-hot value at one.

Count and multi-hot vectors are bag-of-words representations. A dense classifier receives a batch matrix of shape B×V, where B is batch size and V is vocabulary width. The text has variable length, but every document vector has V coordinates.

A related representation: TF–IDF

Term frequency–inverse document frequency reweights counts so very common corpus-wide terms can carry less weight than more distinctive terms. Exact smoothing and normalization formulas vary by library. It is a useful baseline, not a learned contextual understanding of language. Any fitted document-frequency statistics must come from training data only.

4. N-grams: preserve short phrases

A unigram is one token; a bigram is two consecutive tokens; a trigram is three. For “not very good”:

• Unigrams: not, very, good.
• Bigrams: not very, very good.
• Trigram: not very good.

For a document with L tokens, the number of contiguous n-gram occurrences is max(L−n+1,0). There are L unigram occurrences and L−1 bigram occurrences when L≥1. Repetition can make the number of distinct n-grams smaller than the occurrence count.

Adding bigrams lets the representation distinguish “not good” from a document containing the same words in different local arrangements. It does not restore the full sequence once those n-grams are aggregated into a bag. Different passages can still share an n-gram count vector.

With V possible word types, up to V² ordered bigram types are conceivable, although a real corpus observes only a subset. At V=5,000, that theoretical count is 25 million. Expanding max_tokens from 5,000 to 20,000 allows more observed features, not every possible bigram. The vocabulary budget is shared by whatever n-gram orders you select.

In Keras, ngrams=2 includes unigrams and bigrams; ngrams=(2,) selects bigrams only. Larger vocabularies increase the dense head’s parameter count and may retain noisy or identifying phrases, so validate the trade-off.

5. What one-hot vectors and bags cannot express directly

Word order

Unigram count vectors for “the cat sat on the mat” and “the mat sat on the cat” are identical. They contain the same token counts despite very different interpretations. This loss of order comes from aggregation; a sequence of one-hot vectors still preserves token positions.

Sparsity and dimension

If a document uses 100 distinct tokens from a 20,000-feature vocabulary, at most 0.5% of its coordinates are active, so at least 99.5% are zero. Sparse storage can avoid materializing those zeros, and sparse linear models are often strong and efficient baselines.

A dense float32 matrix for 49,000 documents and 20,000 features requires 49,000×20,000×4=3,920,000,000 bytes, approximately 3.92 GB or 3.65 GiB, before model weights and training state. Processing batches avoids storing that complete dense matrix at once, though it does not eliminate all computational costs.

Semantic similarity

Distinct one-hot token vectors have dot product zero and cosine similarity zero. Thus movie and film are no closer geometrically than movie and volcano in this representation. IDs likewise convey no semantic distance.

This does not mean a trained classifier can never learn that movie and film have similar predictive roles. Its learned weights may respond similarly to both if the training data support that. The limitation is that the initial token representation supplies no built-in semantic similarity and no direct help with unseen synonyms.

Embeddings learn dense token vectors; contextual models further change a token’s representation with its surrounding text. Those ideas motivate subsequent NLP architectures, but a transparent bag-of-words baseline remains valuable.

6. Inspect the encodings in Keras

Use a separate environment with compatible Keras 3, TensorFlow, and NumPy. We select the TensorFlow backend because the example includes string preprocessing inside the model. See the Keras setup guide and TextVectorization API.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"

import numpy as np
import tensorflow as tf
import keras

vocabulary = ["the", "cat", "sat", "on", "mat"]
count_layer = keras.layers.TextVectorization(
    vocabulary=vocabulary, output_mode="count",
)
multi_hot_layer = keras.layers.TextVectorization(
    vocabulary=vocabulary, output_mode="multi_hot",
)
samples = tf.constant(["The cat cat!", "the dog"])
print(count_layer.get_vocabulary())
counts = count_layer(samples).numpy()
presence = multi_hot_layer(samples).numpy()
np.testing.assert_array_equal(counts, [[0, 1, 2, 0, 0, 0], [1, 1, 0, 0, 0, 0]])
np.testing.assert_array_equal(presence, [[0, 1, 1, 0, 0, 0], [1, 1, 0, 0, 0, 0]])

Here the vocabulary is supplied explicitly for a deterministic teaching example, so adapt() is unnecessary. In a real classifier, learn it from the training corpus instead.

max_tokens caps total vocabulary size including reserved entries. With max_tokens=5000, multi-hot output can retain at most 4,999 ordinary vocabulary entries plus its unknown bucket; integer output has at most 4,998 ordinary entries plus padding and unknown. pad_to_max_tokens=True fixes a bag output’s width at the cap even when fewer entries were learned. This pads feature coordinates, not token positions.

7. Application: classify lyric genre carefully

The supplied course outline describes about 49,000 lyrics, a rock majority near 50%, and a dense bag-of-words model reaching about 72% test accuracy. These are unverified course-reported figures: the corpus, split, licenses, and exact experiment have not been supplied or reproduced here.

On directly comparable evaluation cases, 72% versus 50% is a 22 percentage-point gain and a relative error reduction of (50−28)/50=44%. That arithmetic alone establishes neither statistical significance nor generalization to new artists.

Genre labels can overlap. A single softmax class assumes each document has one chosen label. If the task allows several simultaneous genres, use an appropriate multi-label formulation instead. Also remember that lyrics alone omit melody, instrumentation, and production, so text classification is not equivalent to classifying the full audio recording.

Split the corpus before fitting vocabulary

Use one UTF-8 .txt file per document in this structure:

genre_text/
  train/
    hip-hop/
    pop/
    rock/
  validation/
    hip-hop/
    pop/
    rock/
  test/
    hip-hop/
    pop/
    rock/

Create these splits yourself before running the example. Deduplicate versions of the same song and keep related versions together. For evaluation on unseen artists, use an artist-grouped split; for future releases, respect release time. Random document splits can inflate performance by allowing artist-specific phrases, duplicates, or album metadata across splits.

Remove accidental label leakage such as genre headers or filenames included in the text. The loader below derives labels from folders but does not add folder names to the document content. Use only a corpus you are authorized to process; the example supplies no copyrighted lyric excerpts.

from pathlib import Path

keras.utils.set_random_seed(42)
root = Path("genre_text")
class_names = ["hip-hop", "pop", "rock"]

def load_split(name, shuffle):
    return keras.utils.text_dataset_from_directory(
        root / name, labels="inferred", label_mode="int",
        class_names=class_names, batch_size=64,
        shuffle=shuffle, seed=42,
    ).prefetch(tf.data.AUTOTUNE)

train_ds = load_split("train", True)
val_ds = load_split("validation", False)
test_ds = load_split("test", False)

vectorizer = keras.layers.TextVectorization(
    max_tokens=20000,
    standardize="lower_and_strip_punctuation",
    split="whitespace", ngrams=2,
    output_mode="multi_hot", pad_to_max_tokens=True,
)
vectorizer.adapt(train_ds.map(lambda text, label: text))
print("Learned vocabulary entries:", len(vectorizer.get_vocabulary()))

adapt() learns token frequencies and the selected vocabulary. Adapting on validation or test text leaks held-out distribution information even without using its labels. The trained vectorizer must remain fixed while evaluating those sets.

8. Build and train a dense bag-of-words classifier

inputs = keras.Input(shape=(), dtype="string", name="document")
x = vectorizer(inputs)
x = keras.layers.Dense(64, activation="relu")(x)
x = keras.layers.Dropout(0.5)(x)
outputs = keras.layers.Dense(len(class_names), activation="softmax")(x)
model = keras.Model(inputs, outputs)
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[keras.metrics.SparseCategoricalAccuracy(name="accuracy")],
)
history = model.fit(
    train_ds, validation_data=val_ds, epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", mode="min", patience=3,
        restore_best_weights=True,
    )], verbose=2,
)

Each document enters as one string. The vectorizer returns 20,000 features, the hidden layer produces 64 activations, and the softmax head predicts one of three genres. Integer labels pair with sparse categorical cross-entropy.

The first dense layer has 20,000×64+64=1,280,064 parameters. At 5,000 input features it would have 320,064. Adding the three-class output gives 64×3+3=195 more parameters. The vectorizer’s vocabulary is fitted state, while dropout adds no trainable parameters.

The dropout rate 0.5 is an illustrative setting to validate, not a rule. During training, retained activations are scaled to preserve their expectation; dropout is disabled during evaluation. More vocabulary entries and bigrams can improve a task while also increasing overfitting risk. Compare a unigram model, a bigram-augmented model, and a smaller head using validation data before touching the test set.

9. Evaluate the chosen model and baseline

Choose the majority class from training labels, then apply that fixed prediction to the test set. Its test accuracy depends on the test label distribution; it is not automatically the training majority percentage.

train_labels = np.concatenate([label.numpy() for _, label in train_ds])
majority = int(np.bincount(train_labels, minlength=len(class_names)).argmax())

# The test loader is deterministic and unshuffled, preserving row alignment.
y_true = np.concatenate([label.numpy() for _, label in test_ds])
test_results = model.evaluate(test_ds, return_dict=True, verbose=0)
probabilities = model.predict(test_ds, verbose=0)
predictions = probabilities.argmax(axis=1)
print("Majority class:", class_names[majority])
print("Majority test accuracy:", float(np.mean(y_true == majority)))
print("Neural-network test metrics:", test_results)

confusion = np.zeros((len(class_names), len(class_names)), dtype=np.int64)
np.add.at(confusion, (y_true, predictions), 1)
support = confusion.sum(axis=1)
recall = np.divide(
    confusion.diagonal(), support,
    out=np.zeros(len(class_names), dtype=float), where=support != 0,
)
print("Confusion matrix: true rows, predicted columns")
print(confusion)
print("Per-class recall:", dict(zip(class_names, recall.tolist())))

model.save("genre_bow.keras")
example = tf.constant(["A quiet lantern shines beside the road."])
print("Invented example probabilities:", model(example, training=False).numpy())

The saved model includes its learned vocabulary and standardization settings. Save the class-name order too. An invented sentence outside the corpus’s domain is useful for testing the interface, but its predicted genre need not be meaningful.

Report accuracy alongside per-class results and, where appropriate, macro-averaged F1, log loss, calibration, and uncertainty. A majority-only classifier can achieve 50% while failing every minority class. Compare a sparse linear classifier on count or TF–IDF features; a neural network is not automatically the strongest bag-of-words model.

For efficiency, consider vectorizing batches in a data pipeline or using sparse features with a compatible model. This example keeps preprocessing in the model for clarity and a consistent prediction interface. Neither choice changes the requirement to fit vocabulary and related statistics on training data alone.

10. Exercises with answers

1. Counts versus presence: With vocabulary [UNK], red, blue, encode “red red blue.” Count: [0,2,1]; multi-hot: [0,1,1].
2. Unknown words: With that vocabulary, what is the count vector for “green green red”? [2,1,0]; both unknown words accumulate in the unknown bucket.
3. Order: Why do “cats chase dogs” and “dogs chase cats” share a unigram count vector? Aggregation retains frequencies but discards token order.
4. Bigrams: How many bigram occurrences are in a five-token sentence? Four. There are five unigram occurrences, for nine combined occurrences before deduplication.
5. Vocabulary cap: With Keras multi-hot output and max_tokens=5000, what is the maximum number of ordinary vocabulary entries? 4,999, because one slot is reserved for unknown tokens.
6. Parameters: How many parameters does a 5,000-feature input followed by a 32-unit dense layer have? 5,000×32+32=160,032.
7. Sparsity: A 10,000-feature multi-hot vector has 80 active entries. What share is zero? 99.2%.
8. Similarity: What is the cosine similarity of distinct one-hot vectors? Zero. Learned downstream weights can still assign the words similar predictive effects.
9. Cleaning: Why can removing “not” hurt sentiment classification? It can turn a negative phrase into the same representation as its positive counterpart.
10. Leakage: Is adapting a vocabulary on test text acceptable because labels are hidden? No. It lets held-out distribution information influence the representation. Fit it on training text only.
11. Baseline gain: What is the absolute gain from 50% to 72% accuracy? 22 percentage points, not a claim that the experiment has been reproduced.

Further reading and sources

• Keras TextVectorization: vocabulary fitting, special tokens, output modes, and n-grams.
• Keras text data loading: directory-based text datasets and label formats.
• TensorFlow basic text classification: a complete text-learning pipeline and training-only vocabulary adaptation.
• scikit-learn text feature extraction: sparse count features, TF–IDF, and bag-of-words limitations.
• Keras Embedding: the next step from token indices to learned dense representations.

The numerical vectors and exercises are constructed teaching examples. The lyric-corpus size and reported accuracy remain attributed course figures rather than independently verified results. The training template requires a prepared corpus and has not been used here to claim a measured test score.

Feynman study lab

Explain it simply

A text model needs numerical representations, but representation choices can discard meaning.

Concrete example. A bag-of-words encoding gives the same word counts to ‘dog bites person’ and ‘person bites dog’.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Why can a model using only these counts fail on this distinction?

Check your explanation after you try

Word counts discard order. A representation retaining sequence or context is needed to distinguish these two sentences reliably.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.