Deep Learning for Natural Language: Transformers and Self-Supervised Learning

A Transformer specifies how token representations interact. A learning objective specifies what those interactions should help predict. Self-supervised learning connects the two by constructing prediction targets from the data itself, allowing text to teach a model before a human supplies task-specific labels.

Learning goals

After this lesson, you should be able to explain learned self-attention, distinguish four different masking operations, construct masked-token and next-token targets, calculate token-weighted cross-entropy, and connect pretraining to a downstream task.

1. Review: make attention tunable

For input states X with L positions and d features, the simplified operation softmax_rows(XXᵀ)X mixes the original vectors using dot products. It introduces no internal projection parameters, although X may come from trainable layers.

Standard Transformer attention instead learns:

Q=XW_Q, K=XW_K, V=XW_V,

H=softmax_rows(QKᵀ/√d_k+M)V.

Queries and keys determine compatibility scores; values supply the mixed content. M encodes permitted attention pairs. Gradient-based optimization changes the projection weights to reduce the chosen training loss. Multi-head attention learns several sets of projections, concatenates the results, and projects back to the model width.

An encoder block also contains a position-wise feed-forward network, residual connections, and layer normalization. For a post-normalization variant:

U=LayerNorm(X+Dropout(MultiHead(X))),

Z=LayerNorm(U+Dropout(FFN(U))).

With suitable dimensions, X and Z both have shape B×L×d_model. Positional information makes order available. Repeating blocks creates progressively transformed contextual states, but does not by itself define a useful prediction task.

2. Padding is not automatically irrelevant

Padding makes different-length sequences fit a rectangular batch. It is an implementation device, not a real word. A zero token ID does not universally mean “ignore this position”: a model can look up a nonzero padding vector, add a position vector, or produce nonzero states through biases and attention.

Keras Embedding(mask_zero=True) can generate a mask, but subsequent operations must support and preserve it. Custom additions or layers can require explicit handling. In a manually built encoder, provide a padding visibility mask and check downstream pooling or loss separately.

For tokens [birds, cross, skies, PAD], the valid-key indicator is [1,1,1,0]. A bidirectional attention visibility matrix can be:

                 Keys
             birds cross skies PAD
query birds    1     1     1    0
query cross    1     1     1    0
query skies    1     1     1    0
query PAD      1     1     1    0

The padded query row may still produce a contextual state. That is acceptable if it is excluded wherever token states are pooled or scored. A padding-key mask alone does not make all padded outputs zero. All-padding documents need an explicit rejection or safe-empty policy.

3. Self-supervised means deriving targets from data

In supervised genre classification, someone supplies a label such as rock. In self-supervised language learning, a text supplies both input material and a target derived from itself.

Two common objectives are:

• Masked language modeling: hide or corrupt selected tokens, then predict their original identities.
• Causal language modeling: predict each subsequent token using the preceding sequence.

These objectives require no manually annotated genre or sentiment labels, but they still involve many human decisions: corpus selection, tokenization, corruption, architecture, and optimization. “Self-supervised” does not mean autonomous truth verification or learning without an objective.

The prediction task must prevent a trivial shortcut. Feeding an unchanged token to a bidirectional encoder and asking it to reproduce itself at every position can reward copying instead of learning useful context.

4. Masked language modeling: BERT’s central idea

Let original tokens be x₁,…,x_L. Choose a set S of eligible positions and construct corrupted input x̃. A basic masked-token objective is:

L_MLM=−(1/|S|)Σ_(i∈S) log Pθ(xᵢ | x̃).

The model receives the corrupted sequence; the target remains the original token, not the replacement token. Bidirectional attention lets the model use permitted information on both sides. Loss is evaluated at selected positions, not automatically at every token.

A worked target example

Original:       birds    cross    blue     skies    PAD
Corrupted:      birds    [MASK]   blue     skies    PAD
Loss weights:     0        1        0        0       0
Target scored:           cross
Attention valid:  1        1        1        1       0

The [MASK] position is visible to attention: it needs a contextual state from which to predict cross. Setting its attention visibility to zero would not be the same operation as replacing its content.

Original BERT’s corruption recipe

Original BERT selected about 15% of eligible token positions. Of those selected positions, 80% were replaced by [MASK], 10% by a random token, and 10% left unchanged. The target stayed original in all three cases. Padding and protected special tokens should not become ordinary prediction targets.

For 1,000 eligible tokens, the expected counts are 150 selected, comprising 120 masked, 15 random replacements, and 15 unchanged selected targets. These are expectations, not guaranteed counts for each batch. A randomly sampled replacement can also coincide with the original token.

This historical recipe is not a requirement for every masked model. Dynamic corruption can select new positions on later visits to a document. A short batch can have no selected positions, so implementations need a deliberate policy such as skipping it or using a documented selection rule.

5. Causal next-token prediction

A causal language model factorizes a sequence probability as:

P(x₁,…,x_L)=Πᵢ P(xᵢ | x₁,…,xᵢ₋₁),

with an appropriate start-of-sequence convention. Teacher-forced training supplies true preceding tokens and predicts the next token at each scored position.

For an invented sequence:

Full sequence:  BOS   birds cross skies EOS
Model inputs:  BOS   birds cross skies
Targets:       birds cross skies EOS

A causal attention mask lets query position i attend only to positions j≤i. The input at i is a preceding token relative to its next-token target, so attending to itself is valid. Allowing it to read the next input position would expose the target and leak future information.

Do not shift twice: some language-model libraries accept aligned labels and perform the shift internally, while a custom training loop may expect already shifted targets. Follow the specific model’s loss contract.

During training, positions can be calculated in parallel with the causal constraint. Ordinary autoregressive generation still creates tokens sequentially, often reusing cached keys and values. Sampling choices affect generated text but do not replace the need for the correct training mask.

6. Four masks that should not be confused

Mechanism	Changes	Typical representation
Padding attention mask	Which source positions can influence attention	Boolean permitted-key indicators
Causal attention mask	Which earlier/later position pairs are visible	Lower-triangular visibility matrix
Input corruption	Which content is supplied at selected positions	[MASK], a replacement ID, or unchanged input
Loss mask	Which predictions contribute to the objective	Per-position weights or a supported ignore-label convention

An ignore-label value such as −100 is a convention used by particular libraries and losses. It is not a token ID to feed into an embedding table, and it is not automatically supported by every Keras loss. The example below uses valid target IDs plus explicit zero/one loss weights.

7. Cross-entropy: normalize by scored tokens

For token probabilities p₁,…,p_V and correct vocabulary index y, cross-entropy is −log p_y. For position weights mᵢ∈{0,1}, average only over scored targets:

L=Σᵢmᵢℓᵢ / Σᵢmᵢ.

Suppose two selected tokens receive probabilities 0.5 and 0.25. Their losses are log 2 and log 4, giving L=(log 2+log 4)/2≈1.03972. Averaging their weighted losses over eight total positions instead would give about 0.25993, changing the gradient scale merely because more unscored positions were present.

Across an evaluation corpus, sum target losses and divide by the total number of scored tokens. An unweighted average of batch means is incorrect when batches contain different numbers of selected targets.

For a consistent autoregressive evaluation, perplexity=exp(mean token negative log-likelihood). Masked-token losses condition on a different corruption task and do not directly define the same left-to-right sequence likelihood; an exponentiated MLM loss should not be compared naively with causal perplexity. Tokenizers and evaluation corpora must also match for meaningful comparisons.

8. Keras: one explicit masked-token update

The following blocks form a small instructional script for Keras 3 with a TensorFlow backend. It uses invented token IDs and a simplified policy replacing every selected token with MASK. It does not reproduce BERT’s complete architecture, tokenizer, corruption recipe, or pretraining scale.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"

import numpy as np
import tensorflow as tf
import keras
from keras import ops

keras.utils.set_random_seed(42)
# 0=PAD, 1=UNK, 2=MASK, 3=birds, 4=cross, 5=skies,
# 6=fish, 7=swim, 8=rivers.
PAD, MASK, vocab_size, length = 0, 2, 9, 4
original = np.array([[3, 4, 5, 0], [6, 7, 8, 0]], dtype="int32")
selected = np.array([
    [False, True, False, False],
    [True, False, False, False],
])
assert not np.any(selected & (original == PAD))
assert selected.any()
corrupted = original.copy()
corrupted[selected] = MASK

ids = keras.Input(shape=(length,), dtype="int32")
valid = ops.not_equal(ids, PAD)
positions = ops.cumsum(ops.ones_like(ids), axis=-1) - 1
x = keras.layers.Embedding(vocab_size, 16)(ids)
x = x + keras.layers.Embedding(length, 16)(positions)
attention = keras.layers.MultiHeadAttention(num_heads=2, key_dim=8)
attended = attention(
    query=x, value=x,
    attention_mask=ops.expand_dims(valid, axis=1),
)
u = keras.layers.LayerNormalization(epsilon=1e-6)(x + attended)
ff = keras.layers.Dense(32, activation="relu")(u)
ff = keras.layers.Dense(16)(ff)
states = keras.layers.LayerNormalization(epsilon=1e-6)(u + ff)
logits = keras.layers.Dense(vocab_size)(states)
model = keras.Model(ids, logits)
assert model.output_shape == (None, length, vocab_size)

The head produces a vocabulary score vector for every position, giving shape B×L×V. Selected positions determine which of those scores train the model. The same mask that hides padding keys does not hide MASK tokens.

optimizer = keras.optimizers.Adam(learning_rate=1e-3)
weights = tf.convert_to_tensor(selected, dtype=tf.float32)
targets = tf.convert_to_tensor(original)
inputs = tf.convert_to_tensor(corrupted)
target_count = tf.reduce_sum(weights)
tf.debugging.assert_positive(target_count)

with tf.GradientTape() as tape:
    predictions = model(inputs, training=True)
    token_losses = keras.losses.sparse_categorical_crossentropy(
        targets, predictions, from_logits=True,
    )
    loss = tf.reduce_sum(token_losses * weights) / target_count

gradients = tape.gradient(loss, model.trainable_weights)
optimizer.apply_gradients([
    (gradient, variable)
    for gradient, variable in zip(gradients, model.trainable_weights)
    if gradient is not None
])
print("One illustrative MLM update; loss:", float(loss.numpy()))

This explicit reduction avoids assuming that a training API’s default sample-weight normalization equals division by selected-token count. The loop has no regularization losses; if you add such penalties, include them deliberately in the objective.

An unselected context token can still receive gradients through its influence on selected predictions. Zero direct loss weight does not freeze that token’s embedding or prevent the shared encoder from learning.

The script includes runtime checks for a compatible environment; it has not been used here to report an executed training result. A real pretraining system needs a fitted tokenizer, nonempty-input handling, protected special-token rules, repeated batches, reproducible corruption, validation, and checkpointing.

9. Pretraining, transfer, and task heads

Self-supervised pretraining learns parameters from a large text corpus. A later application can:

• Freeze the encoder and train a small classifier on its states.
• Fine-tune selected encoder layers or the full encoder with supervised labels.
• Continue self-supervised training on relevant domain text before supervised adaptation.

For genre classification, replace the vocabulary-prediction head with a genre head and pool real token states or use the checkpoint’s documented classification representation. A vocabulary head predicts token identity; it is not already a rock/pop/hip-hop classifier.

Keep the pretrained model’s tokenizer and special-token conventions. Randomly rebuilding the same architecture does not transfer its learned representations. Smaller learning rates and staged unfreezing can help adaptation, but select settings using validation evidence rather than assuming fine-tuning always improves results.

Original BERT combined masked language modeling with next-sentence prediction. That historical design is not the definition of all self-supervised encoders: later recipes such as RoBERTa changed the training choices. Model families differ in architecture, objectives, and usable context.

10. Evaluate what the objective actually teaches

A lower token-prediction loss indicates improvement on the specified text distribution and prediction task. It does not establish truthfulness, fairness, causal understanding, or good performance on every downstream task. A model can learn common errors and stereotypes as well as useful regularities.

For reliable evaluation:

• Split original documents before generating corrupted examples, keeping duplicates and related sources together as appropriate.
• Fit a new tokenizer only on the designated training corpus; retain a pretrained checkpoint’s existing tokenizer when adapting it.
• Keep target text hidden under the correct corruption or causal rule.
• Use a documented, reproducible corruption policy for MLM validation; changing random masks adds evaluation noise.
• Report total scored tokens, token-weighted loss, and downstream metrics on independent data.
• Check for overlap between pretraining material and evaluation documents when making claims about unseen examples.

More training text is not automatically more useful text. Domain match, coverage, data quality, and the intended use determine what transfers. Compare downstream performance with simpler supervised and embedding baselines, especially when labels or compute are scarce.

11. Exercises with answers

1. Parameters: Does the toy rule softmax(XXᵀ)X contain learned projection matrices? No. Standard Transformer attention adds W_Q, W_K, W_V, and multi-head output projections.
2. Padding: Does token ID zero force every later state to zero? No. Correct exclusion requires mask handling and an appropriate loss or pooling rule.
3. Original BERT: Out of 2,000 eligible tokens, what counts are expected under 15% selection and 80/10/10 corruption? 300 selected: 240 MASK replacements, 30 random replacements, and 30 unchanged selected targets.
4. Targets: If cross is replaced by MASK, which token should the model predict? cross, the original identity.
5. Visibility: Should MASK usually be hidden by the padding attention mask? No. It is a real input position whose contextual state is needed for prediction.
6. Causality: Inputs are [BOS, birds, cross] and next-token targets are [birds, cross, skies]. May the BOS query attend to the input birds position? No. That would expose its target.
7. Loss: Two scored tokens receive correct-token probabilities 0.5 and 0.5. What is their mean loss? −log(0.5)≈0.69315, regardless of how many unscored padding positions are present.
8. Zero targets: What should happen if no token is selected in a batch? Use an explicit policy such as skipping the update or applying a documented resampling rule, rather than dividing by zero.
9. Gradients: Can an unselected context token’s embedding be updated? Yes, through its contribution to predictions at selected positions.
10. Transfer: Does a pretrained masked-token head directly predict custom genre labels? No. Add a suitable downstream task head and train it with the corresponding labels.
11. Perplexity: Are exponentiated MLM loss and causal perplexity directly interchangeable? No. Their conditioning and scored targets differ.

Further reading and sources

• Vaswani et al., Attention Is All You Need: trainable attention and encoder–decoder masking.
• Devlin et al., BERT: bidirectional pretraining, masked-token corruption, and adaptation.
• Liu et al., RoBERTa: A Robustly Optimized BERT Pretraining Approach: the effect of pretraining recipe choices.
• Keras MultiHeadAttention: explicit attention masks and causal masking.
• TensorFlow automatic differentiation: gradient tapes and parameter gradients.
• Hugging Face data collators: task-specific batch preparation and label conventions; check the installed model’s shift and ignore-label behavior.

The loss calculations and invented sentences are teaching examples. The code demonstrates one objective and update structure, not a pretrained language model or a measured downstream result.

Feynman study lab

Explain it simply

Self-supervised learning builds prediction targets from the data itself.

Concrete example. A causal model can train on ‘the cat sat’ by predicting ‘cat’ from ‘the’, then ‘sat’ from ‘the cat’.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Does self-supervised mean there is no target or training objective?

Check your explanation after you try

No. Targets come from the data rather than separate human labels. Objective and attention mask determine what information the model may use.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.