Neural Networks and Deep Learning: Training Deep Neural Networks

A neural network learns a prediction rule by adjusting weights and biases. A deep network composes several transformations, allowing it to learn useful intermediate representations as well as the final prediction. This lecture connects that idea to the actual calculations behind training: forward propagation, loss, backpropagation, optimization, and regularization.

Learning goals

You should finish this lesson able to explain why depth is useful, trace a prediction through a dense network, calculate a gradient update, distinguish backpropagation from an optimizer, and design an experiment that detects overfitting. Familiarity with linear regression, vectors, derivatives, and basic probability helps; the examples introduce the essential notation.

We use supervised binary classification as a running example: predicting whether a target event occurs from a vector of measurements. A hypothetical heart-disease dataset supplies intuition only; the numerical examples below are synthetic and do not produce clinically validated risk estimates.

1. History and motivation: why deep learning?

The 1956 Dartmouth summer research project is widely treated as the founding event of AI as an organized field. Its 1955 proposal was written by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon. Ideas about computation and intelligence predated the workshop; neural networks also have a history extending before modern deep learning.

One useful teaching framework distinguishes three overlapping approaches:

Approach	What people specify	What the system learns	Main difficulty
Rule-based AI	Explicit rules, such as “if these conditions hold, take this action”	Often little or nothing from examples	Obtaining and maintaining enough reliable rules
Conventional machine learning	Inputs and often engineered features	A mapping from those features to predictions	Designing informative representations
Deep learning	Architecture, objective, data pipeline, and training procedure	Intermediate representations and the prediction rule jointly	Data quality, computation, optimization, and generalization

An expert system might encode medical knowledge in rules. An image classifier using conventional machine learning might first compute handcrafted edge or texture descriptors, then fit a classifier. A deep image model can learn features from pixels, with early layers sometimes detecting simple patterns and later layers combining them into task-relevant representations.

These are not clean replacements or rigid historical stages. Conventional machine learning includes nonlinear methods and some representation learning; deep models still require substantial human choices and may use engineered inputs. Learning features reduces a bottleneck without eliminating expertise.

Modern deep learning became practical through a combination of algorithmic improvements, larger datasets, and hardware capable of fast parallel numerical operations. GPUs accelerate many matrix operations; they do not themselves supply the learning rule. Depth can represent some compositional functions efficiently, but greater depth does not automatically improve results. On small tabular datasets, simpler models may be competitive or preferable.

2. From regression to neurons and layers

A neuron first computes an affine function of its inputs:

z = wᵀx + b, followed by a = φ(z).

Here x is an input vector, w contains weights, b is a bias, z is a preactivation, and φ is an activation function. The regression intercept becomes a bias; regression coefficients become weights. The neuron analogy is a useful computational metaphor, not a detailed biological model.

• The input layer holds the input variables; it is usually not counted as a trainable layer.
• A hidden layer transforms an intermediate representation.
• The output layer produces a score, probability, or numerical prediction.
• In a fully connected, or dense, layer every output unit has a weight for every input unit.
• The architecture specifies the layers, widths, connections, activations, and other structural choices.

Logistic regression is a neural network with no hidden layers: p = sigmoid(wᵀx+b). “Deep” usually means multiple learned transformation layers, although conventions for counting depth differ. Always state whether a layer count includes the output layer.

Why nonlinear activations matter

Without nonlinearities, stacking affine layers still gives an affine function:

W₂(W₁x+b₁)+b₂ = (W₂W₁)x + (W₂b₁+b₂).

The extra layer does not create a more general nonlinear decision boundary. Nonlinear activation functions make the composition more expressive.

Activation	Formula	Common use and limitation
ReLU	max(0,z)	Hidden layers; gradient is zero on negative inputs, so units can become inactive
Sigmoid	1/(1+exp(−z))	Binary output probabilities; saturates at large positive or negative z
Tanh	(exp(z)−exp(−z))/(exp(z)+exp(−z))	Bounded hidden states; also saturates
Linear	z	Unbounded regression outputs

For mutually exclusive multiclass prediction, softmax turns output scores into probabilities summing to one. For multiple labels that can occur together, independent sigmoid outputs are often appropriate instead.

3. Network notation, dimensions, and parameter counts

Use column vectors for a single example. Let a⁽⁰⁾=x and, for layer l,

z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾, and a⁽ˡ⁾ = φˡ(z⁽ˡ⁾).

If the preceding layer has d units and the current layer has h units, W has shape h×d and b has shape h×1. That dense layer has hd+h trainable parameters. For a batch of B column examples, A has shape d×B, and the bias is broadcast across columns. Some libraries instead place examples in rows; the conceptual calculation is the same, but matrix orientations differ.

Worked parameter count. A network with 10 inputs, hidden layers of 16 and 8 units, and one output has:

• First hidden layer: 16×10+16 = 176 parameters.
• Second hidden layer: 8×16+8 = 136 parameters.
• Output layer: 1×8+1 = 9 parameters.

Total: 321 parameters. The input layer contributes no trainable parameters. Width and depth affect capacity and computational cost; parameter count alone does not determine predictive quality.

4. Define the learning problem before training

Suppose the target y is 1 when a specified event occurs and 0 otherwise. Define the event, prediction horizon, and information available at prediction time before collecting features. A measurement recorded after the outcome can create leakage even if it seems highly predictive.

The task determines the meaning of the inputs and targets. The practitioner selects the representation, architecture, output constraint, loss, optimizer, learning rate, batch size, and regularization. These choices are hyperparameters; weights and biases are parameters learned during training.

Split data into training, validation, and test sets before fitting preprocessing. Estimate means, standard deviations, imputation rules, and learned encodings using training data only. For a continuous feature, standardization commonly uses x̃=(x−μ_train)/s_train, with explicit handling of zero-variance features.

Keep repeated records for the same person in the same split when evaluating generalization to new people. For forecasting, use chronological splits and respect when labels become available. Validation data guides model selection; the test set is reserved for final assessment.

5. The four steps of training

Step 1: forward pass

Apply the network’s transformations to a batch to calculate predictions. For binary classification the final layer can return a logit z, with probability p=sigmoid(z). A logit is an unconstrained score; it is not itself a probability.

Step 2: calculate the loss

Binary cross-entropy for one example is:

ℓ(y,p) = −[y log(p) + (1−y) log(1−p)].

Natural logarithms are used throughout. A correct, confident prediction has a small loss; a confident wrong prediction has a large loss. For y=1, predicting p=0.9 gives loss about 0.1054, whereas p=0.1 gives about 2.3026.

For a batch, use L=(1/B)Σᵢℓ(yᵢ,pᵢ). Regularization penalties may be added to this objective. For numerical stability, use a library loss accepting logits directly, such as BCEWithLogitsLoss; do not apply sigmoid first when calling that loss. Equivalently, binary cross-entropy from a logit is max(z,0)−yz+log(1+exp(−|z|)).

Use squared error for a suitable regression problem, or multiclass cross-entropy for a single class among several categories. Loss choice expresses assumptions and costs; accuracy alone is not a training objective for gradient methods because thresholded decisions do not provide useful ordinary gradients.

Step 3: backpropagation

Backpropagation applies the chain rule backward through the computational graph to compute derivatives of the scalar loss with respect to trainable parameters. Automatic differentiation systems record operations and reuse intermediate values to perform this efficiently.

Backpropagation computes gradients; the optimizer uses those gradients to choose updates. They are distinct operations. Backpropagation works on CPUs too; GPU acceleration improves the speed of supported operations.

Step 4: update parameters

For plain gradient descent with learning rate α:

θ ← θ − α∇θL,

where θ collects every weight and bias. The minus sign moves locally downhill. A step that is too large can increase the loss; deep-network objectives are generally nonconvex, so training does not guarantee a global minimum.

A batch is a subset used for one gradient calculation. An iteration is one update. An epoch is one pass through the training dataset. With 1,000 examples and batches of 100, one epoch has 10 updates. Mini-batch SGD estimates a gradient from each batch; full-batch gradient descent uses all training examples at once.

6. A complete numerical forward and backward pass

Consider one input, one ReLU hidden neuron, and one sigmoid output:

h=ReLU(wx+b), z=vh+c, p=sigmoid(z).

Take x=2, y=1, w=0.5, b=0, v=1, c=0. Use binary cross-entropy and plain SGD with α=0.1.

Forward calculation

The hidden preactivation is 0.5×2+0=1, so h=1. The output logit is 1×1+0=1. Thus p≈0.731059 and ℓ=−log(p)≈0.313262.

Backward calculation

For sigmoid followed by binary cross-entropy, the derivatives simplify to:

∂ℓ/∂z = p−y ≈ −0.268941.

This follows from ∂ℓ/∂p=(p−y)/[p(1−p)] and ∂p/∂z=p(1−p). Their product cancels the denominator.

The output-layer gradients are:

∂ℓ/∂v = (p−y)h ≈ −0.268941, and ∂ℓ/∂c = p−y ≈ −0.268941.

The hidden preactivation is positive, so the ReLU derivative is 1. Therefore:

∂ℓ/∂w = (p−y)v×1×x ≈ −0.537883, and ∂ℓ/∂b = (p−y)v×1 ≈ −0.268941.

Simultaneous update and check

Using gradients evaluated at the original parameter values gives:

w_new≈0.553788, b_new≈0.026894, v_new≈1.026894, c_new≈0.026894.

Recompute the forward pass: h_new≈1.134471, z_new≈1.191875, and p_new≈0.76708. The loss is now about 0.2652, smaller than 0.3133. All updates used the old parameters; using an already updated v in the calculation for w would change the algorithm.

This example has one hidden layer. A deeper network repeats exactly the same chain-rule logic across more layers.

7. Backpropagation through many layers

Let δ⁽ˡ⁾=∂ℓ/∂z⁽ˡ⁾. For a sigmoid binary output with cross-entropy, δ⁽ᴸ⁾=p−y. For an elementwise hidden activation:

δ⁽ˡ⁾ = [(W⁽ˡ⁺¹⁾)ᵀδ⁽ˡ⁺¹⁾] ⊙ φˡ′(z⁽ˡ⁾),

where ⊙ means elementwise multiplication. Then:

∂ℓ/∂W⁽ˡ⁾ = δ⁽ˡ⁾(a⁽ˡ⁻¹⁾)ᵀ, and ∂ℓ/∂b⁽ˡ⁾ = δ⁽ˡ⁾.

For a mean batch loss, average the per-example contributions. These equations describe ordinary dense feedforward layers; more complex operations have their own local derivatives.

Repeated multiplication can shrink or amplify gradients, producing vanishing or exploding gradients. Saturated sigmoid units have small derivatives, making this especially troublesome in deep hidden stacks. ReLU-family activations, suitable initialization, normalization, and residual connections can help, but none guarantees successful training. Gradient clipping can limit excessively large updates; it does not repair incorrect labels or leakage.

Do not initialize every hidden unit identically: symmetry can make them learn the same feature. Random initialization breaks that symmetry. He-style initialization is commonly paired with ReLU; Xavier-style initialization is often used with tanh-like activations. The goal is to keep signal and gradient scales manageable through the network.

8. SGD, momentum, and Adam

Plain SGD uses the current batch gradient. Momentum maintains a moving direction, reducing some oscillation and accumulating consistent movement. Adam tracks moving estimates of both gradients and squared gradients, adapting updates by parameter.

For gradient gₜ, Adam uses elementwise operations:

mₜ=β₁mₜ₋₁+(1−β₁)gₜ, vₜ=β₂vₜ₋₁+(1−β₂)gₜ².

Starting with m₀=v₀=0, bias correction gives m̂ₜ=mₜ/(1−β₁ᵗ) and v̂ₜ=vₜ/(1−β₂ᵗ). The update is:

θₜ=θₜ₋₁−α m̂ₜ/(√v̂ₜ+ε).

The small ε stabilizes division. Adam often makes an effective starting point, but it still needs learning-rate selection and does not guarantee better generalization than SGD. Learning-rate schedules can reduce step sizes later in training. Batch size affects gradient noise, memory requirements, and updates per epoch, so changing it can require retuning other settings.

9. Regularization: learning patterns that generalize

A model can fit training examples well while performing poorly on new ones. Track training and validation performance together. A widening gap can indicate overfitting, but distribution differences or a flawed split can also explain it.

Early stopping

Evaluate validation loss after each epoch. Save parameters whenever it improves by a chosen meaningful amount. Stop after a chosen patience interval without sufficient improvement and restore the best saved checkpoint.

For validation losses 0.50, 0.42, 0.39, 0.40, 0.43, patience of two epochs and zero minimum improvement stops after epoch 5 and restores epoch 3. Training loss may still be falling. Early stopping selects the checkpoint based on generalization evidence, not on reaching the minimum training loss. Repeatedly tuning against validation data can itself overfit that set; keep the test set separate.

Dropout

During training, dropout randomly masks individual activations. Let q be the probability of dropping an activation, and let mⱼ∼Bernoulli(1−q). Inverted dropout computes:

ãⱼ=mⱼaⱼ/(1−q).

This preserves the activation’s expectation: E[ãⱼ]=aⱼ. Standard dropout is disabled at evaluation, when the unmasked activations are used without this training-time scaling. Preserving intermediate expectations does not mean an entire nonlinear network’s prediction equals an exact ensemble average.

For a=[2,4], q=0.5, and mask m=[1,0], the training activation is [4,0]. At evaluation it is [2,4]. Dropout masks outputs temporarily; it does not permanently delete neurons or automatically set their stored weights to zero.

A rate of 0.5 is a historical example for some dense hidden layers, not a universal setting. Tune the rate on validation data; too much dropout can cause underfitting. Different layers and architectures may need different treatment, and combining dropout with normalization requires care.

Other useful controls

Weight penalties, smaller networks, and appropriate data augmentation can also help. For a squared-weight penalty J=L+(λ/2)ΣWⱼₖ², the weight gradient gains λW; specify whether biases are penalized. Decoupled weight decay, as in AdamW, differs from adding this penalty to Adam’s loss. More high-quality representative data can help, while more leaked or mislabeled data may not.

10. A practical training recipe

Split the data; fit preprocessing using training data only.
Initialize model parameters and optimizer state.
For each epoch:
    Enable training behavior, including dropout.
    For each training batch:
        Clear accumulated gradients.
        Compute output logits with a forward pass.
        Compute the mean loss from logits and targets.
        Backpropagate to compute gradients.
        Ask the optimizer to update parameters.
    Enable evaluation behavior; disable gradient recording.
    Compute validation loss and task-relevant metrics.
    Save an improved checkpoint; otherwise update patience.
    Stop when the early-stopping rule is met.
Restore the best checkpoint and assess the held-out test set.

Switching to evaluation mode and disabling gradient recording serve different purposes. The first changes behavior such as dropout and batch normalization; the second avoids building the gradient graph. Many libraries accumulate gradients by default, so explicitly clearing them prevents accidental accumulation across batches.

Start with a simple baseline, such as logistic regression for tabular binary classification. Check whether a small network can fit a tiny training subset before scaling up. Record seeds, splits, preprocessing, hyperparameters, and software versions; some GPU operations can still limit exact reproducibility.

Observation	Possible cause	Useful next check
Both losses remain high	Underfitting, unsuitable inputs, or failed optimization	Inspect labels, preprocessing, gradient scales, and baseline performance
Training loss falls; validation loss rises	Overfitting or a split mismatch	Check leakage and sampling; try early stopping or reduced capacity
Loss becomes non-finite	Excessive learning rate or unstable arithmetic	Inspect inputs, use stable loss functions, and check gradients
Validation performance is implausibly good	Leakage or duplicate records across splits	Audit feature timing and group membership
Accuracy is high but rare events are missed	Class imbalance or unsuitable decision threshold	Inspect precision, recall, confusion matrix, and precision–recall curves

Probabilities also need evaluation: consider calibration and log loss, not only ranking or accuracy. Choose decision thresholds using validation data and the application’s error costs. Predictive association does not establish causation, and performance on one population does not establish performance after distribution shift.

11. Practice questions

1. A dense network has 3 inputs, 4 hidden units, and 1 output. How many weights and biases does it have?
2. Why cannot two affine layers without a nonlinear activation solve every nonlinear classification problem?
3. For a sigmoid output, y=0 and p=0.8, calculate binary cross-entropy and the derivative with respect to the output logit.
4. A parameter is 0.7, its gradient is −0.3, and the SGD learning rate is 0.05. What is its new value?
5. With dropout rate q=0.2, a retained activation originally equals 3. What value is used during training? What is used at evaluation?
6. Training loss decreases every epoch, but validation loss is lowest at epoch 8 and worsens afterward. Which checkpoint should early stopping restore?
7. Why is calculating standardization statistics from the entire dataset before splitting a problem?
8. A dataset has 1,050 examples and batch size 100. If the final incomplete batch is retained, how many updates occur per epoch?

Answers and explanations

1. Hidden weights: 4×3=12; hidden biases: 4; output weights: 4; output bias: 1. Total 21.
2. Their composition is still affine. Nonlinear activations allow more general decision boundaries; depth alone is insufficient.
3. Loss −log(0.2)≈1.60944; logit derivative p−y=0.8. A downward logit update would reduce this incorrect positive prediction.
4. 0.7−0.05×(−0.3)=0.715. A negative derivative leads to an increase in the parameter.
5. During training the retained value is 3/0.8=3.75; dropped values would be zero. At evaluation use 3, with dropout disabled.
6. Restore epoch 8, assuming validation loss is the chosen criterion. A later epoch with lower training loss need not generalize better.
7. Validation and test observations influence the fitted transformation, leaking information into training. Fit preprocessing on the training split and apply it unchanged to the others.
8. 11 updates: ten full batches and one batch of 50. For a dataset-wide mean loss report, weight batch means by their example counts.

12. Further reading and sources

• McCarthy, Minsky, Rochester, and Shannon, A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence: the original 1955 proposal for the 1956 project.
• Goodfellow, Bengio, and Courville, Deep Learning, Chapter 6: Deep Feedforward Networks: network representations and differentiation.
• Goodfellow, Bengio, and Courville, Chapter 7: Regularization for Deep Learning: early stopping, parameter penalties, and dropout.
• Goodfellow, Bengio, and Courville, Chapter 8: Optimization for Training Deep Models: optimization challenges and algorithms.
• Kingma and Ba, Adam: A Method for Stochastic Optimization: the original algorithm and bias corrections.
• Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting: the original dropout paper.
• PyTorch documentation, BCEWithLogitsLoss and Dropout: stable binary loss and training-versus-evaluation dropout behavior.

The numerical examples and exercises are constructed for this lecture. They illustrate the mathematics rather than the measured performance of a trained application.

Feynman study lab

Explain it simply

A neural network predicts, measures error, computes how weights affect that error, and adjusts them.

Concrete example. With weight 2 and loss gradient 3, a learning rate of 0.1 gives the gradient-descent update 2 − 0.1 × 3 = 1.7.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Does a lower training loss guarantee better performance on new examples?

Check your explanation after you try

No. A model can memorize training data. Use validation data and evaluate generalization independently.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.