Deep Learning for Computer Vision: Building Convolutional Neural Networks from Scratch

Images contain structure: neighboring pixels form edges, edges combine into shapes, and shapes help identify objects. A convolutional neural network, or CNN, builds that structure into its architecture through local connections and shared filters. This lesson develops the calculations behind convolution and pooling, then builds a Fashion-MNIST classifier using Keras.

Learning goals

You should finish able to calculate a feature map by hand, distinguish translation equivariance from invariance, track tensor shapes through a CNN, count its trainable parameters, and build a classification experiment with separate training, validation, and test data.

1. Why not start with dense layers on raw pixels?

A grayscale 28×28 image contains 784 scalar values. Flattening it to a length-784 vector and connecting it to 256 hidden units creates 784×256+256=200,960 trainable parameters in that first dense layer alone.

For a 224×224 RGB image, the corresponding input has 150,528 values. A dense layer with 256 outputs would need 150,528×256+256=38,535,424 parameters. The number grows with image area, even if the visual patterns of interest are small.

There are three architectural difficulties:

• Parameter cost: a dense unit has a separate weight for every input pixel. This consumes memory and can make fitting spurious patterns easier when data are limited.
• Spatial structure is implicit: flattening preserves pixel values and can be reversed if the original shape is known. However, a generic dense layer does not encode which positions are neighbors in two dimensions.
• No built-in sharing across locations: a pattern learned at one location does not automatically use the same detector elsewhere. A dense network can learn that behavior, but the architecture does not enforce it.

CNNs use small receptive fields and apply the same weights at many positions. This inductive bias makes local visual regularities easier to learn. It is useful when the task benefits from locality and shared patterns; it is not proof that CNNs are best for every image problem.

Equivariance is not invariance

If shifting an input shifts its feature map correspondingly, the operation is translation equivariant. If shifting the input leaves the final output unchanged, the function is translation invariant.

A stride-one convolution is translation equivariant in the idealized setting away from boundary effects. Shared filters can detect the same pattern at different locations. That is different from guaranteeing identical class predictions after every shift. Finite padding, strided sampling, pooling, and position-dependent dense heads can break exact shift behavior. Pooling and suitable training can encourage tolerance to small translations, but a complete CNN is not automatically translation invariant.

2. Images and feature maps are tensors

We use channels-last layout throughout:

Tensor	Shape	Meaning
One grayscale image	(H,W,1)	Height, width, one intensity channel
One RGB image	(H,W,3)	Red, green, and blue channels
Batch of images	(B,H,W,C)	B examples with C channels
Convolution output	(B,H_out,W_out,F)	F learned output channels

The batch dimension is not part of keras.Input(shape=...). A batch of 32 Fashion-MNIST images has shape (32,28,28,1), while the input declaration uses (28,28,1).

Input channels and output filters are different quantities. A conventional convolutional filter spans every input channel. A 3×3 filter acting on RGB therefore has 3×3×3 weights. Applying F such filters produces F output channels. In later layers, those input channels are learned feature maps rather than literal colors.

3. What a convolutional filter calculates

A kernel is a small array of trainable weights. At each location, multiply a local patch by those weights, sum the products across spatial positions and input channels, add a bias, then apply the chosen activation.

For a channels-last input X, stride one, no padding, and one output channel f:

Z[i,j,f] = b[f] + ΣᵤΣᵥΣ꜀ K[u,v,c,f]X[i+u,j+v,c],

followed by A[i,j,f]=ReLU(Z[i,j,f])=max(0,Z[i,j,f]) in a ReLU layer. The sums cover kernel rows u, kernel columns v, and input channels c.

Deep-learning libraries conventionally call this convolution, although the common operation is technically cross-correlation: the kernel is not flipped spatially. TensorFlow’s convolution definition makes the indexing explicit. Because the kernel is learned, this naming convention does not prevent learning useful features.

Worked example: one filter, four positions

Use a single-channel image X and a 2×2 kernel K, with zero bias:

X = [[1, 2, 0],       K = [[ 1, 0],
     [0, 1, 3],            [ 0,-1]]
     [2, 2, 1]]

With stride one and no padding, the top-left response is:

1×1 + 2×0 + 0×0 + 1×(−1) = 0.

Moving right gives 2×1+0×0+1×0+3×(−1)=−1. The bottom-left and bottom-right responses are −2 and 0 respectively. Therefore:

Before ReLU:           After ReLU:
[[ 0,-1],              [[0,0],
 [-2, 0]]               [0,0]]

The filter compares two diagonal pixel positions. It need not activate on every patch; an all-zero ReLU output is possible for a particular input and filter. Other kernels can respond to edges or textures, but learned filters need not have a simple human-readable interpretation.

For another patch [[3,0],[0,1]], the same kernel produces 3−1=2, which ReLU retains. The key is that every patch uses the same kernel and bias, rather than learning an independent detector at each position.

How filters learn

Initialize filter values, run a forward pass, compute the classification loss, backpropagate gradients, and update the values using an optimizer. Weight sharing means the gradient for a kernel weight accumulates contributions from all positions where that weight was used, and from the batch examples.

ReLU passes a derivative of one for positive preactivations and zero for negative ones; a convention is used at zero. Max pooling routes gradients through selected maximum locations, while average pooling distributes a window’s gradient across its entries. These local derivative rules allow ordinary backpropagation through the complete CNN.

4. Kernel size, stride, padding, and parameter counts

Kernel size determines the local window. Stride determines how far the window moves between outputs. Padding adds values around the input boundary, often zeros, to control output dimensions and boundary coverage.

For one spatial dimension with input size H, kernel size k, stride s, dilation one, and symmetric padding p on each side:

H_out = floor((H+2p−k)/s)+1.

Use the same equation for width. With padding="valid", p=0. With padding="same", Keras chooses padding so the output size is ceil(H/s); at stride one it equals H. Padding can be asymmetric, particularly with even kernels, so do not force the symmetric-padding formula onto every same case.

For an ordinary ungrouped convolution with bias, kernel k_h×k_w, C input channels, and F filters:

Parameter count = (k_h k_w C + 1)F.

It does not depend on input height and width, although the operation’s computational cost does. The formula changes for grouped or depthwise convolutions, which are outside this example.

Worked comparison. A 3×3 RGB convolution with 32 output filters has (3×3×3+1)×32=896 parameters. A 2×2 grayscale convolution with 32 filters has (2×2×1+1)×32=160. Sharing a small number of weights across the image produces many activations from few parameters.

5. Pooling: summarize each feature map locally

Pooling downsamples spatial dimensions. In ordinary 2D pooling it operates separately on each channel, without mixing channels or learning kernel weights.

For a window of values:

[[1,3],
 [2,0]]

Max pooling gives 3; average pooling gives (1+3+2+0)/4=1.5. Max pooling retains a strong local response, while averaging measures the local mean response.

The “OR” analogy for max pooling is useful only informally: if one nonnegative activation is strong, the pooled response stays strong. It is not a literal Boolean OR unless the values and their interpretation make it one.

For a complete 4×4 feature map and nonoverlapping 2×2 windows:

Input:                   Max pooled:     Average pooled:
[[1,3,2,0],              [[3,4],          [[1.5,2.0],
 [2,0,4,2],               [6,3]]           [3.5,1.5]]
 [5,1,0,2],
 [2,6,1,3]]

If the input had 32 channels, the same process would independently produce 32 smaller maps. A 2×2 pool with stride two typically halves each even spatial dimension, reducing activation storage by about a factor of four while preserving channel count.

Pooling discards information. It does not guarantee a useful abstraction, and it can remove precise position or fine detail needed for tasks such as localization. Strided convolutions offer another learned method of downsampling. Pooling is common, not compulsory.

6. Convolutional blocks, receptive fields, and classifier heads

A common block has one or two convolutional layers followed by downsampling. Spatial dimensions decrease, while later convolutions may increase the number of feature channels. Increasing channels is a design choice, not a mathematical requirement; the example below keeps 32 filters in both blocks to match the proposed architecture.

The receptive field of an activation is the input region that can influence it. Stacking local operations grows this region. In one dimension, if r is receptive-field size and j is the spacing between neighboring receptive-field centers in original input coordinates, a layer with kernel k and stride s updates:

r_new=r+(k−1)j, and j_new=js, starting from r=1 and j=1.

For our sequence of 2×2 convolution, 2×2 pooling, 2×2 convolution, and 2×2 pooling, the receptive-field sizes are 2, 3, 5, 7, and the final spacing is 4 pixels. This is the theoretical receptive field; actual influence also depends on learned weights, activations, and pooling selections.

At the end, a classifier head combines features into class predictions. Flatten() preserves feature values but turns their tensor into a vector; a following dense layer can use location-specific weights. Global average pooling instead averages each channel over all spatial positions and yields one number per channel. It can greatly reduce head parameters, although discarding spatial arrangements is not always desirable.

7. Build the proposed CNN in Keras

Use Keras 3 with a compatible TensorFlow installation in a separate Python environment. See the Keras setup guide and TensorFlow installation guide. The website itself does not need TensorFlow dependencies.

The next blocks form one script, executed in order. Training metrics are not supplied as measured results here; run the experiment to obtain them on your environment.

Load and split Fashion-MNIST

Fashion-MNIST provides 60,000 training images and 10,000 test images, each a 28×28 grayscale image with one of ten clothing labels. The dataset authors’ repository documents the categories and benchmark. Keep the supplied test split untouched during development.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"  # Before importing keras.

import numpy as np
import keras

keras.utils.set_random_seed(42)
(images, labels), (test_images, test_labels) = (
    keras.datasets.fashion_mnist.load_data()
)
class_names = [
    "T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
    "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot",
]

# Add the channel axis, keeping original 0–255 pixel values.
images = images[..., np.newaxis]
test_images = test_images[..., np.newaxis]

# Select 600 validation examples per class using only the training split.
rng = np.random.default_rng(42)
validation_indices = np.concatenate([
    rng.permutation(np.flatnonzero(labels == c))[:600]
    for c in range(10)
])
training_mask = np.ones(len(labels), dtype=bool)
training_mask[validation_indices] = False
x_train, y_train = images[training_mask], labels[training_mask]
x_val, y_val = images[validation_indices], labels[validation_indices]
x_test, y_test = test_images, test_labels

assert x_train.shape == (54000, 28, 28, 1)
assert x_val.shape == (6000, 28, 28, 1)
assert x_test.shape == (10000, 28, 28, 1)

Labels remain integer class IDs 0–9. Because normalization happens inside the model, do not also divide these arrays by 255. The fixed split supports comparisons between architectures; changing the validation split between runs would complicate that comparison.

Define the model

inputs = keras.Input(shape=(28, 28, 1), name="image")
x = keras.layers.Rescaling(1.0 / 255, name="rescale")(inputs)

x = keras.layers.Conv2D(
    32, (2, 2), padding="valid", strides=1,
    activation="relu", name="conv1",
)(x)
x = keras.layers.MaxPooling2D(
    pool_size=(2, 2), strides=2, padding="valid", name="pool1",
)(x)

x = keras.layers.Conv2D(
    32, (2, 2), padding="valid", strides=1,
    activation="relu", name="conv2",
)(x)
x = keras.layers.MaxPooling2D(
    pool_size=(2, 2), strides=2, padding="valid", name="pool2",
)(x)

x = keras.layers.Flatten(name="flatten")(x)
x = keras.layers.Dense(256, activation="relu", name="dense")(x)
outputs = keras.layers.Dense(10, activation="softmax", name="class")(x)
model = keras.Model(inputs, outputs, name="fashion_cnn")
model.summary()
assert model.count_params() == 302026

MaxPool2D is an alias of MaxPooling2D. Here kernel sizes, strides, and padding are explicit so the dimensions can be checked without relying on remembered defaults. Rescaling applies the same transformation at training and inference; it has no learned parameters.

Track every layer

The batch dimension is omitted below:

Layer	Output shape	Trainable parameters
Input / rescaling	28×28×1	0
Conv1: 2×2, 32 filters	27×27×32	(4×1+1)×32 = 160
Pool1: 2×2, stride 2	13×13×32	0
Conv2: 2×2, 32 filters	12×12×32	(4×32+1)×32 = 4,128
Pool2: 2×2, stride 2	6×6×32	0
Flatten	1,152	0
Dense: 256 units	256	1,152×256+256 = 295,168
Softmax output	10	256×10+10 = 2,570
Total		302,026

The first pool maps 27 to 13 because the final incomplete window is excluded under valid padding. The 256-unit dense layer contains most of the parameters. Although the convolutional layers are economical, this complete CNN has more parameters than a 784→256→10 dense model, whose total is 203,530. Parameter sharing reduces convolutional-layer costs; it does not guarantee that every CNN has fewer total parameters than every dense network.

Replacing Flatten() with GlobalAveragePooling2D() would give 32 inputs to the same 256-unit dense layer. That alternative would total 160+4,128+8,448+2,570=15,306 parameters. Its capacity and spatial behavior differ, so evaluate it rather than assuming it will perform equally well.

8. Compile, train, and evaluate

With mutually exclusive class labels, softmax maps ten output scores into ten nonnegative probabilities summing to one. For the correct class y, the per-example loss is −log(p_y). Integer labels pair with sparse categorical cross-entropy; one-hot labels would instead pair with ordinary categorical cross-entropy.

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[keras.metrics.SparseCategoricalAccuracy(name="accuracy")],
)
history = model.fit(
    x_train, y_train,
    validation_data=(x_val, y_val),
    batch_size=128, epochs=30, shuffle=True,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", mode="min", patience=4,
        restore_best_weights=True,
    )],
    verbose=2,
)

# Run final test evaluation after selecting the model on validation data.
test_results = model.evaluate(x_test, y_test, return_dict=True, verbose=0)
probabilities = model.predict(x_test, verbose=0)
predictions = probabilities.argmax(axis=1)
print("Held-out test results:", test_results)
print("First prediction:", class_names[int(predictions[0])])
print("First true label:", class_names[int(y_test[0])])

confusion = np.zeros((10, 10), dtype=np.int64)
np.add.at(confusion, (y_test, predictions), 1)
print("Confusion matrix: true classes in rows, predictions in columns")
print(confusion)

Because the output already uses softmax, the loss takes probabilities with its default from_logits=False. An equally valid alternative is a linear ten-unit output and SparseCategoricalCrossentropy(from_logits=True); apply softmax only when probabilities are needed. Do not mix these two conventions.

For 54,000 training examples and batch size 128, retaining the last incomplete batch gives 422 updates per epoch. The final batch has 112 examples. Early stopping limits training based on validation evidence and restores its best weights; the test set is not used for that selection.

9. Establish a fair dense baseline

Train the following model using the same data split and scaling convention. Give it a fresh optimizer and its own early-stopping callback. Compare validation performance while selecting models, and reserve final test comparisons until choices are fixed.

baseline = keras.Sequential([
    keras.Input(shape=(28, 28, 1)),
    keras.layers.Rescaling(1.0 / 255),
    keras.layers.Flatten(),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dense(10, activation="softmax"),
], name="dense_baseline")
baseline.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[keras.metrics.SparseCategoricalAccuracy(name="accuracy")],
)
assert baseline.count_params() == 203530
baseline_history = baseline.fit(
    x_train, y_train, validation_data=(x_val, y_val),
    batch_size=128, epochs=30, shuffle=True,
    callbacks=[keras.callbacks.EarlyStopping(
        monitor="val_loss", mode="min", patience=4,
        restore_best_weights=True,
    )], verbose=2,
)

The supplied course outline cites roughly 85–88% accuracy for a dense model and over 90% for a basic CNN. Treat these as unverified course-reported ranges, not measured outputs or guarantees for this script. Accuracy depends on preprocessing, split, training duration, initialization, and architecture. The MNIST digit benchmark is also different from Fashion-MNIST; a digit-classifier result cannot be substituted for clothing accuracy.

Record parameter counts, training time, validation loss, and test accuracy alongside the exact experiment settings. Inspect per-class confusion: an aggregate score can hide repeated confusion between visually similar items. Multiple seeds help reveal training variability. A higher score alone does not establish robustness to a new camera, background, or clothing population.

10. Improve the experiment thoughtfully

Inspect training and validation curves. Falling training loss with increasing validation loss suggests overfitting or a mismatch between the two datasets. Possible responses include early stopping, a smaller classifier head, weight penalties, or carefully validated dropout.

Image augmentation generates label-preserving variations during training. Small translations may be reasonable for centered objects, but every transformation must preserve the task’s labels; rotations and flips are not automatically appropriate in every vision problem. If augmentation layers are used, keep validation and test transformations deterministic. Avoid augmenting before the split in a way that puts near-duplicates on both sides.

If a convolutional layer produces invalid dimensions, recalculate its output size. If performance stays near chance, inspect label alignment, normalization, loss configuration, and the input channel axis before enlarging the model. Dividing by 255 twice can make inputs unexpectedly small. A missing channel axis or incompatible label encoding can cause shape errors.

For larger real-world datasets, pretrained representations are often useful, but that is a separate experiment from learning every kernel from random initialization. Save the model together with the class-name ordering and input conventions. A saved rescaling layer does not resize an arbitrary photograph or convert its color channels automatically.

11. Exercises and answers

1. Local computation: For patch [[4,1],[2,3]], kernel [[1,0],[0,-1]], and bias 0.5, what is the ReLU output? max(0,4−3+0.5)=1.5.
2. Valid convolution: What output shape results from a 32×32×3 input, 3×3 kernels, stride one, no padding, and 16 filters? 30×30×16.
3. Parameter count: How many parameters does that convolution have, including biases? (3×3×3+1)×16=448.
4. Pooling: What happens to a 30×30×16 tensor under valid 2×2 max pooling with stride two? 15×15×16, with no trainable parameters.
5. Padding: A stride-one 3×3 convolution uses same padding. What are the output height and width for a 28×28 input? 28×28; output channels depend on the filter count.
6. Pooling values: For [[1,7],[2,2]], what are max and average pooling? 7 and 3.
7. Equivariance: If moving an object moves its activation peak, is that invariance? No. It illustrates equivariance. Invariance would leave the output unchanged under the considered shift.
8. Classification loss: If the correct class has probability 0.8, what is its cross-entropy contribution? −log(0.8)≈0.2231.
9. Head design: Why might global average pooling reduce overfitting? It removes many location-specific dense-head parameters, reducing capacity. It may also discard spatial detail that the task needs, so validation is necessary.
10. Evaluation: Why not tune architecture until test accuracy exceeds 90%? Repeatedly choosing models against the test set makes it part of model selection and biases the reported estimate. Use validation data for choices and retain an independent final evaluation.

Further reading and sources

• Goodfellow, Bengio, and Courville, Deep Learning, Chapter 9: Convolutional Networks: sparse interactions, parameter sharing, and equivariance.
• Keras Conv2D: filter, stride, padding, and layout options.
• Keras MaxPooling2D: pooling windows and output dimensions.
• Keras Rescaling: preprocessing inside the saved model.
• Keras GlobalAveragePooling2D: spatial averaging by channel.
• TensorFlow: Classify images of clothing: Fashion-MNIST and a dense baseline workflow.
• Xiao, Rasul, and Vollgraf, Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms: the dataset paper.

The small matrices, parameter comparisons, and exercises are constructed teaching examples. The listed training script is an experiment to run, not a claim that a particular test accuracy has already been achieved.

Feynman study lab

Explain it simply

A convolution reuses a small pattern detector across an image instead of learning separate weights at every location.

Concrete example. A 3 × 3 filter over one input channel has 9 weights, plus one bias if used.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Why does moving the filter over a larger image not multiply its parameter count?

Check your explanation after you try

The same weights are shared across locations. More input channels or output filters increase parameters; more locations mainly increase computation.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.