Deep Learning for Computer Vision: Transfer Learning, Fine-Tuning, and Hugging Face

Training every visual feature from random initialization can be wasteful when you have only a few hundred labeled images. Transfer learning starts from representations learned on another task, then adapts them to your data. This lecture develops a small-data image-classification workflow, from augmentation and frozen features to careful fine-tuning and Hugging Face inference.

Learning goals

After this lesson you should be able to distinguish training from scratch, feature extraction, and fine-tuning; choose label-preserving augmentations; adapt a pretrained ResNet; handle preprocessing and BatchNorm correctly; and understand what a Hugging Face pipeline does and does not provide.

1. Why small datasets are difficult

A large CNN can memorize a small training set. Training accuracy may rise while validation performance stagnates or deteriorates. This can happen quickly, but “overfits after three epochs” is not a universal law: model capacity, initialization, regularization, image diversity, and the split all matter.

With web-sourced product images, shortcuts are especially easy to learn. Handbags might have white backgrounds while shoes have colored backgrounds. Multiple photographs of the same product might appear in different splits. A model can then appear accurate without learning features that generalize to new products.

Before changing the architecture:

• Define the target classes and ambiguous cases, including photographs containing both objects.
• Remove exact and near-duplicate images, and inspect labels.
• Keep images of the same product or photo session in one split.
• Reserve validation and test groups representative of the intended use.
• Check whether background, retailer, or watermark predicts the label.

For 200 independent images, a 60/20/20 split gives approximately 120 training, 40 validation, and 40 test examples. Group constraints may change those counts. If 36 of 40 test images are correct, accuracy is 90%, but one changed decision moves it by 2.5 percentage points. Small evaluation sets produce substantial uncertainty.

2. Data augmentation: useful, but not a free lunch

Augmentation randomly transforms training images while preserving the intended label. It encourages the model to behave consistently under plausible variations rather than memorize exact pixels.

If x is an image, y its label, T a sampled transformation, and fθ the model, an augmented objective is conceptually:

J(θ) = (1/n)Σᵢ E_T[ℓ(fθ(T(xᵢ)),yᵢ)].

Training samples transformations to approximate this expectation. The critical assumption is that T preserves the label and represents relevant variation.

Transformation	Possible benefit	Potential failure
Small rotation	Tolerance to camera tilt	Changes meaning for orientation-sensitive tasks
Modest crop or zoom	Tolerance to framing	Crops out the object or diagnostic feature
Horizontal flip	Tolerance to viewing direction	Reverses text or meaningful left/right information
Mild color adjustment	Tolerance to lighting	Removes color information needed for the task

Augmentation creates additional views, not additional independent products. A thousand transformations of one handbag do not provide the diversity of a thousand handbags. It cannot repair wrong labels, missing populations, or leakage.

Apply random augmentation after the split and only to training examples. Validation and test preprocessing should be deterministic for this experiment. Test-time augmentation is a separate design choice that must be fixed before evaluating the test set.

A Keras detail: degrees versus fractions of a turn

RandomRotation uses fractions of a full turn. A symmetric range of ±10 degrees therefore uses factor=10/360≈0.02778, not 10 and not 0.1. A factor of 0.1 permits approximately ±36 degrees. In the example we add small rotations and horizontal flips, assuming those transformations preserve the handbag/shoe labels.

3. What transfers from a pretrained model?

A source task supplies representations that may help a target task. For example, ImageNet-1k pretraining uses more than a million training images across 1,000 categories. This is a particular benchmark subset, not the entirety of the broader ImageNet collection.

CNN representations are often described as a hierarchy: early layers respond to local color contrasts or oriented patterns, middle layers combine textures and shapes, and later layers encode more task-specific combinations. That is an intuition rather than a fixed dictionary of meanings. A particular channel need not be “the torso detector,” and feature usefulness depends on both model and input distribution.

ResNet uses residual connections, with a block conceptually calculating h_next=h+F(h) when dimensions match. These shortcut paths help optimize deep networks. The pretrained feature extractor is useful because of its learned weights, not merely because the architecture is called ResNet.

Transfer can fail through negative transfer: source features may be unsuitable for the target. Natural-image pretraining may help product photographs but need more adaptation for very different imaging modalities. Verify improvements against a simple baseline.

4. Three training strategies

Write a classifier as p=gφ(fθ(x)), with backbone fθ and task head gφ.

Strategy	Backbone initialization	Parameters updated
Train from scratch	Random	Backbone and head
Frozen feature extraction	Pretrained	New head only
Fine-tuning	Pretrained	Head plus selected backbone layers, or the full backbone

For frozen extraction, optimize φ with θ fixed. For fine-tuning, allow selected components of θ to change as well. Fine-tuning a pretrained representation requires starting from pretrained weights; starting the same architecture randomly is training from scratch instead. The new target head normally starts randomly because its output labels differ from the source labels.

Fine-tuning does not require unfreezing every layer. With little data, updating only a late block may reduce the risk of destroying useful features. Full-network adaptation is possible when justified by validation evidence and resources.

“Headless” ResNet and feature shapes

In Keras, ResNet50(weights="imagenet", include_top=False) loads pretrained weights while omitting the original classification top. For a 224×224 RGB input, its standard final convolutional representation has shape 7×7×2048. Global average pooling reduces this to 2048 values per image.

A new single sigmoid output attached to that pooled vector needs 2048+1=2,049 parameters. Adding a 64-unit ReLU hidden layer first gives:

2048×64+64=131,136, then 64×1+1=65, totaling 131,201 head parameters.

The backbone still consumes memory and compute when frozen; freezing stops its parameter updates, not its forward computation. A larger head is not automatically better for 120 training images.

5. Cached features versus an online frozen backbone

There are two common ways to train a head on frozen representations:

• Cache features: run the frozen backbone once per image and store its vectors with their labels. Head training is inexpensive, but one fixed cache supplies only one preprocessing or augmentation view per image.
• Keep the backbone in the model: compute features on each training batch. This costs more but allows fresh random augmentation each epoch.

Caching is appropriate only while the backbone and cached preprocessing remain fixed. If you unfreeze the backbone, stale cached features no longer reflect its updated parameters. Also keep feature rows aligned with labels: caching a shuffled dataset and collecting labels in a different pass can silently corrupt the training pairs.

The example below keeps the frozen backbone online, then adapts its top block.

6. Prepare a small image dataset

Use a separate Python environment with compatible Keras 3, TensorFlow, and NumPy. The Hugging Face section uses its own PyTorch-based example. These are lesson dependencies, not additions to the website’s dependencies.

Organize images after making a group-aware split:

product_images/
  train/
    handbag/
    shoe/
  validation/
    handbag/
    shoe/
  test/
    handbag/
    shoe/

Keep each product identity entirely within one split. The code assumes you have already done that; directory loading does not perform deduplication or group-aware splitting. Use images you have permission to use, and document provenance and labeling rules.

import os
os.environ["KERAS_BACKEND"] = "tensorflow"

from pathlib import Path
import numpy as np
import keras
import tensorflow as tf

keras.utils.set_random_seed(42)
root = Path("product_images")
class_names = ["handbag", "shoe"]  # 0 = handbag; 1 = shoe.

def load_split(name, shuffle):
    return keras.utils.image_dataset_from_directory(
        root / name,
        labels="inferred", label_mode="binary",
        class_names=class_names,
        image_size=(224, 224), color_mode="rgb",
        batch_size=16, shuffle=shuffle, seed=42,
    ).prefetch(tf.data.AUTOTUNE)

train_ds = load_split("train", shuffle=True)
val_ds = load_split("validation", shuffle=False)
test_ds = load_split("test", shuffle=False)

Labels have shape (batch,1) and images have shape (batch,224,224,3). Resizing to a square can distort aspect ratios; consistent padding or a justified crop can be an alternative. Whatever you choose, match training and deployment conventions.

Match the checkpoint’s preprocessing

Keras ResNet50’s preprocess_input expects RGB image values on the 0–255 scale, converts their channel ordering to BGR, and subtracts ImageNet channel means. Do not first divide these inputs by 255, and do not substitute preprocessing for ResNetV2 or a different checkpoint.

Hugging Face’s checkpoint processor may use different transformations. Matching the name “ResNet” is not enough to make two software ecosystems’ inputs or weights interchangeable.

7. Stage one: learn a head with the backbone frozen

augmentation = keras.Sequential([
    keras.layers.RandomFlip("horizontal", seed=42),
    keras.layers.RandomRotation(10 / 360, seed=43),
], name="augmentation")

base = keras.applications.ResNet50(
    weights="imagenet", include_top=False,
    input_shape=(224, 224, 3),
)
base.trainable = False

inputs = keras.Input(shape=(224, 224, 3), name="image")
x = augmentation(inputs)
x = keras.applications.resnet.preprocess_input(x)
x = base(x, training=False)
x = keras.layers.GlobalAveragePooling2D(name="pooled_features")(x)
x = keras.layers.Dense(64, activation="relu", name="head_hidden")(x)
outputs = keras.layers.Dense(1, activation="sigmoid", name="shoe_probability")(x)
model = keras.Model(inputs, outputs)

def configure(learning_rate):
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=learning_rate),
        loss=keras.losses.BinaryCrossentropy(),
        metrics=[keras.metrics.BinaryAccuracy(name="accuracy")],
    )

def stopping():
    return keras.callbacks.EarlyStopping(
        monitor="val_loss", mode="min", patience=5,
        restore_best_weights=True,
    )

configure(1e-3)
head_history = model.fit(
    train_ds, validation_data=val_ds,
    epochs=30, callbacks=[stopping()], verbose=2,
)
frozen_val_loss = model.evaluate(val_ds, verbose=0)[0]
model.save_weights("frozen_head.weights.h5")

The maximum of 30 epochs and learning rate 0.001 are starting experiment settings, not requirements. Early stopping selects within the allowed training period using validation loss. The augmentation layers act randomly during training and are inactive during evaluation.

Why train the head first? A randomly initialized head can initially send unhelpful gradients into a pretrained backbone. Letting the head acquire a useful mapping before unfreezing is a practical way to reduce that disruption.

8. Stage two: fine-tune conservatively

Unfreeze the late conv5_ block while keeping other backbone layers frozen. Use a new optimizer with a lower learning rate, then recompile so the training configuration reflects the changed trainable parameters.

base.trainable = True
for layer in base.layers:
    layer.trainable = (
        layer.name.startswith("conv5_")
        and not isinstance(layer, keras.layers.BatchNormalization)
    )
assert any(layer.trainable_weights for layer in base.layers)

configure(1e-5)  # 100 times smaller than the head-training rate.
fine_history = model.fit(
    train_ds, validation_data=val_ds,
    epochs=20, callbacks=[stopping()], verbose=2,
)
fine_val_loss = model.evaluate(val_ds, verbose=0)[0]

# Keep fine-tuning only if it improves the chosen validation criterion.
if fine_val_loss >= frozen_val_loss:
    model.load_weights("frozen_head.weights.h5")
    selected_stage = "frozen backbone"
else:
    selected_stage = "fine-tuned top block"
print("Selected stage:", selected_stage)

The rates differ by a factor of 100: a plain SGD gradient of 0.2 would produce steps of 0.0002 and 0.000002 at those rates. This illustrates scale only; Adam’s state and adaptive normalization determine its actual updates. Validate the rate and number of unfrozen blocks rather than treating these values as optimal.

Trainable weights and training behavior are separate

layer.trainable controls which parameters are updated. The training argument controls behavior such as BatchNorm statistics and dropout. Calling base(x, training=False) does not block gradients through trainable convolutions.

BatchNorm uses moving statistics during inference. Small target-data batches can destabilize those statistics if they are updated carelessly. This workflow both freezes backbone BatchNorm layers and keeps the base call in inference mode, preserving their stored statistics while allowing selected convolutional weights to change.

The pretrained weights are loaded once in stage one and retained into stage two. Do not rebuild the base with weights=None, which would discard the transferred initialization. Full-backbone fine-tuning is another possible experiment, but it can require more data and smaller updates. Catastrophic forgetting and overfitting remain possible.

9. Evaluate the selected model once

Choose architecture, augmentation, stopping rules, and threshold using validation data. Then evaluate the selected model on held-out test images.

test_metrics = model.evaluate(test_ds, return_dict=True, verbose=0)

# Both passes preserve order because test_ds has shuffle=False.
y_true = np.concatenate([y.numpy().ravel() for _, y in test_ds]).astype("int32")
p_shoe = model.predict(test_ds, verbose=0).ravel()
y_pred = (p_shoe >= 0.5).astype("int32")
confusion = np.zeros((2, 2), dtype=np.int64)
np.add.at(confusion, (y_true, y_pred), 1)
print("Held-out test metrics:", test_metrics)
print("Rows=true, columns=predicted; classes:", class_names)
print(confusion)
model.save("handbag_shoe.keras")

Here positive means shoe, because the class order was explicit. The saved model includes augmentation and ResNet preprocessing, but callers still need to decode an image, obtain RGB channels, and resize consistently. Save the class order and input contract alongside the model.

Compare against a majority-class rule and a frozen-feature baseline. Report split sizes, per-class errors, and uncertainty alongside accuracy. With tiny datasets, grouped cross-validation can support development, but repeated decisions against the test set compromise its independence. No universal accuracy follows from having 100 examples per class; diverse, well-labeled examples and a representative evaluation matter more than that count alone.

10. Introduction to the Hugging Face ecosystem

The Hugging Face Hub is a repository ecosystem for model weights, configurations, processors, datasets, and model cards. The Transformers Python library supplies model classes and high-level interfaces for many tasks, including vision. Not every Hub repository is a Transformers model, and not every task uses a transformer architecture: ResNet remains a CNN.

Before using a checkpoint, read its model card for the intended task, training data, label space, preprocessing, license, and limitations. For reproducible experiments, record package versions and pin the checkpoint to an immutable repository revision rather than relying indefinitely on the moving default branch. Hub availability alone does not establish suitability for your task.

A local image-classification pipeline

In a separate environment, install compatible transformers, torch, and pillow packages. Place a local image at example.jpg, then run:

from PIL import Image
from transformers import pipeline

classifier = pipeline(
    task="image-classification",
    model="microsoft/resnet-50",
    device=-1,  # CPU.
)
with Image.open("example.jpg") as source:
    image = source.convert("RGB")
results = classifier(image, top_k=5)
for result in results:
    print(result["label"], round(result["score"], 4))

The microsoft/resnet-50 model card describes an ImageNet-1k pretrained ResNet-50 v1.5 checkpoint. The pipeline loads the model and its processor, transforms the image, runs inference, and returns scored labels from the existing classification head. On first use it downloads the needed files; this local example performs inference on your machine rather than invoking a hosted prediction API.

This does not train a handbag-versus-shoe classifier. The output vocabulary remains the checkpoint’s ImageNet classes, not two custom labels. A high score is relative to that output vocabulary and is not a calibrated guarantee that an unfamiliar image belongs to the predicted class. New labels require a suitable adapted head and labeled training, or a different explicitly chosen approach such as a compatible zero-shot model.

Look underneath the pipeline

This alternative loads the same model family explicitly, illustrating the processor/model boundary. Run it separately from the pipeline if you want to avoid holding duplicate model copies in memory.

import torch
from transformers import AutoImageProcessor, AutoModelForImageClassification

model_id = "microsoft/resnet-50"
processor = AutoImageProcessor.from_pretrained(model_id)
hf_model = AutoModelForImageClassification.from_pretrained(model_id)
hf_model.eval()

batch = processor(images=image, return_tensors="pt")
with torch.inference_mode():
    logits = hf_model(**batch).logits
probabilities = logits.softmax(dim=-1)
index = int(probabilities.argmax(dim=-1).item())
print(hf_model.config.id2label[index], float(probabilities[0, index]))

Feed the processor an RGB image rather than an image already transformed with Keras ResNet preprocessing. The checkpoint’s processor defines the required resize and normalization. eval() sets inference behavior, while inference_mode() avoids building the gradient graph; these are related but distinct controls.

Keras Applications and Hugging Face are different routes to pretrained representations. Architecture names, checkpoint recipes, and normalization conventions can differ, so do not assume weights or preprocessing can be swapped without conversion and verification.

11. Exercises with answers

1. Rotation: What RandomRotation factor corresponds to ±15 degrees? 15/360≈0.04167.
2. Independent evidence: Do 50 random crops of each of 100 products give 5,000 independent products? No. They are correlated views of 100 source products.
3. Head size: How many parameters does a single sigmoid neuron need on 2,048 pooled features? 2,049, including its bias.
4. Hidden head: Add a 32-unit dense hidden layer before that output. How many head parameters? 2048×32+32+32+1=65,601.
5. Frozen backbone: Does freezing eliminate backbone computation? No. It prevents weight updates; the forward pass still calculates features unless you cache them.
6. Fine-tuning: Is a randomly initialized ResNet trained end-to-end an example of transferring pretrained features? No. It is training the architecture from scratch.
7. Recompile: What should follow a change in Keras layer trainability? Recompile, using the intended optimizer configuration, before further fitting.
8. BatchNorm: Does training=False prevent gradients from reaching unfrozen convolution weights? No. It controls inference behavior; trainable weights can still receive gradients.
9. Evaluation: Thirty-six of forty test images are correct. What is accuracy, and how much does one error change it? 90%, with 2.5 percentage points per image.
10. Hugging Face: Does pipeline("image-classification", model="microsoft/resnet-50") learn your two custom labels? No. It performs inference with its existing ImageNet label space.

Further reading and sources

• Keras transfer learning and fine-tuning guide: staged training, trainability, recompilation, and BatchNorm behavior.
• Keras ResNet models: pretrained initialization, include_top, pooling, and input preprocessing.
• Keras RandomRotation: rotation factors and training behavior.
• Keras image loading utilities: class ordering, image sizes, and binary labels.
• He et al., Deep Residual Learning for Image Recognition: the original ResNet paper.
• ImageNet ILSVRC 2012: the benchmark classification task and data.
• Hugging Face pipeline documentation: inference interfaces and supported tasks.
• Hugging Face image-classification task guide: adapting image classifiers.

The dataset counts and calculations above are teaching examples. Transfer-learning success, speed, and accuracy must be measured on the target task rather than inferred from a model name or a small number of examples.

Feynman study lab

Explain it simply

Transfer learning reuses features learned elsewhere and adapts them to a new task.

Concrete example. Freezing an image backbone and training a new classifier changes classifier weights while leaving backbone weights fixed.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Why might a pretrained model still perform badly on your images?

Check your explanation after you try

The new domain, labels or preprocessing can differ. Validate on representative held-out data; fine-tuning does not guarantee a cure.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.