Transfer Learning

Transfer learning takes a network trained on a huge dataset like ImageNet and reuses its features for your much smaller problem. You freeze most of the pretrained weights, attach a new classifier, and train only that head. It is the fastest route to a good model when you have thousands of labeled examples rather than millions.

The idea

Early convolutional layers learn edges, color blobs, and textures; later layers learn parts and whole objects. The early features are generic, so they transfer across very different tasks. The recipe:

1. Load a pretrained base with include_top=False to drop its original classifier.
2. Freeze it: base.trainable = False.
3. Add your own head — usually global average pooling followed by a dense layer with your class count.
4. Train the head while the base is frozen.
5. Optionally unfreeze the top blocks and continue with a much smaller learning rate.

A frozen base runs batch normalization in inference mode, using the ImageNet statistics. That matters: if those statistics were recomputed from your small batches, the pretrained features would degrade.

Worked example

Where the parameters live for InceptionV3 on a five-class problem:

The head is 2048 × 5 + 5 = 10,245 parameters while the frozen base is 21.8 million — the head is a rounding error. The whole point of transfer learning is that you only pay to train the rounding error.

In code

import tensorflow as tf

base = tf.keras.applications.InceptionV3(
    weights="imagenet", include_top=False, input_shape=(299, 299, 3))
base.trainable = False

model = tf.keras.Sequential([
    base,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(5, activation="softmax"),
])
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
              loss="categorical_crossentropy",
              metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=10)

Then fine-tune the top of the base:

base.trainable = True
for layer in base.layers[:-30]:          # keep most of the base frozen
    layer.trainable = False
for layer in base.layers:
    if isinstance(layer, tf.keras.layers.BatchNormalization):
        layer.trainable = False          # keep ImageNet statistics

model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),   # 100x smaller
              loss="categorical_crossentropy", metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=5)

Two details cause most transfer-learning bugs: recompile after changing trainable rather than assuming the optimizer notices, and keep the fine-tuning rate far below the head-training rate.

Check yourself

1. Why are the early layers of a pretrained CNN reusable across very different tasks?
2. What happens to batch normalization statistics if the whole base is unfrozen with a large learning rate?
3. Why does the head have about 10,000 trainable parameters while the base has 21.8 million?

Key takeaways

• Freeze a pretrained base, train a new head, then optionally fine-tune with a small learning rate.
• include_top=False gives you the feature extractor; global pooling turns 8×8×2048 into a 2048-vector.
• Recompile after changing trainable, and keep batch normalization frozen while fine-tuning.