Fine-Tuning

Fine-tuning means deliberately unfreezing part of a pretrained network and continuing training with a very small learning rate. It is the step that turns a generic ImageNet feature extractor into a specialist for your data. The failure mode is always the same: too high a learning rate, too many unfrozen layers, or unfrozen batch normalization, and the pretrained features are destroyed.

The idea

Fine-tuning rests on one asymmetry: the later layers of a pretrained network are task-specific, while the early ones are generic. So you unfreeze from the top down and use learning rates that decrease with depth. Common patterns:

• Unfreeze the last block or two. Cheap, safe, and usually enough.
• Discriminative learning rates. Early layers at 1e-5, the new head at 1e-3.
• Keep batch normalization frozen. ImageNet statistics generalize better than statistics estimated from a few hundred of your batches.

The other half is regularization, because a pretrained model with tens of millions of parameters will overfit a small dataset immediately. Dropout on the head, early stopping on validation loss, and a learning rate that decays on a plateau are the standard trio.

Worked example

Unfreezing the top of InceptionV3 after training a fresh head:

Two orders of magnitude separate the two rates. At 1e-3 the fine-tuning phase would overwrite the pretrained filters within a few hundred steps, because the gradients are large while the new head is still predicting poorly.

In code

Train the head with the base frozen, then unfreeze its top layers:

import tensorflow as tf

base = tf.keras.applications.InceptionV3(
    weights="imagenet", include_top=False, input_shape=(299, 299, 3))
base.trainable = False

inputs = tf.keras.Input(shape=(299, 299, 3))
x = base(inputs, training=False)          # keeps BN in inference mode
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.3)(x)
outputs = tf.keras.layers.Dense(5, activation="softmax")(x)
model = tf.keras.Model(inputs, outputs)

model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
              loss="categorical_crossentropy", metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=10,
          callbacks=[tf.keras.callbacks.EarlyStopping(patience=3,
                                                      restore_best_weights=True)])

base.trainable = True
for layer in base.layers[:-30]:
    layer.trainable = False
for layer in base.layers:
    if isinstance(layer, tf.keras.layers.BatchNormalization):
        layer.trainable = False

model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
              loss="categorical_crossentropy", metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=5,
          callbacks=[tf.keras.callbacks.ReduceLROnPlateau(patience=2)])

Three practical notes. base(inputs, training=False) plus frozen batch normalization layers is belt and braces — both keep the running statistics from being overwritten. Recompile after changing trainable; do not assume the optimizer notices. And restore_best_weights=True matters, because otherwise you keep the last epoch, which is usually the most overfit one.

Check yourself

1. Why must the fine-tuning learning rate be much smaller than the head-training rate?
2. What goes wrong when unfrozen batch normalization layers are updated with small batches?
3. Why is EarlyStopping(restore_best_weights=True) more than a convenience on a small dataset?

Key takeaways

• Fine-tuning means unfreeze the top, drop the learning rate by about 100×, and watch validation loss.
• Keep batch normalization frozen; small batches give unreliable statistics.
• Recompile after changing trainable, and regularize hard because the model is large and your data is not.