Image Captioning

Image captioning joins vision and language: a CNN reads the image, and a recurrent decoder writes a sentence about it. It is the smallest model that contains both a perception stack and a language model, which makes it a good checkpoint before transformer-era architectures.

The idea

• Encoder — a pretrained CNN such as InceptionV3 with include_top=False turns (299, 299, 3) into a spatial feature map; GlobalAveragePooling2D collapses it to (batch, 2048), the image representation.
• Decoder — a Dense projection followed by an LSTM, fed a caption shifted by one token. The image vector initializes the decoder state.
• Vocabulary — build from captions, add <start> and <end>, cap at about 5,000 tokens, pad to max_length.
• Training objective — next-token categorical cross-entropy, the same as translation, with padding masked out.
• Inference — greedy decoding, or better, beam search over a few beams to avoid a locally optimal first word.

Feature extraction happens once and is cached; only the decoder is trained. That cuts training time dramatically on a small dataset.

Worked example

Flickr8k: 8,000 images, 5 captions each.

• Features: InceptionV3 yields (8000, 2048), saved once.
• Caption "<start> a dog runs through the grass <end>", padded to length 35.
• Model: Dense(256, relu) on the image vector, RepeatVector(35), LSTM(256), Dense(vocab, softmax).
• Teacher forcing, categorical cross-entropy, epochs=20, batch_size=64.
• Beam search with beam_width=3 typically reads better than greedy even when perplexity is identical.

In code

import tensorflow as tf

img_in = tf.keras.Input((2048,))
cap_in = tf.keras.Input((35,))

x = tf.keras.layers.Dense(256, activation="relu")(img_in)
x = tf.keras.layers.RepeatVector(35)(x)
x = tf.keras.layers.Concatenate()(
    [x, tf.keras.layers.Embedding(5000, 256)(cap_in)])
out = tf.keras.layers.Dense(5000, activation="softmax")(
    tf.keras.layers.LSTM(256)(x))

model = tf.keras.Model([img_in, cap_in], out)
model.compile(optimizer="adam", loss="categorical_crossentropy")
model.fit([features, captions[:, :-1]],
          tf.one_hot(captions[:, 1:], 5000),
          epochs=20, batch_size=64)

Check yourself

1. Why is the image vector repeated across every decoder timestep rather than fed once?
2. Why cache CNN features instead of recomputing them each epoch?
3. How does beam search differ from greedy decoding, and what does the beam width control?

Key takeaways

• Captioning is a CNN encoder plus an autoregressive language decoder.
• Caching image features makes the decoder the only thing trained.
• Beam search trades compute for better sentences at the same model quality.