Style Transfer

Style transfer separates the content of a photograph from the style of a painting and recombines them into a new image. It works because a pretrained CNN represents content in its activations and style in the correlations between channels. Only the output image is updated; the network stays frozen.

The idea

Two losses, both computed through a frozen feature extractor, usually VGG19:

• Content loss — mean squared error between the feature maps of a content image and the generated image at a deep layer. Deep layers encode what is in the picture, not how it is rendered.
• Style loss — mean squared error between Gram matrices of style and generated features at several layers. The Gram matrix G = F Fᵀ, with flattened features F of shape (positions, channels), discards spatial arrangement and keeps channel correlations — the texture statistics.

Total loss combines them:

Typical weights put style far higher, for example content_weight=1e3 and style_weight=1e-2, though the useful range depends on the layer set. Optimize the pixels with Adam or L-BFGS.

Worked example

VGG19 feature maps for a 400×400×3 input:

• block5_conv2 — (1, 25, 25, 512); content layer, weight 1.0.
• Style layers — block1_conv1 through block5_conv1, each weighted 1 / channels².
• Gram of block4_conv1 is (512, 512), from a flattened (2500, 512).
• Initialize the generated image from the content image; it converges faster and more stably than noise.

After about 1,000 optimizer steps the texture is usually recognizable. Too many steps over-fits the style statistics and muddies the content.

In code

import tensorflow as tf

vgg = tf.keras.applications.VGG19(include_top=False, weights="imagenet")
style_layers = ["block1_conv1", "block2_conv1", "block3_conv1",
                "block4_conv1", "block5_conv1"]
extractor = tf.keras.Model(
    vgg.inputs,
    [vgg.get_layer(n).output for n in style_layers + ["block5_conv2"]],
)

def gram(f):
    f = tf.reshape(f, [-1, f.shape[-1]])
    return tf.matmul(f, f, transpose_a=True) / tf.cast(tf.shape(f)[0], tf.float32)

gen = tf.Variable(content_image)
opt = tf.keras.optimizers.Adam(5.0)
for step in range(1000):
    with tf.GradientTape() as tape:
        feats = extractor(gen)
        content_loss = tf.reduce_mean((feats[-1] - content_feats[-1]) ** 2)
        style_loss = sum(tf.reduce_mean((gram(a) - gram(b)) ** 2)
                         for a, b in zip(feats[:-1], style_feats[:-1]))
        loss = 1e3 * content_loss + 1e-2 * style_loss
    opt.apply_gradients([(tape.gradient(loss, gen), gen)])

Check yourself

1. Why does the Gram matrix drop spatial information, and why is that desirable for style?
2. Why is the generated image initialized from the content image rather than noise?
3. What would happen if you used a shallow layer for content loss instead of block5_conv2?

Key takeaways

• Style transfer optimizes an image, not the network, through frozen VGG features.
• Content is captured by deep activations; style is captured by Gram matrices at many layers.
• Gram normalization by channel count keeps layers of different widths comparable.