Visual Analysis

A trained network is a stack of numeric feature maps, and nothing about that stack is inherently visible. Visual analysis makes it inspectable: it renders what each filter responds to and how strongly each input region drives a prediction. The payoff is debugging — you can see dead filters, saturated activations, and shortcuts the model learned.

The idea

A convolution layer returns a tensor of shape (batch, height, width, filters). Each slice [:, :, :, c] is the activation map of filter c, and its values say how strongly that learned pattern fired at each spatial location.

Two complementary techniques:

• Activation maximization — freeze the weights and treat the input image as the trainable variable. Use gradient ascent to maximize a chosen activation, then visualize the resulting image. This reveals the "preferred stimulus" of a filter.
• Saliency and occlusion — measure how much each input pixel or patch changes a chosen output logit, either with gradients or by zeroing regions and re-running.

Pure gradient ascent produces adversarial-looking high-frequency noise, so the objective adds regularizers: total variation encourages smoothness, an L2 penalty limits pixel magnitude, and a periodic Gaussian blur stabilizes the result.

Worked example

Take a small MNIST CNN: Conv2D(32, 3, activation="relu") on a (28, 28, 1) input gives a (26, 26, 32) map.

• Feed one digit; filter c = 7 has mean activation 0.42 over its 26×26 cells.
• To see filter 7's preferred pattern, start from a random (1, 28, 28, 1) image and run 100 ascent steps on the pixels with learning rate 1.0, blurring after each step.
• Drop the blur and the same filter converges to pixel-level speckle. The image still maximizes the activation, but it tells you nothing about the pattern.

In code

import tensorflow as tf

model = tf.keras.models.load_model("mnist_cnn.keras")
conv = tf.keras.Model(model.inputs, model.get_layer("conv2d").output)

img = tf.Variable(tf.random.uniform((1, 28, 28, 1)))
opt = tf.keras.optimizers.Adam(1.0)

for step in range(100):
    with tf.GradientTape() as tape:
        acts = conv(img)
        loss = tf.reduce_mean(acts[..., 7])                       # filter 7
        loss -= 0.1 * tf.reduce_mean(tf.image.total_variation(img))
    opt.apply_gradients([(tape.gradient(loss, img), img)])
    img.assign(tf.nn.avg_pool2d(img, 3, 1, "SAME"))               # keep it smooth

tf.keras.utils.save_img("filter7.png", img[0])

Check yourself

1. Why are the weights frozen and the image updated during activation maximization?
2. What artifact appears when you remove total-variation and blur regularization, and why?
3. How would occlusion differ from gradient-based saliency on a model that uses a background shortcut?

Key takeaways

• Feature-map slices are per-filter activations indexed by spatial location.
• Activation maximization learns an input, not a parameter; regularization is what makes it readable.
• Saliency and occlusion answer "where did this prediction come from", not "what does this filter want".