Video Data

Video is a dataset of images with a time axis, and every practical question — memory, augmentation, model shape — follows from that extra dimension. Most video work in TensorFlow is about decoding frames efficiently and sampling them into fixed-size clips. The model is usually the easy part; the tf.data pipeline is where the effort goes.

The idea

A single video becomes a tensor of shape [T, H, W, 3], where T is the number of sampled frames, and a batch is [B, T, H, W, 3]. That extra axis is expensive: at 1920×1080 and 3 bytes per pixel one frame is about 6.2 MB, so 30 frames is roughly 186 MB before any model processing. Sample, resize, and decode together — never materialize all frames.

Three modeling families:

• 2D CNN plus temporal pooling — run the CNN on each frame, then average or max over time. Cheapest, ignores order.
• CNN plus RNN — feed per-frame features into an LSTM or GRU. Captures order, trains slowly.
• 3D convolution — convolve over T × H × W. Most expressive, most expensive; ConvLSTM2D sits in between.

Worked example

A 30 fps clip that is 5 seconds long has 150 frames. Sampling every 5th frame gives 30 frames at an effective 6 fps, a 5× reduction in work. Resizing 1920×1080 to 224×224 cuts pixels per frame by (1920 × 1080) / (224 × 224) ≈ 41×.

That batch is 8 × 30 × 224 × 224 × 3 × 4 ≈ 145 MB, so batch size is the first knob to turn.

In code

TensorFlow has no stable general-purpose video decoder in the core API — tf.io.decode_video is experimental. For real work, decode with ffmpeg or decord in a preprocessing step, then let tf.data handle batching and augmentation:

import tensorflow as tf

def load_clip(path, num_frames=30, size=224, raw_shape=(360, 640, 3)):
    # `path` is a packed uint8 blob: ffmpeg -i clip.mp4 -pix_fmt rgb24 -f rawvideo clip.raw
    raw = tf.io.read_file(path)
    frames = tf.io.decode_raw(raw, tf.uint8)
    frames = tf.reshape(frames, [num_frames, *raw_shape])
    frames = tf.image.resize(frames, [size, size])
    frames = tf.cast(frames, tf.float32) / 255.0
    frames = tf.image.random_flip_left_right(frames)   # one flip for all frames
    return frames

ds = (tf.data.Dataset.from_tensor_slices(paths)
      .map(load_clip, num_parallel_calls=tf.data.AUTOTUNE)
      .batch(8)
      .prefetch(tf.data.AUTOTUNE))

Augment the clip as a whole: tf.image.random_flip_left_right on [T, H, W, 3] draws one choice and applies it to every frame, preserving the motion. A per-frame flip would scramble it.

Check yourself

1. Why is per-frame random flipping a bug for video classification?
2. Compute the memory for a batch of 8 clips of 30 frames at 224×224×3 in float32.
3. When is a 3D convolution worth its cost over a 2D CNN plus temporal pooling?

Key takeaways

• Video adds a time axis: a clip is [T, H, W, 3] and a batch is [B, T, H, W, 3].
• Sample and resize early; raw pixel volume grows fast with resolution and frame rate.
• Decode outside the graph with ffmpeg or decord — TF's own video IO is experimental.