Transposing Tensors

Transposing swaps the axes of a tensor without moving data in memory. It is essentially free, and that cheapness is exactly why a missed transpose survives silently into a wrong matmul.

The idea

Three calls cover every case:

• Tensor.T reverses all dimensions.
• Tensor.transpose(dim0, dim1) swaps exactly two.
• torch.permute(t, dims) reorders any number of axes.

Common shapes:

• (B, T, D) with .transpose(1, 2) → (B, D, T)
• (B, H, T, D) with .transpose(-2, -1) → (B, H, D, T)
• (B, H, T, D) with .permute(0, 2, 1, 3) → (B, T, H, D)

The subtlety is memory layout. transpose and permute return a view with new strides, not a copy, so the result is usually non-contiguous. Most operations handle that fine, but .view() does not — it only accepts contiguous input and raises otherwise. .reshape() copies when it has to, so it always works. When you just need flat elements, call .contiguous() first.

.T on a 1-D tensor is a no-op, and on 3-D or higher it reverses every axis, which is rarely what you want. Prefer explicit transpose or permute.

Worked example

Attention needs a (T, T) score matrix from q and k of shape (T, D):

• k.T → (D, T)
• q @ k.T → (T, T): every token scored against every other token
• q.T @ q → (D, D): a Gram matrix over feature directions

For a = torch.arange(6).reshape(2, 3), a.T has shape (3, 2) and holds [[0, 3], [1, 4], [2, 5]]. Also a.T.is_contiguous() is False, which is why a.T.contiguous() is sometimes required before a view.

In code

import torch

a = torch.arange(6).reshape(2, 3)
print(a.T.shape)                          # torch.Size([3, 2])
print(a.T.is_contiguous())                # False
print(a.T.contiguous().is_contiguous())   # True

q = torch.randn(8, 16)                    # (T, D)
print((q @ q.T).shape)                    # torch.Size([8, 8])   scores
print((q.T @ q).shape)                    # torch.Size([16, 16]) Gram matrix

x = torch.randn(4, 6, 8)                  # (B, T, D)
print(x.transpose(1, 2).shape)            # torch.Size([4, 8, 6])

Check yourself

1. For a 4-D tensor (B, H, T, D), what shape does .T produce, and why is .transpose(-2, -1) usually the right call instead?
2. Why can x.transpose(1, 2).view(4, 48) raise an error, and what fixes it without changing values?
3. For q of shape (2, 4, 8, 16), which call produces (2, 4, 16, 8)?

Key takeaways

• transpose and permute swap axes as cheap views; .T reverses all axes.
• Transposed tensors are non-contiguous, so .view() may reject them.
• Attention transposes only the last two axes so tokens compare within each head.