Special Tensors (eye, rand, arange, linspace)

PyTorch ships constructors for the arrays that show up in every loop: identity matrices, random weights, integer ranges, and evenly spaced grids. Knowing which one returns which shape and dtype removes a lot of guesswork.

The idea

• torch.eye(n, m=None) — identity matrix. torch.eye(3) → (3, 3) with ones on the diagonal. Useful for residual checks and one-hot construction.
• torch.rand(*size) — uniform in [0, 1), dtype float32. torch.randn(*size) — standard normal. torch.randint(low, high, size) — uniform integers, dtype int64, exclusive of high.
• torch.arange(start, end, step) — like Python range, exclusive of end. torch.arange(0, 10, 2) → tensor([0, 2, 4, 6, 8]), dtype int64 unless you pass floats.
• torch.linspace(start, end, steps) — inclusive of both endpoints, always exactly steps values.

The arange versus linspace split causes classic off-by-one bugs. arange(0, 1, 0.25) yields 4 values and cannot promise it lands on 1.0 exactly, because floating-point addition drifts. linspace(0, 1, 5) yields 5 values and always includes both ends. linspace takes a count, not a gap.

Call torch.manual_seed(0) before random constructors to make a run reproducible. Every constructor also accepts dtype and device.

Worked example

Positional encodings for T = 8 tokens with model width D = 4 start from a geometric frequency range:

• p = torch.arange(8) → (8,)
• f = torch.arange(0, 4, 2) → tensor([0, 2]), shape (2,)
• p.unsqueeze(1) / 10000 ** (f / 4) → (8, 2), broadcasting positions against frequencies
• repeating each column twice and applying sin/cos gives the (8, 4) encoding

Two more fixtures you will write constantly:

• torch.rand(2, 3) < 0.5 → a boolean (2, 3) mask
• torch.eye(4) → (4, 4), and x @ torch.eye(4) returns x unchanged for any (B, 4)

In code

import torch

torch.manual_seed(0)
print(torch.eye(3).shape)                 # torch.Size([3, 3])
print(torch.rand(2, 3).dtype)             # torch.float32
print(torch.randint(0, 10, (5,)).dtype)   # torch.int64
print(torch.arange(0, 10, 2))             # tensor([0, 2, 4, 6, 8])
print(torch.linspace(0, 1, 5))
# tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])

p = torch.arange(8)                        # (8,)
f = torch.arange(0, 4, 2).float()          # (2,)
angle = p.unsqueeze(1) / 10000 ** (f / 4)  # (8, 2)
print(angle.shape)                         # torch.Size([8, 2])

Check yourself

1. What is the difference between arange(0, 1, 0.2) and linspace(0, 1, 5)?
2. What shape and dtype does torch.randint(0, 10, (3, 4)) have?
3. Why does p.unsqueeze(1) / f produce (8, 2) instead of raising an error?

Key takeaways

• arange excludes its endpoint and takes a step; linspace includes both ends and takes a count.
• Random constructors return float32 by default; randint and arange with integers return int64.
• Seed before randomness, and check dtype before feeding a model.