Practice
Paper-linked challenges
Each challenge is derived from a Fanout Daily paper. Predict the mechanism, implement it, debug a broken version, or transfer it to a new setting — the judge runs your code against hidden tests and scores by cases passed.
Adam: one bias-corrected update step
Implement `run(m, v, g, t, beta1, beta2, lr, eps)` returning the Adam parameter step (delta) for one gradient g, given running first moment m and second moment v at timestep t. Update m <- beta1*m + (1-beta1)*g; v <- beta2*v + (1-beta2)*g^2; then delta = lr * m_hat / (sqrt(v_hat) + eps) with bias-corrected moments m_hat = m/(1-beta1^t), v_hat = v/(1-beta2^t). Return delta.
Attention: scaled dot-product output
Implement `run(q, k, v)` returning the attention output for a single query. q, k, v are lists of lists (one row per position). Compute scores = Q·K^T / sqrt(d_k) where d_k = len(q), apply softmax along the last axis, then take the weighted sum with V. Return the output vector as a list of floats (rows must sum to 1 in the attention weights).
Switch: auxiliary load-balancing loss
Implement `run(fractions, probabilities, alpha)` returning the auxiliary load-balancing loss for a Switch/GShard-style router: alpha * N * sum(f_i * P_i), where N is the number of experts, f_i is the fraction of tokens routed to expert i, P_i is the fraction of router dispatch probability assigned to expert i, and alpha is the loss coefficient. Return the float value.
Dropout: inverted scaling keeps expectations
Implement `run(x, keep_prob)` returning the inverted-dropout output for input vector x: mask each element with probability p=1-keep_prob (zeros), and scale SURVIVING elements by 1/keep_prob so the expected output equals the input. You may use a fixed deterministic mask for reproducibility: mask[i] = 1 if i % 5 != 0 else 0. The scale factor is 1/keep_prob.
Expert choice: capacity-constrained token routing
Implement `run(affinities, e, n, c)` returning the number of tokens k each expert selects in expert-choice routing, where each of the e experts picks its top-k tokens and k = ceil((n * c) / e). Round k UP to the nearest integer. n is the number of tokens, c the capacity factor (default 2).
Gradient descent: predict convergence on a ravine
Implement `run(lr, x0, y0, steps)` that applies gradient descent on f(x, y) = x^2 + 10y^2 and returns the final loss value after `steps` iterations. Update rule: x <- x - lr * 2x, y <- y - lr * 20y. On this ravine, stability breaks near lr ~ 0.1 (the steep y-axis overcorrects). Return the final loss rounded to 6 decimal places.
KV cache: memory per sequence
Implement `run(layers, kv_heads, head_dim, tokens, batch, bits)` returning the total KV-cache memory in bytes for one model: 2 (K and V) * layers * kv_heads * head_dim * tokens * batch * (bits / 8). Return the integer byte count. Hint: the Fanout kv-cache lab verifies 7B GQA / 16-bit / 16K tokens / 8 seqs = 16.0 GiB.
PPO: clipped surrogate objective
Implement `run(old_prob, new_prob, advantage, epsilon)` returning the PPO clipped surrogate value L^CLIP for a single (state, action): r = new_prob / old_prob, surrogate = r * A, clipped = clip(r, 1-eps, 1+eps) * A, and L = min(surrogate, clipped).
Tail at scale: bounded retry with jitter
Implement `run(attempt, base_ms, cap_ms)` returning the jittered backoff delay (ms) for retry `attempt` (0-indexed): exponential backoff base = base_ms * 2^attempt, capped at cap_ms, then apply FULL jitter: a uniformly random value in [0, capped]. For determinism, use a fixed pseudo-random sample: rand = ((attempt * 7919) % 1000) / 1000 (a value in [0,1)). Return the integer floor of rand * capped. Also return the raw capped backoff so the learner can see the bound.
ZeRO: per-device model-state memory planner
Implement `run(psi, nd, stages)` returning the per-device model-state memory in bytes for mixed-precision Adam, given parameter count Ψ, data-parallel degree Nd, and a list of ZeRO stages (0 = full DP replication, 1 = P_os, 2 = P_os+P_g, 3 = all). Use K=12 optimizer multiplier. Round to the nearest integer byte.