Machine Translation

Machine translation maps a source sequence to a target sequence whose length and alignment are unknown in advance. The sequence-to-sequence model handles that with an encoder that reads the source and a decoder that emits the translation one token at a time. Attention is what lets the decoder look back at the right source position.

The idea

• Encoder — an Embedding plus an LSTM that consumes the source and returns its final states.
• Decoder — an Embedding plus an LSTM initialized from the encoder states; at each step it predicts the next target token from the previous token and its own hidden state.
• Teacher forcing — during training the decoder input at step t is the true previous token, not its own prediction. This trains faster but creates a train/inference mismatch.
• Attention (Bahdanau or Luong) — at each decoder step, score the encoder's per-token outputs, softmax to weights, and take a weighted sum as extra decoder input.
• Evaluation — BLEU compares n-gram overlap between generated and reference translations.

Word-level tokenization with a small vocabulary, say 4,000 tokens per language, and <start>/<end> markers is enough to see the mechanism work.

Worked example

English to Spanish on short sentence pairs.

• Source padded to T_in = 20, target to T_out = 20.
• Embedding(4000, 128), LSTM(256, return_sequences=True, return_state=True).
• Attention over encoder outputs with the decoder query via tf.keras.layers.Attention().
• Loss is SparseCategoricalCrossentropy(from_logits=True, reduction="none") with a mask that zeroes positions where the target is padding.
• Inference: greedy decode, feeding each predicted token back in, stopping at <end>.

Masking the padding is essential. Without it the loss is dominated by pad tokens and the model learns to emit padding.

In code

import tensorflow as tf

enc_in = tf.keras.Input((20,))
enc_emb = tf.keras.layers.Embedding(4000, 128, mask_zero=True)(enc_in)
enc_out, h, c = tf.keras.layers.LSTM(
    256, return_sequences=True, return_state=True)(enc_emb)

dec_in = tf.keras.Input((19,))
dec_emb = tf.keras.layers.Embedding(4000, 128, mask_zero=True)(dec_in)
dec_out = tf.keras.layers.LSTM(256, return_sequences=True)(
    dec_emb, initial_state=[h, c])
context = tf.keras.layers.Attention()([dec_out, enc_out])
logits = tf.keras.layers.Dense(4000)(
    tf.keras.layers.Concatenate()([dec_out, context]))

model = tf.keras.Model([enc_in, dec_in], logits)
model.compile(optimizer="adam",
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])

The decoder input is target[:-1] and the label is target[1:]: the one-token shift that makes next-token prediction supervised.

Check yourself

1. What problem does attention solve that a fixed-size context vector cannot?
2. Why does teacher forcing hurt at inference time?
3. Why is the decoder input shifted one token from its label?

Key takeaways

• Seq2seq splits the problem into an encoder, a decoder, and a token-by-token objective.
• Attention replaces a single bottleneck vector with a learned, dynamic alignment.
• Masking padding and shifting the target are the two details that make training work.