Natural Language Processing

Text is variable-length and discrete, and neural nets want fixed-size tensors of numbers. The NLP pipeline is therefore three steps: tokenize, map tokens to integer IDs, then learn or load embeddings. Everything after that is a sequence model.

The idea

• Standardize and tokenize — lowercase, strip punctuation, split. A TextVectorization layer learns a vocabulary and maps strings to integer IDs.
• Vocabulary size — cap it, say 10,000 to 20,000 tokens, and route everything unseen to an out-of-vocabulary bucket. Bigger vocabularies mean bigger embedding tables.
• Sequence length — pad or truncate to a fixed length T. Ragged batching saves compute, but a fixed T is simpler.
• Embedding — tf.keras.layers.Embedding(vocab_size, dim) is a lookup table of shape (vocab_size, dim). IDs index rows; the rows are learned.
• Sequence model — LSTM, GRU, or a 1D convolution over the time axis, ending in a pooling layer and a dense output.

Worked example

Sentiment on IMDB: 25,000 training reviews, binary labels.

• TextVectorization(max_tokens=10_000, output_sequence_length=200).
• Embedding(10_000, 16) gives (batch, 200, 16).
• GlobalAveragePooling1D gives (batch, 16).
• Dense(16, relu) then Dense(1, sigmoid).

Scores above 0.5 decode to positive. A model this small is a strong baseline. An LSTM(64) usually improves on it but is slower and can overfit 25k examples without dropout. The embedding table alone holds 10,000 × 16 = 160,000 parameters, which already dwarfs the classifier head.

In code

import tensorflow as tf

vectorizer = tf.keras.layers.TextVectorization(
    max_tokens=10_000, output_sequence_length=200)
train_text = tf.data.Dataset.from_tensor_slices(x_train).batch(64)
vectorizer.adapt(train_text)                 # vocabulary from training text only

model = tf.keras.Sequential([
    vectorizer,
    tf.keras.layers.Embedding(10_000, 16),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(x_train, y_train, validation_split=0.2, epochs=5, batch_size=64)

print(model.predict(["this film was a joy to watch"]))

Check yourself

1. Why must TextVectorization be adapted on training text only?
2. What do the two axes of the embedding table's shape (10000, 16) mean?
3. Why does GlobalAveragePooling1D discard word order, and is that a problem for sentiment?

Key takeaways

• Tokenize, index, embed: three steps turn text into a tensor.
• The vocabulary size trades embedding-table size against unknown-word coverage.
• Pooling loses order; recurrent or convolutional layers keep it when that matters.