Building a Fine-Tuning Dataset

A fine-tune learns whatever is actually in the dataset, including the parts you did not intend to teach. Most failed runs are data problems — inconsistent formats, duplicated rows, or prompt tokens the loss was trained to predict.

The idea

Define the target behavior before collecting anything. Write down the exact input and output schema, then collect examples that follow it. A few hundred consistent, reviewed examples usually beat tens of thousands of scraped ones, because the model imitates the average of what it sees.

The checklist that prevents most surprises:

• Deduplicate exact and near-duplicates. Repeated rows get over-weighted and can leak from train into evaluation.
• Split by source, not by row. If several rows come from the same document, conversation, or user, keep them on the same side of the split. Random row splits leak near-copies into validation.
• Balance the answer space. A label with four examples contributes almost no gradient signal.
• Mask the prompt. Compute loss only on response tokens. Otherwise the model spends capacity learning to generate your instructions.
• Match the chat template. The special tokens and turn markers used in training must be the tokens used at inference.
• Resolve licensing and PII first. Once trained, data cannot be unlearned.

Worked example

Suppose you assemble 1,200 support-reply examples. Near-duplicate detection removes 180, leaving 1,020. You split by ticket thread: 900 train and 120 validation. You then measure token lengths and set max_seq_len to the 95th percentile, truncating the rest at a turn boundary rather than mid-sentence. Finally you print the labels once and confirm that every prompt token is -100, so only the response contributes to the loss.

In code

IGNORE = -100  # index ignored by cross-entropy loss

def encode(example, tok, max_len=1024):
    prompt = tok.apply_chat_template(
        example["messages"][:-1], tokenize=False, add_generation_prompt=True
    )
    answer = example["messages"][-1]["content"] + tok.eos_token
    prompt_ids = tok(prompt, add_special_tokens=False)["input_ids"]
    answer_ids = tok(answer, add_special_tokens=False)["input_ids"]
    ids = (prompt_ids + answer_ids)[:max_len]
    labels = ([IGNORE] * len(prompt_ids) + answer_ids)[:max_len]
    return {"input_ids": ids, "labels": labels, "attention_mask": [1] * len(ids)}

The mask is the line worth checking twice: labels is -100 for the prompt and real token ids for the answer, so gradients flow only where the model is supposed to produce text.

Check yourself

1. Why mask prompt tokens with -100 instead of training on the whole sequence?
2. You split 10,000 rows randomly and see suspiciously high validation accuracy. What most likely went wrong?
3. One label has 200 examples and another has 4. What will the model learn?

Key takeaways

• Define the target behavior and schema first, then collect examples that match it exactly.
• Deduplicate, split by source, and mask prompt tokens so the loss covers only responses.
• Dataset errors are learned; review the data the way you review model outputs.