Prompting vs Fine-Tuning

Every behavior you want from a language model can be pursued two ways: describe it in the prompt, or bake it into the weights. Prompting is fast, reversible, and paid for on every call. Fine-tuning is slow, versioned, and paid for once. Choosing between them is mostly a question of scale, stability, and how much of the task can actually be expressed in words.

The idea

Prompting leaves the weights frozen and steers behavior with instructions, examples, and reasoning steps. You need no training data, ship a change in minutes, and can carry fresh facts in the context. The price is paid on every request: prompt tokens, extra latency, and a context window that example-heavy prompts can crowd out.

Fine-tuning updates the weights with supervised examples. Behavior becomes the model's default, so prompts shrink, formats stay consistent, and per-call cost and latency fall. The price is paid up front: collecting and cleaning data, running a training job, and evaluating the result. Each change means another training run, and a narrow dataset can degrade general ability.

Three decision rules cover most cases:

• Prompt when the task is expressible in words, examples are few, or requirements change weekly.
• Fine-tune when the prompt is enormous, the output format must be rigid, per-call latency or cost matters at volume, or the target style is hard to describe but easy to demonstrate.
• Fine-tuning cannot inject fresh knowledge reliably. Use prompting for facts that change, fine-tuning for behavior that should be stable.

The two compose: fine-tune for the behavior, then prompt for the specifics of each request.

Worked example

Suppose a ticket classifier prompts with eight labeled examples at roughly 150 tokens each, so about 1,200 instruction tokens per request. At one million requests, that is 1.2 billion prompt tokens spent re-teaching the same task. A fine-tuned model that has the task in its weights needs only a short prompt, so the per-request overhead drops sharply.

That saving is only real if accuracy holds. Always compare the fine-tune against the prompted base model on the same held-out set before counting the win.

In code

Prompting keeps the behavior in a messages array that is rebuilt on every call. Fine-tuning turns the same behavior into one training record:

messages = [  # few-shot prompt: behavior lives in the context window
    {"role": "system", "content": "Classify the ticket as billing, bug, or other."},
    {"role": "user", "content": "Charged twice this month"},
    {"role": "assistant", "content": "billing"},
    {"role": "user", "content": "App crashes on upload"},
    {"role": "assistant", "content": "bug"},
    {"role": "user", "content": "Can you add dark mode?"},
]

record = {"messages": [  # the same task as one supervised example
    {"role": "system", "content": "Classify the ticket as billing, bug, or other."},
    {"role": "user", "content": "Can you add dark mode?"},
    {"role": "assistant", "content": "other"},
]}

Check yourself

1. Your task depends on a policy document that changes weekly. Why is prompting the safer choice?
2. Name two costs that prompting pays on every request but a fine-tuned model largely avoids.
3. You fine-tune on 500 support tickets and accuracy on new tickets drops. What is the first baseline you should compare against?

Key takeaways

• Prompting steers a frozen model; fine-tuning changes the model.
• Fine-tuning trades a one-time data and compute cost for cheaper, more consistent, lower-latency inference.
• Fine-tuning teaches behavior, not fresh facts; prompt for knowledge and fine-tune for form.