Hyper-Parameter Optimization

Hyperparameters — learning rate, layer sizes, dropout rate, batch size — control how well a model can fit, and they are not learned by gradient descent. Hyperparameter optimization is a search over that space, and the honest goal is a model that generalizes, not a lucky score on the test set.

The idea

Two search strategies dominate:

• Grid search — evaluate every combination. Exhaustive and simple, but cost grows multiplicatively with each axis.
• Random search — sample combinations. For the same budget it often beats grid search, because only a few hyperparameters actually matter and random sampling explores those axes more finely.

Bayesian and bandit methods such as successive halving and Hyperband adapt: they spend more budget on configurations that look promising early.

Critical discipline: split data into train, validation, and test. Tune on validation, report on test exactly once. Reusing the test set for selection leaks information and inflates the score.

Useful axes for a small Keras model: learning rate, hidden units, dropout, batch size, optimizer, weight decay. Learning rate matters more than the rest, so search it on a log scale.

Worked example

Tune a small model on tabular data with keras_tuner:

• Objective val_loss, direction min.
• max_trials=20, executions_per_trial=2 to average over random initializations.
• Learning rate sampled 1e-4 to 1e-2, log uniform.
• First dense layer 16 to 128 units in steps of 16.
• Dropout 0.0 to 0.5 in steps of 0.1.

With two executions per trial the reported score is the mean validation loss, which cuts the noise that otherwise makes two identical configs look different.

In code

import keras_tuner as kt, tensorflow as tf

def build(hp):
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(hp.Int("units", 16, 128, step=16),
                              activation="relu", input_shape=(n_features,)),
        tf.keras.layers.Dropout(hp.Float("dropout", 0.0, 0.5, step=0.1)),
        tf.keras.layers.Dense(1, activation="sigmoid"),
    ])
    model.compile(
        optimizer=tf.keras.optimizers.Adam(
            hp.Float("lr", 1e-4, 1e-2, sampling="log")),
        loss="binary_crossentropy", metrics=["accuracy"])
    return model

tuner = kt.RandomSearch(build, objective="val_loss", max_trials=20,
                        executions_per_trial=2, directory="kt", project_name="demo")
tuner.search(x_train, y_train, validation_split=0.2, epochs=30, verbose=0)

best = tuner.get_best_hyperparameters(1)[0]
print(best.get("lr"), best.get("units"), best.get("dropout"))

Check yourself

1. Why does random search often beat grid search at equal cost?
2. Why must the test set be used only once, after tuning?
3. Why is running two executions per trial worth the extra time?

Key takeaways

• Hyperparameters are chosen by search, not gradients; validation decides the winner.
• Random beats grid because few axes carry most of the effect.
• Tuning on the test set is leakage, no matter how the numbers are reported.