Ensemble Learning

An ensemble trains several models and combines their predictions. It works because partly independent errors cancel: when members are wrong on different examples, a vote is right more often than any single member. The cost is linear in the number of members for both training and inference, so ensembles are usually a final accuracy push rather than a first move.

The idea

Three families, distinguished by how they manufacture diversity:

• Bagging — train the same architecture on bootstrap resamples of the training set, then average. This is the idea behind random forests.
• Boosting — train models sequentially, each focused on the examples the previous ones got wrong. This is gradient boosting.
• Stacking — train a small meta-model that takes the members' predictions as inputs.

Diversity is the currency: two identical models average to one model. You buy diversity with different random seeds, different architectures, different data orderings, or different feature subsets.

For classification, average the probabilities and take the argmax. Averaging hard votes is simpler but throws away confidence, and averaging raw logits across models with differently scaled outputs is worse than either.

Worked example

Suppose five classifiers each err with probability $p = 0.2$ and their errors are independent. A majority vote is wrong only if three or more members are wrong:

The independence assumption is the whole trick. Real members trained on the same data and the same architecture are strongly correlated, so the true error lands somewhere between 0.058 and 0.2, and closer to 0.2 when the members share everything but a seed. That is why strong ensembles mix genuinely different architectures rather than five copies of one network.

In code

import tensorflow as tf

def make_member(seed, width=64):
    tf.keras.utils.set_random_seed(seed)
    return tf.keras.Sequential([
        tf.keras.layers.Input(shape=(28, 28, 1)),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(width, activation="relu"),
        tf.keras.layers.Dense(10),
    ])

members = []
for seed in range(5):
    m = make_member(seed)
    m.compile(optimizer="adam",
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])
    m.fit(x_train, y_train, epochs=3, batch_size=128, verbose=0)
    members.append(m)

probs = tf.add_n([tf.nn.softmax(m.predict(x_test)) for m in members]) / len(members)
y_pred = tf.argmax(probs, axis=1)

tf.add_n sums a list of same-shaped tensors, and dividing by the member count turns the sum into a mean.

Check yourself

1. Why does averaging probabilities usually beat majority voting on hard labels?
2. Why does the binomial calculation above overstate the benefit on real data?
3. What is the inference cost of a five-member ensemble compared to one model, and how would you reduce it?

Key takeaways

• Ensembles reduce error through diversity; identical members add only cost.
• Average probabilities and then take the argmax, rather than voting on hard labels.
• Bagging, boosting, and stacking differ in how they manufacture diversity.