Entropy measures impurity in machine learning. Learn the entropy formula, a worked example, and how it drives decision-tree splits and cross-entropy loss.
Updated August 2026
Uncertainty has an exact number, and that number is entropy: one bit for a coin flip, zero for a sure thing. Machine learning puts it to work in two very different places. It decides where a decision tree splits, and it measures how far off a classifier's probabilities are during training.
Quick answer: Entropy in machine learning measures the impurity, or uncertainty, in a set of data. It is highest when the classes are evenly mixed and zero when a set holds a single class. Decision trees use it to choose splits, and it underlies cross-entropy loss. The formula is H = -Σ p × log2(p), summed over each class.
Entropy measures unpredictability, not size or error. It comes from information theory, where Claude Shannon defined it as the average uncertainty in a set of outcomes. Machine learning borrows that idea to score how mixed a dataset is, or how unsure a model's prediction is.
That distinction is what makes it useful. Entropy lets a model ask a sharper question than "how many are in each class": it asks "how certain is this group." A model that lowers entropy makes its groups or its predictions more certain, not merely larger or more numerous.
Entropy is the negative sum, over every class, of each class probability times the base-2 log of that probability:
This content was generated with the assistance of AI. Our AI prompt chain workflow is carefully grounded and preferences .gov and .edu citations when available. All content is reviewed by a Telnyx employee to ensure accuracy, relevance, and a high standard of quality.
H = -Σ p × log2(p)
Take a node with two classes split evenly, so each probability is 0.5. Entropy is -(0.5 × log2(0.5) + 0.5 × log2(0.5)) = -(0.5 × -1 + 0.5 × -1) = 1.0. That is one bit, the maximum for two classes. Now take a pure node, where one class has probability 1.0. Entropy is -(1.0 × log2(1.0)) = 0. A partly mixed node falls in between: a 75/25 split has entropy -(0.75 × log2(0.75) + 0.25 × log2(0.25)) ≈ 0.81 bits, less uncertain than an even split but far from pure. The log base sets the unit: base 2 gives bits, the natural log gives nats. scikit-learn uses this formula for its entropy splitting criterion.
The logarithm turns a probability into information. A rare event carries more information than a common one, and the negative log captures that: the lower the probability, the larger the value. Base 2 measures that information in bits, the number of yes-or-no questions needed to pin down the outcome.
The log is also additive. The information from two independent events adds up, which is the property that lets entropy behave like a measure of total uncertainty rather than a single probability.
A decision tree uses entropy to score candidate splits through information gain. Information gain is the entropy before a split minus the weighted entropy of the groups after it. The tree tries each feature, measures how far entropy drops, and keeps the split with the largest gain.
Splitting the spam node by a feature that sends all spam one way and all legitimate mail the other takes entropy from 1.0 to 0. That is a gain of one full bit, the best possible result. A split that leaves both groups still mixed gains little, and the tree passes on it. See decision trees for the full structure and information gain for how the gain is scored.
Both measure node impurity, but they compute it differently. Gini impurity is the chance of misclassifying a random sample if you labeled it by the node's class distribution. Entropy is the information-theoretic uncertainty of that distribution. In practice, the two usually choose similar splits.
| Measure | Formula | Range, two classes | Note |
|---|---|---|---|
| Entropy | -Σ p × log2(p) | 0 to 1 | Slightly more sensitive to shifts in class balance |
| Gini impurity | Σ p × (1 - p) | 0 to 0.5 | Cheaper to compute, the common default |
Entropy costs a logarithm for every class. Gini skips it, which is why Gini is the default criterion in most libraries even though the resulting trees are close.
Cross-entropy loss is the training-time relative of entropy. It measures the gap between the true class distribution and the probabilities a model predicts, so minimizing it pushes the predicted distribution toward the truth. PyTorch's CrossEntropyLoss implements it directly, combining log-softmax with negative log-likelihood on raw logits. See cross-entropy for the loss in full.
This is why entropy appears in two places at once: as a splitting score inside a classification tree, and as the loss that trains a classifier through its objective function. Both ask the same thing, how far a set of predictions sits from certainty.
High entropy means high uncertainty: the classes are close to evenly mixed, and a prediction is close to a guess. Low entropy means the opposite, a set that is mostly or entirely one class. A model lowers entropy as it learns, either by splitting data into purer groups or by sharpening its predicted probabilities.
Watch entropy at the leaves of a tree, not just the root. A tree that reports low average entropy can still hold a high-entropy leaf covering the cases that matter most.
Start by measuring entropy where a decision actually gets made: at each candidate split, or at the output layer during training. A single impurity number for the whole dataset tells you little. The value is in the change, how much a split or a training step reduces it.
Then check the tradeoff. Driving entropy to zero on training data is easy and usually means the model has memorized noise. The goal is lower entropy on held-out data, which is a sign the groups or predictions are genuinely more certain, not just fitted to the sample.
Entropy shows up wherever a model measures or reduces uncertainty. Four common places:
The common thread is that entropy turns "how mixed" or "how unsure" into a single number a model can act on.
Entropy measures the impurity or uncertainty in a set of data. It is zero when the set holds one class and highest when the classes are evenly mixed. Models use it to find splits that make groups purer and to train classifiers through cross-entropy loss.
Entropy is H = -Σ p × log2(p), summed over each class, where p is the proportion of that class. With base-2 logs the unit is bits. A two-class node split evenly has entropy 1.0; a pure node has entropy 0.
In a decision tree, entropy scores how mixed a node is. The tree chooses splits by information gain, the drop in entropy from before a split to the weighted entropy after it. The larger the drop, the better the split.
Both measure node impurity. Entropy is the information-theoretic uncertainty of the class distribution; Gini is the chance of misclassifying a random sample. They usually pick similar splits, and Gini is cheaper because it skips the logarithm.
High entropy means the data is close to evenly split across classes, so the outcome is hard to predict. Low entropy means the set is mostly one class and the outcome is nearly certain. Learning is, in part, the work of lowering entropy on data the model has not seen.
