---
title: What is gradient boosting? How GBMs and boosted trees work
canonical: https://telnyx.com/learn-ai/gradient-boosting-machines
---
# What is gradient boosting? How GBMs and boosted trees work

[Start building](https://telnyx.com/sign-up)

### Ask AI

* [![GPT](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/chatgpt_icon-ea55e09d.jpeg?width=92&format=webp)](https://chat.openai.com/?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Claude](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/claude_icon-c7c8eef3.webp?width=92&format=webp)](https://claude.ai/new?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Perplexity](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/perplexity_icon-cd5b74f1.webp?width=92&format=webp)](https://www.perplexity.ai/search?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Gemini](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/gemini_icon-4ff1cbf4.webp?width=92&format=webp)](https://www.google.com/search?udm=50&aep=11&q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Grok](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/grok_icon-e588e5bd.webp?width=92&format=webp)](https://x.com/i/grok?text=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)

What is gradient boosting? How GBMs and boosted trees work

[Back to Glossary](https://telnyx.com/learn-ai)

# What is gradient boosting? How GBMs and boosted trees work

Gradient boosting builds a model from many small trees, each fixing the errors of the last. How GBMs work, XGBoost vs random forest, and Python code.

Editor: [Emily Bowen](https://www.linkedin.com/in/emilywynne/)

**Updated October 2026**

Gradient boosting is a machine learning technique that builds a prediction model one step at a time. Each step adds a small model, usually a decision tree, trained to correct the errors left by the steps before it. The finished model adds up the predictions of every step. In machine learning, GBM stands for gradient boosting machine, a model trained this way.

Each tree in a GBM is too simple to predict well on its own, yet a few hundred of them added in sequence make one of the most accurate models for tables of data.

In a [2022 benchmark](https://arxiv.org/abs/2207.08815) of 45 tabular datasets, with every model tuned, tree-based models such as XGBoost still beat deep neural networks on tables of about 10,000 rows.

This content was generated with the assistance of AI. Our AI [prompt chain](https://telnyx.com/learn-ai/promptchain-ai) workflow is carefully [grounded](https://telnyx.com/learn-ai/grounding-ai) and preferences .gov and .edu citations when available. **All content is reviewed by a Telnyx employee to ensure accuracy, relevance, and a high standard of quality.**

## Sign up and start building.

[Sign Up](https://telnyx.com/sign-up)[Contact Us](https://telnyx.com/contact-us)

## What is gradient boosting?

Gradient boosting is a form of boosting, the family of methods that train simple models in sequence. Each new model concentrates on what the combined model still gets wrong.

In gradient boosting, each new model learns the negative gradient of a loss function: the direction the predictions should move to lower the error. Jerome Friedman of Stanford introduced the method in 2001 in [Greedy function approximation](https://doi.org/10.1214/aos/1013203451).

The loss is a choice, so the same algorithm covers regression with squared or absolute error and classification with log loss.

## How does gradient boosting work?

Gradient boosting starts from one constant prediction, then repeats a single step: fit a small tree to the current errors, and add a fraction of that tree's prediction to the model.

For regression with squared error, each round runs like this:

1. The model starts by predicting the average of the target for every row.
2. Each row's residual is its actual value minus the current prediction.
3. A small decision tree learns to predict those residuals from the features.
4. The tree's predictions are added to the model, scaled down by the learning rate.
5. Steps 2 to 4 repeat for a set number of trees, or until a validation score stops improving.

In a toy set of six support calls, the one feature is how many issues each caller raised: 1, 1, 2, 3, 3 and 4\. The target is each call's length in minutes: 5, 6, 7, 10, 11 and 15.

The starting prediction is the average, 9 minutes, for every call. The residuals run from -4 to +6 minutes. The mean squared error, the average of the squared residuals, is 11.7.

A decision tree divides rows on one feature at a time and predicts one constant value for each group it ends with, called a leaf. Each tree in this example makes a single split.

The first tree splits the calls at two issues or fewer, the split that most reduces the squared error of the residuals. It predicts a residual of -3 minutes for those three calls and +3 for the other three.

A learning rate of 0.5 adds half of each value, a larger step than the usual 0.1, chosen here to keep each round visible.

Calls with one or two issues now get 7.5 minutes, calls with three or four get 10.5\. The error falls to 4.9\. The second tree splits at three issues or fewer and the third at one, cutting the error to 1.9, then 1.0.

After 50 trees the model predicts 5.5, 5.5, 7, 10.5, 10.5 and 15 minutes. Calls with the same number of issues get the same prediction, because no tree can split rows that share every feature value.

At [inference time](https://telnyx.com/resources/machine-learning-inference), a trained GBM runs a new row through every tree and adds the starting average to the sum of the trees' scaled outputs.

## Gradient boosted decision trees

Gradient boosted decision trees (GBDT), also called gradient tree boosting, are gradient boosting with small decision trees as the models it adds. XGBoost, LightGBM, CatBoost and scikit-learn all implement this form.

Boosting keeps each tree shallow, so each one underfits on its own. The sequence of trees does the learning.

Tree depth sets which feature interactions a GBM can learn. A path through a tree three levels deep tests at most three features, such as a customer's plan, the time of day and the number of issues.

A tree with a single split, called a stump, can only learn each feature's effect on its own. scikit-learn's GradientBoostingRegressor defaults to trees three levels deep; its histogram-based estimators cap each tree at 31 leaves instead.

## Gradient boosting for regression and classification

Gradient boosting runs the same loop for regression and classification with a different [loss function](https://telnyx.com/learn-ai/loss-function), which decides what each new tree learns.

For regression with squared error, the negative gradient of the loss is the residual itself: the gap between the true value and the prediction. That is why each tree in the six-call example could fit the residuals directly.

Absolute error and Huber loss, which squares small errors and takes the absolute value of large ones, give outliers less pull on the model. Friedman's paper covers both.

For [classification](https://telnyx.com/learn-ai/classification-ai), a GBM adds its trees on the log-odds scale, the logarithm of p / (1 - p), where 0 means even odds. A sigmoid function turns the sum into a probability between 0 and 1.

The loss is log loss, which grows as the predicted probability of the true label falls. Each new tree fits the gap between the true label, 0 or 1, and the current predicted probability.

That gap is on the probability scale, while the model adds on the log-odds scale. So before a tree is added, each leaf's value is reset to the log-odds step that most lowers the loss for its rows, the line search in Friedman's algorithm.

With more than two classes, each round adds one tree per class.

### Overfitting and early stopping

A GBM overfits when it keeps adding trees after its loss on held-out data stops falling. Tracking that loss shows where to stop.

On scikit-learn's diabetes dataset (442 patients, 10 features), a GradientBoostingRegressor with depth-3 trees and a learning rate of 0.1 trained on 331 rows and was scored on the other 111 (a random split, seed 0).

Validation error was lowest at 13 trees, a mean squared error of 3,514, against 4,965 for predicting the average. The same 111 rows chose that count, so 3,514 flatters the model.

By 500 trees, training error had fallen to 35 while validation error had climbed to 4,364, 24% above its low.

The extra trees had fit noise in the 331 training rows. Early stopping ends training once the validation loss has gone a set number of rounds without improving.

## Gradient boosting in Python

scikit-learn's HistGradientBoostingClassifier trains a gradient boosted model in a few lines, with early stopping ending training once the held-out loss stops improving.

The example trains it on the breast cancer dataset that ships with scikit-learn (569 tumors, 30 features) and holds out 20% of the rows for testing:

Python

12345678910111213141516171819

from sklearn.datasets import load\_breast\_cancerfrom sklearn.ensemble import HistGradientBoostingClassifierfrom sklearn.model\_selection import train\_test\_splitX, y \= load\_breast\_cancer(return\_X\_y\=True)X\_train, X\_test, y\_train, y\_test \= train\_test\_split(X, y, test\_size\=0.2, random\_state\=0)model \= HistGradientBoostingClassifier( learning\_rate\=0.1, \# how much of each new tree to add max\_iter\=500, \# upper limit on the number of trees max\_leaf\_nodes\=15, \# size of each tree early\_stopping\=True, \# hold out 10% of the training data... n\_iter\_no\_change\=10, \# ...and stop after 10 trees without improvement random\_state\=0,)model.fit(X\_train, y\_train)print("trees built:", model.n\_iter\_)print("test accuracy:", round(model.score(X\_test, y\_test), 3))

The model stopped after 55 trees, 10 rounds after its best one. It classified 111 of the 114 test rows (97.4%) correctly. Four settings control it:

* `learning_rate` scales each tree's contribution. scikit-learn's guide recommends 0.1 or lower, with `max_iter` set high enough that early stopping, not the cap, ends training.
* `max_iter` caps the number of trees.
* `max_leaf_nodes` (or `max_depth`) sets the size of each tree, which sets how many features can interact.
* `early_stopping` holds out 10% of the training rows and stops after `n_iter_no_change` rounds without improvement.

Early stopping takes care of the tree count, which leaves the learning rate and tree size for [hyperparameter tuning](https://telnyx.com/learn-ai/hyperparameter-optimization).

The older GradientBoostingClassifier has a `subsample` setting that trains each tree on a random fraction of the rows, typically half. Friedman called this [stochastic gradient boosting](https://doi.org/10.1016/S0167-9473%2801%2900065-2).

Combined with a small learning rate, subsampling can further increase accuracy, according to scikit-learn's guide.

HistGradientBoostingClassifier bins each feature into at most 255 values before training and handles missing values and categorical features natively.

scikit-learn says the histogram version can be orders of magnitude faster than the older class above tens of thousands of rows.

## XGBoost, LightGBM, and CatBoost

XGBoost, LightGBM and CatBoost are open-source libraries that implement gradient boosted decision trees. They run the same algorithm and differ in how they find splits, sample rows and handle categories.

| Library  | Paper                                                                                                                          | What it adds to gradient boosting                                                                                                                                                                                                            |
| -------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| XGBoost  | [Chen and Guestrin, 2016](https://arxiv.org/abs/1603.02754)                                                                    | A regularized objective, second-order gradient statistics, a learned default branch for missing values, and a system design that scales past billions of rows                                                                                |
| LightGBM | [Ke et al., 2017](https://proceedings.neurips.cc/paper%5Ffiles/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html) | Histogram-based splits, sampling that keeps the rows with large gradients, bundling of features that are rarely nonzero together, and [leaf-wise growth](https://lightgbm.readthedocs.io/en/latest/Features.html)                            |
| CatBoost | [Prokhorenkova et al., 2018](https://arxiv.org/abs/1706.09516)                                                                 | Ordered boosting, which computes each row's residual with a model that never saw that row, and an ordered encoding of categorical features; both stop the label leaking into the model's own inputs. Symmetric trees use one split per level |

The [LightGBM paper](https://proceedings.neurips.cc/paper%5Ffiles/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html) reports training up to more than 20 times faster than conventional GBDT at almost the same accuracy. LightGBM targets data with many rows and many features. CatBoost targets data where many features are categorical.

## Gradient boosting vs random forest

Gradient boosting adds shallow trees one after another, while a random forest grows deep trees independently and averages them.

In Leo Breiman's [random forest](https://doi.org/10.1023/A:1010933404324), each tree trains on a sample of the rows drawn with replacement and picks from a random subset of features at every split. The trees grow deep and are not pruned.

A random forest averages deep trees that each overfit, which lowers [variance](https://telnyx.com/learn-ai/bias-variance-tradeoff). Gradient boosting adds shallow trees that each underfit, which lowers bias.

Adding trees to a random forest does not make it overfit: Breiman showed that its error converges to a limit as the forest grows. Adding trees to a GBM can, as the diabetes run showed, so the tree count has to be chosen.

Random forest trees can train in parallel because no tree depends on another. Boosting's shallow trees are faster to fit and predict than a forest's deep ones, according to scikit-learn's guide.

In the 2022 benchmark, the best tabular models were tree ensembles of both kinds, bagged random forests and boosted trees. scikit-learn's guide advises trying both on the problem at hand.

## Advantages and limitations of gradient boosting

The main advantages of gradient boosting are accuracy on tabular data, little data preparation and interpretability. Its main limits are sequential training, overfitting, no extrapolation and weak results on raw text and images.

The advantages:

* Strong accuracy on tables of numeric and categorical features.
* Little preparation. Trees split on thresholds, so features need no scaling. XGBoost and scikit-learn's histogram estimators route missing values without imputation.
* Interpretability. Permutation importance measures how much the score drops when one feature's values are shuffled. Partial dependence plots show how the prediction changes as one feature varies.

The limitations:

* Sequential training. Each tree waits for the one before it. Libraries parallelize the work inside each tree, such as building feature histograms, but not across rounds.
* Overfitting. More trees can raise validation error, as the diabetes run showed, so the tree count needs early stopping or a validation check.
* No extrapolation. Each tree predicts constants learned from the training range. In the six-call example, a call with 6 issues, or 10, gets the same 15 minutes as a call with 4.
* Raw text and images. Deep learning has driven the progress on text and image data, as the authors of the 2022 benchmark note. A GBM needs features someone has already extracted.

That limit sets the division of labor in a voice AI agent: the language model handles the conversation, while fields extracted from each finished call become a GBM's training rows.

The Telnyx [Voice API](https://telnyx.com/products/voice-api) hangup event records when a call started, why it ended and its audio quality. [AI Insights](https://developers.telnyx.com/docs/inference/ai-insights/structured-insights) extracts fields such as a 1-to-5 sentiment score from conversations with Telnyx [voice AI agents](https://telnyx.com/products/voice-ai-agents).

## Frequently asked questions

### What does GBM stand for in machine learning?

In machine learning, GBM stands for gradient boosting machine: a model built by gradient boosting, usually from decision trees. The name comes from Friedman's 2001 paper.

In medicine, GBM means glioblastoma, a type of brain tumor.

### What is the difference between gradient boosting and gradient descent?

Gradient descent improves a model by adjusting its parameters step by step, while gradient boosting improves it by adding a new model at each step.

Friedman framed boosting as [gradient descent](https://telnyx.com/learn-ai/batch-gradient-descent) in function space: each step adds a whole function, a tree, instead of nudging a parameter.

### What is the difference between AdaBoost and gradient boosting?

[AdaBoost](https://doi.org/10.1006/jcss.1997.1504), from Yoav Freund and Robert Schapire, raises the weight of every training example the previous models misclassified. Gradient boosting instead fits each new model to the negative gradient of a chosen loss.

AdaBoost is a special case: gradient boosting with exponential loss recovers the AdaBoost algorithm, as scikit-learn's documentation notes.

### Is XGBoost the same as gradient boosting?

XGBoost is a library that implements gradient boosting, so it runs the same algorithm. Its additions, such as a regularized objective, change how the trees are built, not what the model is.

### How is the learning rate used in gradient boosting?

The learning rate scales each tree's contribution before it is added to the model. At 0.1, each tree moves the predictions a tenth of the way toward what it learned.

Smaller steps need more trees and tend to give better test error, according to scikit-learn's guide.

### What are weak learners?

A weak learner is a model that predicts only a little better than chance, such as a decision stump with one split. Boosting combines many weak learners, trained in sequence, into one accurate model.

## Sources

* Friedman, J. H. (2001). [Greedy function approximation: a gradient boosting machine](https://doi.org/10.1214/aos/1013203451). _Annals of Statistics_, 29(5), 1189 to 1232.
* Friedman, J. H. (2002). [Stochastic gradient boosting](https://doi.org/10.1016/S0167-9473%2801%2900065-2). _Computational Statistics and Data Analysis_, 38(4), 367 to 378.
* Freund, Y., and Schapire, R. E. (1997). [A decision-theoretic generalization of on-line learning and an application to boosting](https://doi.org/10.1006/jcss.1997.1504). _Journal of Computer and System Sciences_, 55(1), 119 to 139.
* Breiman, L. (2001). [Random forests](https://doi.org/10.1023/A:1010933404324). _Machine Learning_, 45, 5 to 32.
* Chen, T., and Guestrin, C. (2016). [XGBoost: a scalable tree boosting system](https://arxiv.org/abs/1603.02754). KDD 2016.
* Ke, G., et al. (2017). [LightGBM: a highly efficient gradient boosting decision tree](https://proceedings.neurips.cc/paper%5Ffiles/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html). NeurIPS 2017.
* Prokhorenkova, L., et al. (2018). [CatBoost: unbiased boosting with categorical features](https://arxiv.org/abs/1706.09516). NeurIPS 2018.
* Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). [Why do tree-based models still outperform deep learning on tabular data?](https://arxiv.org/abs/2207.08815) NeurIPS 2022 Datasets and Benchmarks.
* LightGBM documentation. [Features: leaf-wise tree growth](https://lightgbm.readthedocs.io/en/latest/Features.html).
* scikit-learn documentation. Ensembles: gradient boosting, random forests and other methods; GradientBoostingRegressor, GradientBoostingClassifier and HistGradientBoostingClassifier reference (version 1.6).
* Telnyx. [Structured insights](https://developers.telnyx.com/docs/inference/ai-insights/structured-insights) and [insight groups](https://developers.telnyx.com/docs/inference/ai-insights/insight-groups). Telnyx developer documentation.
* Telnyx. [Call hangup webhook](https://developers.telnyx.com/api-reference/callbacks/call-hangup). Telnyx API reference.

Share on Social

## Jump to:

[What is gradient boosting?](#what-is-gradient-boosting)[How does gradient boosting work?](#how-does-gradient-boosting-work)[Gradient boosted decision trees](#gradient-boosted-decision-trees)[Gradient boosting for regression and classification](#gradient-boosting-for-regression-and-classification)[Gradient boosting in Python](#gradient-boosting-in-python)[XGBoost, LightGBM, and CatBoost](#xgboost-lightgbm-and-catboost)[Gradient boosting vs random forest](#gradient-boosting-vs-random-forest)[Advantages and limitations of gradient boosting](#advantages-and-limitations-of-gradient-boosting)[Frequently asked questions](#frequently-asked-questions)[Sources](#sources)

## Sign up for emails of our latest articles and news