---
title: What is deep double descent? Model-wise, epoch-wise, and sample-wise
canonical: https://telnyx.com/learn-ai/double-descent-deep-learning
---
# What is deep double descent? Model-wise, epoch-wise, and sample-wise

[Start building](https://telnyx.com/sign-up)

### Ask AI

* [![GPT](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/chatgpt_icon-ea55e09d.jpeg?width=92&format=webp)](https://chat.openai.com/?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Claude](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/claude_icon-c7c8eef3.webp?width=92&format=webp)](https://claude.ai/new?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Perplexity](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/perplexity_icon-cd5b74f1.webp?width=92&format=webp)](https://www.perplexity.ai/search?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Gemini](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/gemini_icon-4ff1cbf4.webp?width=92&format=webp)](https://www.google.com/search?udm=50&aep=11&q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Grok](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/grok_icon-e588e5bd.webp?width=92&format=webp)](https://x.com/i/grok?text=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)

[Back to Glossary](https://telnyx.com/learn-ai)

# What is deep double descent? Model-wise, epoch-wise, and sample-wise

Deep double descent is when a neural network's test error rises, then falls again as the model grows, trains longer, or gets more data. Causes and fixes.

Editor: [Emily Bowen](https://www.linkedin.com/in/emilywynne/)

**Updated October 2026**

Deep double descent is double descent in deep neural networks: test error that falls, rises, and falls again as the network gets bigger or trains for more epochs. [Nakkiran and colleagues](https://arxiv.org/abs/1912.02292) documented it in 2019 in convolutional networks, ResNets, and transformers. When the curve rises, the rise comes where the network is just barely able to fit its training data.

That condition produces a result the classical bias-variance picture does not predict: for a network near that edge, more training data can make test error worse. Whether  survives tuned regularization is a question for every kind of model. The evidence that tuning can remove the peak is why weight decay comes before a bigger network in the levers below. Deep networks add three ways to reach the peak and one measure that ties them together.

This content was generated with the assistance of AI. Our AI [prompt chain](https://telnyx.com/learn-ai/promptchain-ai) workflow is carefully [grounded](https://telnyx.com/learn-ai/grounding-ai) and preferences .gov and .edu citations when available. **All content is reviewed by a Telnyx employee to ensure accuracy, relevance, and a high standard of quality.**

## Sign up and start building.

[Sign Up](https://telnyx.com/sign-up)[Contact Us](https://telnyx.com/contact-us)

double descent

## Understanding double descent

Double descent in deep learning is best understood through one quantity that model size, training time, and regularization all change, measured against the size of the training set. Nakkiran and colleagues call it effective model complexity: the largest number of training examples on which a given network and training procedure reach near-zero training error. The authors use an average training error of 10% or less as near zero, a working choice rather than a constant. A bigger network raises it, training for longer raises it, and stronger regularization lowers it.

The test error peak falls where effective model complexity roughly equals the size of the training set, a zone the authors call the critical regime. Well below it, the network cannot fit its data, and more capacity helps. Well above it, the network fits its data with room to spare, and more capacity also helps. Inside it, the network can only just fit the data. That single picture explains all three types of deep double descent, because each is a different way to move a network into or out of the critical regime.

## Types of double descent

The three types of deep double descent differ in what changes along the horizontal axis: the size of the network, the length of training, or the amount of data.

### Model-wise double descent

Model-wise double descent appears when networks of increasing width are trained to completion on the same data. Nakkiran and colleagues varied the width of ResNet18 models on the CIFAR-10 and CIFAR-100 image datasets, training each for 4,000 epochs with the Adam optimizer. Test error fell, rose to a peak around the width at which the network first reached zero training error, and then fell again as width kept growing. The peak was sharpest with label noise, 15% of labels randomly corrupted in their headline experiment. Without noise, the same region often showed a plateau instead, though some settings, such as ResNets on CIFAR-100, kept a peak even with clean labels.

Training length matters here. The authors report that many of these effects often do not occur with optimal early stopping, which keeps the checkpoint with the lowest test error from anywhere in the run. In practice the same pick is made on a validation set. They explain it with the same picture: if stopping early keeps a network from reaching near-zero training error, the network never enters the critical regime. One exception they show is ResNets on CIFAR-100 without label noise, which showed model-wise double descent even with optimal early stopping.

### Epoch-wise double descent

Epoch-wise double descent appears in a single large network as it trains. Nakkiran and colleagues showed it in ResNet18 models on CIFAR-10 with 20% of labels corrupted, trained for 4,000 epochs with Adam. They found it across many settings that had label noise. The network's effective complexity grows with every epoch, so what happens depends on how far it has grown by the end of the run. A large network passes the training set size well before the run ends: its test error falls early, rises as it crosses the critical regime, and falls again, sometimes below its first minimum. A medium network only barely reaches near-zero training error by the end of the run. It shows only the first half of that curve, the classical U, and for it stopping early is better. A small network never reaches near-zero training error, and its test error keeps falling as training goes on.

### Sample-wise non-monotonicity

Sample-wise non-monotonicity is the most surprising type: for a fixed network, more training data can raise test error. A larger training set takes more capacity to fit, so adding data moves the critical regime to larger models. A network that sat comfortably past the peak on the smaller dataset can land on the peak with the larger one. Nakkiran and colleagues trained transformers for German-to-English translation on 4,000 and on 18,000 samples. The larger set lowered test loss for most model sizes, but it shifted the peak to larger models, and near that peak the 4.5 times larger dataset made test performance worse. They did not observe more data hurting when optimal early stopping was used.

## Mechanisms behind double descent

The mechanisms behind deep double descent come down to how a network fits data it can only barely fit. In the critical regime, the few fits that reach zero training error must bend the whole function to pass through every mislabeled or hard example. Nakkiran and colleagues describe this as forcing the network to wreck its overall structure. Well past the critical regime, the network also fits every training point, noise included, but it has many ways to do so. Stochastic gradient descent tends to find one that absorbs each noisy point locally and stays simple elsewhere, so the memorized noise does little damage.

In linear models the same contrast has a proof. At the threshold, the only exact fit comes from inverting a nearly singular matrix, which amplifies noise into very large weights. Past it, the fit with the smallest weights ignores directions the data does not constrain, which is why it generalizes. Gradient descent started from zero lands on exactly that smallest-weight fit, so in linear models the optimizer's preference is exactly the smallest-weight rule. Stochastic gradient descent's preference for simple fits, often called implicit regularization, plays the same role in deep networks, although the authors note that for networks this is an intuition rather than a proof. Explicit regularization, such as weight decay, adds a deliberate preference for small weights, and a well-tuned amount can shrink the peak or remove it.

What the network has to bend around sets the height of the peak. Nakkiran and colleagues found the peak grew as they corrupted a larger share of the labels. They argue the effect is not fundamentally about label noise but about mis-specification: any mismatch between what the network can express and what the labels demand, which is why some clean datasets still show a peak.

## Double descent vs grokking

Double descent and grokking both involve a network that generalizes after it has already fit its training data, but they describe different curves. Double descent plots test error against model size, training time, or data, and shows a rise and a second fall around the critical regime. [Grokking](https://arxiv.org/abs/2201.02177), described by Power and colleagues in 2022, plots accuracy against training steps. Training accuracy reaches 100% early, test accuracy stays near chance for a long time, and then it jumps to near perfect.

The settings differ too. Grokking was found on small algorithmic datasets, such as tables of modular arithmetic. Epoch-wise double descent was found on standard image and text datasets, and was strongest with label noise. For a ResNet trained on real images, double descent is the documented effect. [Davies, Langosco, and Krueger](https://arxiv.org/abs/2303.06173) proposed that both come from the same learning dynamics, in a framework built on how fast a network learns different patterns. They also demonstrated a model-wise version of grokking, which suggests the two are closer than their curves look.

## Mitigating double descent

Mitigating deep double descent means keeping a network out of the critical regime, or flattening the peak when it can't be avoided. The first step is to check whether a model is on the peak at all.

### Checking whether a model is on the peak

The definition of effective model complexity gives a direct check from training logs. The check compares the training error each network reached by the end of its run, with the paper's cutoff of 10% or less counting as near zero. If the smaller network never got there, the larger one got there only late in its run, and validation error got worse, the larger network has most likely moved into the critical regime. If both got there well before their runs ended, the cause is probably something else, such as a change in learning rate or augmentation. The same check covers the other two cases. After a longer run, the question is whether training error crossed the cutoff around the time validation error turned up. After adding data, it is whether a network that fit the smaller set reaches the cutoff on the larger one only late in the run, or not at all. In every case, validation error means each run's best checkpoint, so the comparison does not depend on where a run happened to stop.

### The levers, roughly by cost

The best checkpoint is the cheapest lever, because it reuses runs already paid for. Many of these effects often disappear with optimal early stopping, as the model-wise results show. The first move is to keep the checkpoint with the lowest validation error from the whole run rather than the last one. It is not always enough, as the CIFAR-100 case shows.

A tuned penalty is next, at the cost of a sweep of several runs at the current size. [Nakkiran, Venkat, Kakade, and Ma](https://arxiv.org/abs/2003.01897) proved that optimally tuned ridge regularization removes the peak in certain linear regression settings, and showed that it can mitigate the peak in more general models, including neural networks. The penalty in that result is retuned at each size, and the curve of each size's best-penalty error is what loses its peak. In deep learning, the matching tool is weight decay, chosen by validation error rather than left at a default. The tuning matters in both directions. The deep double descent experiments also swept weight decay itself and found the same rise and fall as sweeping width, so an untuned value can put a network on the peak.

A clearly larger network is the third lever, and it costs runs at several larger sizes. A network slightly past the size that first fits the data can be worse than a smaller one; one well past it is usually better. How far is far enough has no fixed answer. The authors report that the width of the critical interval depends on the data and the training procedure in ways they do not yet fully understand. A few sizes that span the critical regime, instead of a single step up, show which side of the peak a model is on.

Label quality is the fourth, and on a large dataset it can be the most expensive. Cleaning labels should lower the peak, since the peak grew as more labels were corrupted, though none of the cited experiments cleaned labels. It cannot remove the part of the peak that comes from mis-specification, which is why clean-label CIFAR-100 still showed one.

Longer training comes last, because the published second descents took thousands of epochs, far longer than most schedules. With epoch-wise double descent, the first minimum of validation error is not always the best one. At first, a rise in validation error looks the same whether it is classical overfitting or the critical regime, and the difference is timing. If training error reached near zero early in the run and validation error turned up around then, the network is most likely the large case. The published runs needed thousands of epochs to come back down. If training error only reaches near zero as the budget runs out, it is the medium case, the classical U, and the best checkpoint is the answer. None of these experiments [fine-tuned](https://telnyx.com/resources/what-is-fine-tuning-ai) a pre-trained model, so on a small fine-tuning dataset the best-checkpoint rule is the safe default.

The levers interact. Weight decay lowers a network's effective complexity, and longer training raises it, so the two push a fixed network in opposite directions. Weight decay can pull a network that sat past the peak back onto it. Longer training can carry a network onto the peak and, if the network is large enough, through it.

## Implications for deep learning

In networks trained for many epochs on a fixed dataset, deep double descent is why having far more parameters than training examples is not by itself a warning sign. It is also why the rule "if the model overfits, make it smaller" can point the wrong way. [Over-parameterized](https://telnyx.com/learn-ai/overparameterization-in-llm) networks in that setting often generalize better than smaller ones.

For practitioners, the lesson is about evaluation. A single training run or a single model size can land on the peak and make a sound architecture look bad. Sweeps across width, training length, and data size catch the critical regime before it misleads a decision.

The largest language models are unlikely to reach the critical regime at all. Compute-optimal training, as [Hoffmann and colleagues](https://arxiv.org/abs/2203.15556) found, grows training data in step with model size: Chinchilla, with 70 billion parameters, trained on 1.4 trillion tokens. The runs behind their analysis each saw less than one pass over their data. A network that sees most of its data only about once never gets the repeated passes that fitting a training set takes. Its effective model complexity stays far below the size of that set. [Scaling-law studies](https://arxiv.org/abs/2001.08361) report loss improving smoothly with size, data, and compute, which fits that picture. Deep double descent matters most in the common setup outside large language models: many epochs over a fixed dataset.

## Frequently asked questions

### What is double descent in deep learning?

Double descent in deep learning is test error that falls, rises, and falls again as a neural network grows or trains longer. The peak sits in the critical regime, where the network is just able to fit its training data. Near that peak, more training data can also hurt.

### Can more training data hurt a neural network?

More training data can hurt a neural network of fixed size when the extra data moves the critical regime onto that network's size. Nakkiran and colleagues showed this for a transformer on a translation task. At most sizes more data helps; the harm is confined to a band of sizes near the peak, and they did not see it with optimal early stopping.

### Should training stop at the first minimum of validation error?

Stopping at the first minimum of validation error is right when a network only reaches near-zero training error as its run ends, and can be premature when it got there early. A network that never reaches near-zero training error has no early minimum to stop at, because its validation error keeps falling. With epoch-wise double descent, a network that fit its data early can see validation error rise and then fall below its first minimum. The decision depends on the whole validation curve and on the training budget, not its first turn. Keeping the checkpoint with the lowest validation error from the whole run avoids the choice, at the cost of running to the end.

### What is effective model complexity?

Effective model complexity is the largest number of training examples on which a network and its training procedure reach near-zero training error. Nakkiran and colleagues introduced it to explain deep double descent: the test error peak appears where effective model complexity roughly matches the training set size.

### Does deep double descent affect large language models?

Deep double descent has been measured in transformers trained for translation. Large language models trained the compute-optimal way are unlikely to reach the peak, because they see most of their training data only about once and never come close to fitting it. Smaller models trained for many epochs on limited data are the ones where it can appear.

## Sources

* Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. [Deep double descent: Where bigger models and more data hurt](https://arxiv.org/abs/1912.02292), ICLR, 2020.
* Nakkiran, P., Venkat, P., Kakade, S., and Ma, T. [Optimal regularization can mitigate double descent](https://arxiv.org/abs/2003.01897), ICLR, 2021.
* Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://arxiv.org/abs/2201.02177), arXiv, 2022.
* Davies, X., Langosco, L., and Krueger, D. [Unifying grokking and double descent](https://arxiv.org/abs/2303.06173), arXiv, 2023.
* Kaplan, J., et al. [Scaling laws for neural language models](https://arxiv.org/abs/2001.08361), arXiv, 2020.
* Belkin, M., Hsu, D., Ma, S., and Mandal, S. [Reconciling modern machine-learning practice and the classical bias-variance trade-off](https://www.pnas.org/doi/10.1073/pnas.1903070116), PNAS, 2019.
* Telnyx. [Understanding fine-tuning in AI models](https://telnyx.com/resources/what-is-fine-tuning-ai).
* Hoffmann, J., et al. [Training compute-optimal large language models](https://arxiv.org/abs/2203.15556), arXiv, 2022.

Share on Social

## Jump to:

[Understanding double descent](#understanding-double-descent)[Types of double descent](#types-of-double-descent)[Mechanisms behind double descent](#mechanisms-behind-double-descent)[Double descent vs grokking](#double-descent-vs-grokking)[Mitigating double descent](#mitigating-double-descent)[Implications for deep learning](#implications-for-deep-learning)[Frequently asked questions](#frequently-asked-questions)[Sources](#sources)

## Sign up for emails of our latest articles and news

What is deep double descent? Model-wise, epoch-wise, and sample-wise