ReLU is the most-used activation function: f(x) = max(0, x). Learn how it works, why it beats sigmoid, the dying ReLU problem, and how it compares to GELU and SiLU.
Updated August 2026
The rectified linear unit is the most widely used activation function in deep learning, and it is almost embarrassingly simple. If the input is positive, pass it through unchanged. If it is negative, output zero. That one rule, max(0, x), replaced the smooth sigmoid and tanh functions that came before it and made training deep networks practical. The simplicity is the point, not a compromise.
Quick answer: The rectified linear unit (ReLU) is an activation function defined as f(x) = max(0, x): it outputs the input when positive and zero otherwise. Applied after each layer, it gives a network the non-linearity it needs to learn complex patterns. Because it does not saturate for positive inputs, it also avoids the vanishing gradients that slowed the older sigmoid and tanh functions. It is the default activation in most modern neural networks.
The ReLU activation function is a rule applied to the output of a neuron that keeps positive values and zeroes out negative ones. Written as f(x) = max(0, x), its graph is a flat line at zero for negative inputs and a straight 45-degree line for positive ones, with a sharp bend at the origin.
That bend is what matters. A network built only from linear layers can represent only linear relationships, no matter how many layers it stacks. Inserting a non-linear function like ReLU between the layers breaks that limit and lets the network model curves and interactions. Nair and Hinton popularized ReLU in 2010, and within a few years it had become the standard choice for hidden layers.
ReLU is so widely used because it trains deep networks faster and more reliably than the activations it replaced. Three properties explain the popularity:
Together these make ReLU the default for the hidden layers of convolutional networks, and a common choice across most architectures that are not transformers.
ReLU is one of a family of activation functions, and choosing between them is a common decision. The table compares the most common options:
| Function | Formula | Range | Key trait |
|---|---|---|---|
| ReLU | max(0, x) | 0 to ∞ | Simple, fast, can "die" on negatives |
| Sigmoid | 1 / (1 + e^-x) | 0 to 1 | Saturates, used for binary outputs |
| Tanh | tanh(x) | -1 to 1 | Zero-centered but still saturates |
| Leaky ReLU | max(0.01x, x) | -∞ to ∞ | ReLU with a small negative slope |
| GELU | x · Φ(x) | ≈ -0.17 to ∞ | Smooth, weights inputs by a Gaussian |
| SiLU (Swish) | x · σ(x) | ≈ -0.28 to ∞ | Smooth, non-monotonic, sigmoid-gated |
The practical read: ReLU beats sigmoid and tanh in hidden layers, because both saturate and slow deep training. Sigmoid still earns a place at the output of a binary classifier, where its 0-to-1 range reads as a probability. The smooth activations, GELU and SiLU, often edge out ReLU in the largest models. ReLU stays the simple, fast default whenever a smooth curve is not worth the extra cost.
GELU and SiLU are smooth activation functions that behave like ReLU for large inputs but curve gently through the origin instead of bending sharply. Each multiplies the input by a gate between 0 and 1, so small negative inputs are shrunk rather than zeroed, and each is differentiable everywhere. That smoothness is why they have largely replaced plain ReLU inside large transformers.
GELU, the Gaussian error linear unit, is f(x) = x · Φ(x), where Φ is the cumulative distribution function of the standard normal. It weights each input by the probability that a random Gaussian value falls below it, so more-positive inputs are kept more fully. Hendrycks and Gimpel introduced it in 2016 and reported gains over ReLU across vision, language, and speech tasks. It is now the default activation in most large language models.
SiLU, the sigmoid linear unit (also called Swish), is f(x) = x · σ(x), the input times its own sigmoid. It is non-monotonic: for small negative inputs its output dips slightly below zero before climbing back, a bump that helps gradients flow. It came out of a 2017 search for better activation functions, where swapping ReLU for SiLU raised ImageNet top-1 accuracy by up to about 0.9% on the image classifiers tested.
The trade-off against ReLU is cost. GELU and SiLU both evaluate an exponential, so each is slower per pass, and the accuracy gain is usually small on ordinary networks. In a model with billions of parameters that gain is worth paying for. In a standard convolutional network it usually is not.
The dying ReLU problem is when a neuron gets stuck outputting zero for every input and stops learning. It happens because ReLU's gradient is zero for any negative input. If a neuron's weights push it into the negative range for all the training data, no gradient flows back to it, so its weights never update and the neuron is effectively dead.
The usual fixes replace the flat negative side with a small slope. Leaky ReLU outputs 0.01x for negatives instead of zero, so a small gradient always flows and the neuron can recover. The exponential linear unit (ELU) and the smooth GELU and SiLU do the same thing in different ways. When a large fraction of a network's neurons die, switching to one of these variants, or lowering the learning rate, usually brings the network back.
The derivative of ReLU is 1 for positive inputs and 0 for negative inputs. At exactly zero it is technically undefined, since the function has a sharp corner there, but frameworks simply use 0 (or sometimes 1) at that single point. This simple derivative, either fully on or fully off, is part of why ReLU is cheap to train.
ReLU is used instead of sigmoid in hidden layers because sigmoid saturates: its output flattens near 0 and 1, so its gradient vanishes and deep networks train slowly. ReLU keeps a constant gradient of 1 on its positive side, so it does not choke the signal, and it is cheaper to compute. Sigmoid is still useful at the output layer for probabilities.
GELU often performs slightly better than ReLU in large transformer models, which is why it is the standard activation there, but it is not universally better. GELU stays smooth and keeps a small gradient for negative inputs, so it sidesteps the dying ReLU problem. ReLU is cheaper to compute, because it avoids the Gaussian, and on smaller networks the accuracy gap is usually too small to justify GELU's extra cost.
Use Leaky ReLU when a network suffers from the dying ReLU problem, meaning many neurons output zero and stop learning. Leaky ReLU's small negative slope keeps a gradient alive for those neurons so they can recover. If plain ReLU trains well, there is usually no need to switch, since the difference on healthy networks is small.
This content was generated with the assistance of AI. Our AI prompt chain workflow is carefully grounded and preferences .gov and .edu citations when available. All content is reviewed by a Telnyx employee to ensure accuracy, relevance, and a high standard of quality.
