Beam search keeps several promising sequences during decoding. Learn how beam width works, see an example, and compare beam search with greedy decoding.
Updated August 2026
Beam search is an approximate decoding algorithm that keeps a limited number of the most promising partial solutions at each step. In language generation, it expands several candidate token sequences instead of committing to only the highest-scoring next token. The number kept is the beam width.
Beam width is a measurable control: a width of 1 is greedy decoding, while larger values retain more candidates and require more computation. Hugging Face documents this relationship for transformer generation.
At each decoding step, beam search expands every sequence currently in the beam. It scores the resulting candidates, then retains only the top k sequences. The process repeats until a stop condition, such as an end token or maximum length. Published beam-search research describes this core procedure and the scoring choices used to rank candidates.
With a beam width of 3, the algorithm keeps three partial sequences after every step. A wider beam explores more alternatives, but requires more computation and can still miss the globally best sequence because it prunes candidates along the way.
Imagine a model starts a translation with three likely first tokens: The, A, and This. It expands each one at the next step, scores all resulting two-token sequences, and retains the three highest-scoring sequences. The best final sentence can come from a second-ranked first token, which greedy decoding would have discarded.
| Method | What it keeps | Main tradeoff |
|---|---|---|
| Greedy decoding | One best next choice | Fast, but can miss better full sequences |
| Beam search | Top k partial sequences | Better search coverage at higher compute cost |
| Sampling | Randomized candidates from a distribution | More variety, less deterministic output |
Beam width is the maximum number of candidate sequences retained per step. A width of 1 is greedy decoding. Increasing the width allows more alternatives to survive, but it increases memory and compute use.
Do not assume a larger beam always produces a better output. In text generation, wider beams can favor generic or overly short sequences unless the scoring method includes length normalization or other constraints.
Beam search is common in sequence-to-sequence tasks such as machine translation, speech recognition, summarization, and structured prediction. It is helpful when the model scores one step at a time but the system needs a strong full sequence.
It is not the default answer for every generative task. Systems that need diverse outputs often use sampling or diversification methods, while systems with strict constraints may use constrained decoding.
Beam search is approximate, not exhaustive. It can prune the sequence that would eventually become best, and its output quality depends on the model’s score calibration. Its computational cost grows with beam width and vocabulary expansion.
Evaluate decoding choices with representative inputs and task-specific metrics. A gain in likelihood is not necessarily a gain in usefulness, factuality, or diversity.
Choose beam width by measuring the task, not by assuming a larger value is safer. In one neural machine translation study, beam sizes larger than 5 reduced translation quality; the authors' rescoring method improved results by 2.0 BLEU over the tested length-normalization heuristic. That is evidence for task-specific tuning, not a universal beam-width limit.
Begin with a small set of widths, such as 1, 2, 4, and 8. Compare output quality, latency, token cost, and failure modes on a fixed evaluation set.
Beam width 1 gives greedy decoding. Higher values add candidate paths and increase the number of model scores that must be processed. At some point, the added compute may produce almost no improvement, or it may make the output more repetitive. Keep the smallest width that meets the task target.
Sequence probabilities are multiplied across tokens, so longer sequences can receive lower total scores even when they are sensible. Length normalization adjusts scoring so that the decoder does not favor a short sequence only because it contains fewer probability terms.
The exact method depends on the model and library. Document the scoring rule with the beam width. Without that detail, two systems described as using beam search can produce very different output behavior.
Beam search usually aims for high-scoring deterministic sequences. Sampling intentionally introduces randomness by choosing among candidates according to the model's probability distribution. It can produce more varied outputs, but the run-to-run result can differ.
For generation systems, test both approaches against the job. A structured extraction task may need repeatable decoding. A creative writing interface may value controlled diversity. The LLM API guide explains the implementation context for calling a language model, while the hard versus soft token guide helps clarify why decoding happens at the token level.
Beam search belongs to a broader set of model-inference choices. Read machine learning inference for the production side of running models, and inference latency before treating a decoding-quality improvement as free.
The training versus inference guide explains where decoding fits after model training is complete.
Neural machine translation is the clearest documented application in the sources used here. Freitag and Al-Onaizan evaluated beam-search strategies for translation and compared how candidate pruning and scoring affected decoded sentences. Yang, Huang, and Ma later examined rescoring and stopping criteria on Chinese-to-English translation.
Both studies treat beam width, candidate scoring, and stopping behavior as parts of one decoding design. That is the practical lesson: benchmark the full configuration on the target task instead of selecting a width in isolation.
Build a small evaluation set that includes short prompts, long prompts, repeated phrases, rare terms, and examples where a wrong token has a high cost. Run the same inputs with each candidate beam width and scoring rule. Record output quality, generation time, and the number of tokens processed.
Review the worst outputs by hand. An average metric can hide a pattern where the decoder produces fluent text but drops a required identifier, skips a negation, or ends too early. Those are the errors that should shape the decoding choice.
Beam search is greedy in the sense that it prunes candidates at every step, but it is less myopic than standard greedy decoding. Greedy decoding keeps one candidate, while beam search keeps a fixed number of the highest-scoring partial candidates.
Beam search with a width of 1 is equivalent to greedy decoding. The system chooses the highest-scoring next token at each step and never keeps an alternative path alive.
No. It explores only a limited subset of all possible sequences. It can produce a strong result efficiently, but an optimal search would need to consider many more paths and may be impractical for large vocabularies or long sequences.
This content was generated with the assistance of AI. Our AI prompt chain workflow is carefully grounded and preferences .gov and .edu citations when available. All content is reviewed by a Telnyx employee to ensure accuracy, relevance, and a high standard of quality.
