What is reinforcement learning? How it works, examples, and RLHF

Updated October 2026

Reinforcement learning (RL) is a type of machine learning in which an agent learns what to do by trial and error. It takes actions in an environment, receives rewards or penalties, and adjusts its behavior to maximize the reward it collects over time. No one shows it the right answer: it learns from the consequences of its own actions.

Reinforcement learning helped AlphaGo become the first program to defeat a world champion at Go. It also tuned the answers ChatGPT gives.

What is reinforcement learning?

Every reinforcement learning setup has the same six parts: an agent, an environment, states, actions, rewards and a policy. A warehouse robot learning to stack boxes shows each one.

• The agent is the robot's control software, the part that makes decisions.
• The environment is everything the agent acts on: the shelf, the boxes and gravity.
• A state is the situation at a moment, such as the stack's height and where the top box sits. The agent may see only part of it.
• An action is a choice the agent makes, such as where to place the next box.
• A reward is the number the environment sends back: points for a stable pile, lost points when the stack falls.
• The policy is the agent's rule for picking an action in each state. Learning means improving the policy.

In their textbook, Sutton and Barto name two features that set reinforcement learning apart: trial-and-error search and delayed reward.

An action can change rewards that arrive much later. For the robot, a box placed slightly off-center can look fine and still bring the stack down three boxes later.

Formally, the problem is a Markov decision process: the next state and reward depend only on the current state and the action taken.

In psychology, reinforcement makes a behavior more likely by tying it to a consequence. Machine learning borrowed the idea and the name; this page covers the machine learning method.

How does reinforcement learning work?

Reinforcement learning works as a loop. The agent observes the state and picks an action with its policy. The environment returns a reward and a new state.

To learn, the agent compares what it expected with what happened: the reward and the new state. It nudges its estimates toward the result, so actions that turn out better than expected become more likely.

In many tasks, the loop repeats over episodes, each a run from a starting state to an end, such as one attempt at a stack.

The agent aims at the return, not the next reward: the sum of the rewards still to come. Later rewards are often discounted by a rate between 0 and 1, so sooner rewards count more and the total stays finite in a task that never ends.

With a discount rate of 0.99, each step of delay multiplies a reward's weight by 0.99. A reward 100 steps further away counts about 0.37 times as much.

To choose well, many agents learn a value function: an estimate of the return to expect from a state. An action value, or Q-value, is the same estimate for taking one particular action in that state.

The central trade-off is between exploration and exploitation. The agent must exploit actions it already knows pay off, but it must also explore untried ones to find better choices.

A common rule is epsilon-greedy: take the best-known action most of the time and a random one with probability epsilon.

An agent that learned Atari games started at epsilon 1.0, fully random, and lowered it to 0.1 over its first million frames.

Types of reinforcement learning algorithms

Most reinforcement learning algorithms fall into two families. Value-based methods learn how good each action is, and policy-based methods learn the behavior directly. Actor-critic methods combine them.

Model-based and model-free methods split the algorithms a second way. Model-based methods learn or are given a model of the environment and plan with it; model-free methods learn from experience alone.

Here a model means a way to predict what the environment will do next, not a neural network. AlphaGo is model-based: it knows the rules of Go, so its search can play moves out before choosing one.

Value-based methods: Q-learning and deep Q-networks

Q-learning, introduced by Chris Watkins in 1989, learns a Q-value for each state and action, then picks the action with the highest value. It is model-free: it needs experience, not a model of the environment.

A table stops working when there are too many states to list, a problem known as the curse of dimensionality. Deep Q-networks replace the table with a neural network.

The first deep Q-network learned Atari games straight from screen pixels. Combining reinforcement learning with deep neural networks is called deep reinforcement learning.

Policy-based methods: policy gradients and PPO

The most common policy-based methods are policy gradient methods. They adjust the policy's parameters directly, making actions that led to a high return more likely.

Proximal policy optimization (PPO) caps how far each update can move the policy, because its authors found that large steps often wrecked a working policy.

PPO is the algorithm that tuned ChatGPT.

Actor-critic methods

An actor-critic method is a policy gradient method that also learns a value function.

The actor is the learned policy, which picks actions. The critic is the learned value function, which scores them, and the actor updates from those scores.

PPO, as usually run, is an actor-critic method: it trains a value function alongside the policy.

One actor-critic method handles continuous actions, such as a steering angle rather than a choice between left and right.

Examples of reinforcement learning

The simplest example of reinforcement learning is a video game: the score is the reward, and the player learns which moves raise it.

The Atari agent learned 49 games from pixels and the score alone, with the same algorithm and settings for every game.

On 29 of the 49 games, it scored more than 75% of a professional human games tester's score.

AlphaGo learned Go from human expert games and then from games against itself.

It beat the European Go champion 5 games to 0, the first win by a program over a human professional on a full-sized board. AlphaGo later became the first program to defeat a world champion.

AlphaGo Zero learned from self-play alone, starting from the rules, and beat AlphaGo 100 games to 0.

In robotics, a robot hand learned to manipulate a Rubik's cube with control policies trained only in simulation. A standard cube-solving program chose the moves; the hand learned to carry them out.

In driving research, agents have learned to keep a car in its lane in a racing simulator. A driving research survey reports that continuous steering gave smoother paths than a few fixed steering choices.

Spoken dialogue systems are an older example. In 2000, Levin, Pieraccini and Eckert learned an air travel information system's dialogue strategy with reinforcement learning, training against a simulated caller.

The strategy it learned resembled ones that research groups had designed by hand. Deciding what to ask a caller next is now the job of the language model inside voice AI agents.

Reinforcement learning in large language models

Large language models use reinforcement learning after pretraining, the stage in which a large language model learns by predicting text. Reinforcement learning makes their answers more helpful and, more recently, improves their reasoning.

ChatGPT was trained with reinforcement learning from human feedback (RLHF). OpenAI's launch post describes three steps: supervised fine-tuning, a reward model, and PPO.

First, people wrote example conversations for supervised fine-tuning. Then people ranked several model answers, and a reward model learned those rankings. Finally, PPO tuned the model toward answers the reward model scored highly.

In RLHF, the language model is the agent, and its policy is how it chooses a reply to a prompt. The prompt is the state, the reply is the action, and the reward model's score is the reward.

The method came from InstructGPT. Its paper treats each prompt and reply as a one-step episode: the reward model scores the reply, and the episode ends.

Human raters preferred answers from the 1.3 billion parameter InstructGPT model to those of the 175 billion parameter GPT-3, a model more than 100 times larger.

The idea predates chatbots. In 2017, Christiano and colleagues trained Atari and simulated-robot agents from people's choices between pairs of short clips.

A newer approach uses rewards a program can check. DeepSeek-R1-Zero was trained with reinforcement learning directly on a base model, with no supervised fine-tuning first.

Its rewards were rules: a correct final answer, checked against the known result for math or by test cases for code, and reasoning written out in a marked section before the answer.

On the AIME 2024 math exam, its pass@1 score, the share of problems solved on the first attempt, rose from 15.6% to 71.0%. The Tulu 3 authors call this approach reinforcement learning with verifiable rewards (RLVR).

DeepSeek trained with GRPO, a PPO variant that drops the critic, the value function PPO trains alongside the policy. GRPO compares each answer with the average reward of several answers to the same question.

Not every preference method is reinforcement learning. Direct preference optimization (DPO) learns from the same kind of human preferences with a simple classification loss, instead of RLHF's reward model and RL loop.

Challenges of reinforcement learning

Reinforcement learning is hard to get working for four reasons. It needs a lot of experience, the reward must say exactly what is wanted, exploring can be unsafe, and simulation differs from the real world.

It needs a lot of experience. The Atari agent trained on 50 million frames, around 38 days of game experience.

The reward must say exactly what is wanted. In a boat-racing game, an RL agent scored more by circling a lagoon to hit the same three targets than by finishing the race.

It averaged 20% more points than human players without completing the course. Amodei and colleagues call this failure reward hacking.

The reward works as the agent's objective function, so a reward that misses the goal trains an agent that misses it too.

Language models show the same failure. One study found that RLHF often makes answers longer. A reward based only on length reproduced most of RLHF's gains over supervised fine-tuning.

Another study found that people and reward models sometimes prefer answers that agree with the user over correct ones, which pushes models toward flattery.

InstructGPT added a penalty for drifting far from its supervised model, to limit over-optimization of the reward model.

Exploring can be unsafe. A game agent can try random moves, but a robot or a car cannot. Amodei and colleagues list safe exploration among five open problems in AI safety.

Simulation helps with cost and safety, but a policy trained in simulation can fail on real hardware. The Rubik's cube team randomized the simulator's physics, which they report improved transfer to the real robot.

Reinforcement learning vs supervised and unsupervised learning

Reinforcement learning learns from rewards, supervised learning from labeled examples of the right answer, and unsupervised learning from structure in unlabeled data.

So reinforcement learning is not supervised learning: no one supplies the correct action.

Unsupervised learning also works without labels, but it looks for hidden structure, while reinforcement learning maximizes a reward.

A spam filter trained on emails labeled spam or not spam is supervised learning. Grouping customers by purchase history is unsupervised learning. The warehouse robot learning to stack boxes is reinforcement learning.

Supervised and unsupervised learning work from a fixed dataset. A reinforcement learning agent creates its data by acting, so what it learns depends on what it tried.

Real systems mix them. AlphaGo learned from human expert games before playing itself, and ChatGPT's RLHF begins with supervised fine-tuning.

Reinforcement learning in voice AI agents

Voice AI agents descend from spoken dialogue systems, and research versions of those systems learned their dialogue policies with reinforcement learning from 2000 onward. Most deployed systems used hand-built dialogue managers instead.

Today's agents run on language models instead, such as chat models trained with RLHF.

By 2013, a review by Young and colleagues described dialogue systems that tracked uncertainty about what the caller had said, to cope with speech recognition errors. Those systems optimized the dialogue policy with rewards.

What carries over is the reward problem. A team has to define what a successful call is before it can tell whether a change helped. An agent tuned toward the wrong measure gets better at the wrong thing.

In Telnyx AI Assistants, AI Tests check an assistant against success criteria before it goes live, and the results show whether each criterion was met.

A success criterion plays the part a reward plays in reinforcement learning, but nothing is trained on it: the team reads the result and changes the assistant.

Assistant versions can split live calls by percentage, and the remaining calls go to the main version as a safety fallback. That works like controlled exploration: a small share of calls tries the new version.

For testing at scale, the Coval integration simulates many conversations from a few seed cases.

Useful criteria include task success and timing, such as time to first audio: the delay between the end of the caller's speech and the first audio of the reply.

The same approach applies to agents built on the Voice API with a team's own models, which receive call audio through media streaming.

Frequently asked questions

Is ChatGPT trained with reinforcement learning?

Yes. ChatGPT was trained with reinforcement learning from human feedback, with PPO tuning it toward answers a reward model scored highly.

Who is the father of reinforcement learning?

No single person founded reinforcement learning. Richard Sutton and Andrew Barto received the 2024 ACM A.M. Turing Award for developing its conceptual and algorithmic foundations, in papers beginning in the 1980s.

Is reinforcement learning deep learning?

No. Reinforcement learning is a way of learning from rewards, and deep learning is a way of building models from many-layered neural networks.

Deep reinforcement learning combines them, as deep Q-networks did. Q-learning on its own works with a simple table.

What is inverse reinforcement learning?

Inverse reinforcement learning works backward: it infers the reward an agent is pursuing from the agent's observed behavior, as a survey by Arora and Doshi defines it.

In driving research, it has been used to learn what makes a trajectory comfortable from human drivers' demonstrations, according to the driving survey.

What is negative reinforcement in reinforcement learning?

In reinforcement learning, a negative reward, or penalty, lowers the return of the actions that led to it, so the agent shifts toward actions with a higher return. A penalty can count against actions several steps back.

In psychology, negative reinforcement means removing an unpleasant stimulus after a behavior, which makes the behavior more likely.

Sources

• Sutton, R. S., and Barto, A. G. (2018). Reinforcement learning: an introduction, second edition. MIT Press.
• Watkins, C. J. C. H., and Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
• Mnih, V., et al. (2013). Playing Atari with deep reinforcement learning. NIPS Deep Learning Workshop 2013.
• Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529-533.
• Lillicrap, T. P., et al. (2016). Continuous control with deep reinforcement learning. ICLR 2016.
• Schulman, J., et al. (2017). Proximal policy optimization algorithms.
• Silver, D., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484-489.
• Silver, D., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354-359.
• Akkaya, I., et al. (2019). Solving Rubik's cube with a robot hand.
• Kiran, B. R., et al. (2020). Deep reinforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Transportation Systems, 23(6), 4909-4926 (2022).
• Christiano, P., et al. (2017). Deep reinforcement learning from human preferences. NeurIPS 2017.
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
• OpenAI (2022). Introducing ChatGPT.
• Shao, Z., et al. (2024). DeepSeekMath: pushing the limits of mathematical reasoning in open language models.
• DeepSeek-AI (2025). DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. Published in Nature, 645, 633-638.
• Lambert, N., et al. (2024). Tulu 3: pushing frontiers in open language model post-training.
• Rafailov, R., et al. (2023). Direct preference optimization: your language model is secretly a reward model. NeurIPS 2023.
• Amodei, D., et al. (2016). Concrete problems in AI safety.
• Singhal, P., et al. (2023). A long way to go: investigating length correlations in RLHF.
• Sharma, M., et al. (2023). Towards understanding sycophancy in language models.
• OpenAI (2016). Faulty reward functions in the wild.
• Levin, E., Pieraccini, R., and Eckert, W. (2000). A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on Speech and Audio Processing, 8, 11-23.
• Young, S., Gasic, M., Thomson, B., and Williams, J. D. (2013). POMDP-based statistical spoken dialog systems: a review. Proceedings of the IEEE, 101, 1160-1179.
• Arora, S., and Doshi, P. (2018). A survey of inverse reinforcement learning: challenges, methods and progress.
• ACM (2025). 2024 ACM A.M. Turing Award.
• American Psychological Association. APA Dictionary of Psychology: reinforcement and negative reinforcement.
• Telnyx. Testing and traffic distribution for AI Assistants and Media streaming. Telnyx developer documentation.