Reinforcement Learning

Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make sequences of decisions by interacting with an environment and maximising cumulative rewards. Unlike supervised learning, which relies on labelled examples, RL agents explore through trial and error, receiving scalar feedback in the form of rewards. At each discrete time step an agent observes its current state, selects an action according to its policy, and receives a reward and a new state from the environment. Over many episodes—each comprising a series of state-action-reward transitions—the agent refines its policy so as to achieve narrowly defined goals, whether that be navigating a maze, balancing a pole on a cart, playing chess or optimising energy dispatch. Central concepts include value functions, which estimate expected future rewards, and policy functions, which map states to actions. Advances in deep neural network approximators have enabled RL to handle high-dimensional inputs and continuous control tasks, driving breakthroughs in games, robotics, autonomous vehicles and resource management.

Research from Nature Portfolio

A deep-reinforcement-learning framework has been developed to discover efficient and provably correct algorithms for matrix multiplication. By formulating algorithm search as a single-player game and applying Monte Carlo tree search guided by neural ‘policy’ and ‘value’ networks, the agent—AlphaTensor—identifies novel tensor decompositions that exceed the performance of long-standing human-designed methods such as Strassen’s algorithm. The approach yields state-of-the-art complexity for various matrix sizes and demonstrates flexibility across structured and hardware-specific optimisation objectives, illustrating the power of self-play and planning in automating the discovery of fundamental algorithms.

Research from all publishers

Counterfactual multi-agent policy gradients introduce a centralised critic that evaluates joint actions while employing a counterfactual baseline to isolate each agent’s contribution to shared rewards. This actor-critic method sharply improves credit assignment in cooperative multi-agent environments under partial observability. A hierarchical deep-reinforcement-learning framework has been proposed for fast and safe robot navigation. By decomposing the task into low-level policies for obstacle avoidance and high-level waypoint selection, and by integrating sim-to-real transfer techniques, the method achieves robust convergence, efficient sampling and reliable performance in both simulation and real-world tests. In microgrid energy management, distributional reinforcement learning with prioritised experience replay has been applied to battery dispatch under dynamic pricing. By modelling entire return distributions rather than expectations alone, the approach better captures uncertainties in renewable output and tariff signals, resulting in smoother convergence and more resilient storage operation under time-of-use regimes.

Reinforcement Learning publication trend

The graph below shows the total number of articles in reinforcement learning across all publications each year (not limited to Nature Index journals).

Technical terms

Markov decision process (MDP): A mathematical framework for sequential decision problems defined by a set of states, actions, transition probabilities and rewards.

Policy: A (deterministic or stochastic) mapping from states to actions that guides the agent’s behaviour.

Value function: A function that estimates expected cumulative future rewards starting from a given state (state-value) or state-action pair (action-value).

Bellman equation: A recursive identity expressing the value of a policy in terms of immediate reward plus discounted future value.

Q-learning: A model-free algorithm that iteratively refines an action-value table toward the optimal action-value function through temporal-difference updates.

Deep Q-network (DQN): A Q-learning extension replacing the Q-table with a neural network to handle high-dimensional inputs and enable function approximation.

Actor-critic: An RL architecture comprising an actor (policy) network for action selection and a critic (value) network for policy evaluation and guidance.

Exploration–exploitation trade-off: The balance between exploring new actions to improve policy estimates and exploiting current best actions to maximise immediate reward.

References

1. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature (2022).
2. Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence (2018).
3. A Hierarchical Deep Reinforcement Learning Framework With High Efficiency and Generalization for Fast and Safe Navigation. IEEE Transactions on Industrial Electronics (2022).
4. Prioritized experience replay based deep distributional reinforcement learning for battery operation in microgrids. Journal of Cleaner Production (2024).
5. Introduction to Reinforcement Learning.