Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2002.04676 · Machine Learning Science and Technology

Reinforcement learning enhanced quantum-inspired algorithm for combinatorial optimization

Dmitrii Beloborodov, Alexander E. Ulanov, Jakob Foerster, Shimon Whiteson, Alexander I. Lvovsky

Published Oct 20, 2020• 18 citations
arXivHTMLPDF

Abstract

Quantum hardware and quantum-inspired algorithms are becoming increasingly popular for combinatorial optimization. However, these algorithms may require careful hyperparameter tuning for each problem instance. We use a reinforcement learning agent in conjunction with a quantum-inspired algorithm to solve the Ising energy minimization problem, which is equivalent to the Maximum Cut problem. The agent controls the algorithm by tuning one of its parameters with the goal of improving recently seen solutions. We propose a new Rescaled Ranked Reward (R3) method that enables a stable single-player version of self-play training and helps the agent escape local optima. The training on any problem instance can be accelerated by applying transfer learning from an agent trained on randomly generated problems. Our approach allows sampling high quality solutions to the Ising problem with high probability and outperforms both baseline heuristics and a black-box hyperparameter optimization approach.

DOI
10.1088/2632-2153/abc328

Related papers

Reinforcement Learning for Variational Quantum Circuits Design

Simone Foderà, Gloria Turati, Riccardo Nembrini +2 more

Variational Quantum Algorithms have emerged as promising tools for solving optimization problems on quantum computers. These algorithms leverage a parametric quantum circuit called ansatz, where its parameters are adjusted by a classical optimizer with the goal of optimizing a certain cost function. However, a significant challenge lies in designing effective circuits for addressing specific problems. In this study, we leverage the powerful and flexible Reinforcement Learning paradigm to train an agent capable of autonomously generating quantum circuits that can be used as ansatzes in variational algorithms to solve optimization problems. The agent is trained on diverse problem instances, including Maximum Cut, Maximum Clique and Minimum Vertex Cover, built from different graph topologies and sizes. Our analysis of the circuits generated by the agent and the corresponding solutions shows that the proposed method is able to generate effective ansatzes. While our goal is not to propose any new specific ansatz, we observe how the agent has discovered a novel family of ansatzes effective for Maximum Cut problems, which we call $R_{yz}$-connected. We study the characteristics of one of these ansatzes by comparing it against state-of-the-art quantum algorithms across instances of varying graph topologies, sizes, and problem types. Our results indicate that the $R_{yz}$-connected circuit achieves high approximation ratios for Maximum Cut problems, further validating our proposed agent. In conclusion, our study highlights the potential of Reinforcement Learning techniques in assisting researchers to design effective quantum circuits which could have applications in a wide number of tasks.

arXiv:2409.05475•Sep 9, 2024•1 citation

Automated quantum programming via reinforcement learning for combinatorial optimization

Keri A. McKiernan, Erik Davis, M. Sohaib Alam +1 more

We develop a general method for incentive-based programming of hybrid quantum-classical computing systems using reinforcement learning, and apply this to solve combinatorial optimization problems on both simulated and real gate-based quantum computers. Relative to a set of randomly generated problem instances, agents trained through reinforcement learning techniques are capable of producing short quantum programs which generate high quality solutions on both types of quantum resources. We observe generalization to problems outside of the training set, as well as generalization from the simulated quantum resource to the physical quantum resource.

arXiv:1908.08054•Aug 21, 2019•12 citations

Reinforcement-Learning-Based Variational Quantum Circuits Optimization for Combinatorial Problems

Sami Khairy, Ruslan Shaydulin, Łukasz Cincio +2 more

Quantum computing exploits basic quantum phenomena such as state superposition and entanglement to perform computations. The Quantum Approximate Optimization Algorithm (QAOA) is arguably one of the leading quantum algorithms that can outperform classical state-of-the-art methods in the near term. QAOA is a hybrid quantum-classical algorithm that combines a parameterized quantum state evolution with a classical optimization routine to approximately solve combinatorial problems. The quality of the solution obtained by QAOA within a fixed budget of calls to the quantum computer depends on the performance of the classical optimization routine used to optimize the variational parameters. In this work, we propose an approach based on reinforcement learning (RL) to train a policy network that can be used to quickly find high-quality variational parameters for unseen combinatorial problem instances. The RL agent is trained on small problem instances which can be simulated on a classical computer, yet the learned RL policy is generalizable and can be used to efficiently solve larger instances. Extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our trained RL policy can reduce the optimality gap by a factor up to 8.61 compared with other off-the-shelf optimizers tested.

arXiv:1911.04574•Nov 11, 2019•21 citations

Reinforcement learning assisted recursive QAOA

Yash J. Patel, Sofiène Jerbi, Thomas Bäck +1 more

In recent years, variational quantum algorithms such as the Quantum Approximation Optimization Algorithm (QAOA) have gained popularity as they provide the hope of using NISQ devices to tackle hard combinatorial optimization problems. It is, however, known that at low depth, certain locality constraints of QAOA limit its performance. To go beyond these limitations, a non-local variant of QAOA, namely recursive QAOA (RQAOA), was proposed to improve the quality of approximate solutions. The RQAOA has been studied comparatively less than QAOA, and it is less understood, for instance, for what family of instances it may fail to provide high-quality solutions. However, as we are tackling NP-hard problems (specifically, the Ising spin model), it is expected that RQAOA does fail, raising the question of designing even better quantum algorithms for combinatorial optimization. In this spirit, we identify and analyze cases where (depth-1) RQAOA fails and, based on this, propose a reinforcement learning enhanced RQAOA variant (RL-RQAOA) that improves upon RQAOA. We show that the performance of RL-RQAOA improves over RQAOA: RL-RQAOA is strictly better on these identified instances where RQAOA underperforms and is similarly performing on instances where RQAOA is near-optimal. Our work exemplifies the potentially beneficial synergy between reinforcement learning and quantum (inspired) optimization in the design of new, even better heuristics for complex problems.

arXiv:2207.06294•EPJ Quantum Technology•Jan 17, 2024•29 citations

Quantum Machine Learning

Quantum Machine Learningby Jordi RIU This project applies reinforcement learning techniques to optimize a quantum algorithm.The upcoming quantum computers are capable of optimization tasks in an exponentially large parameter space.Having access to such a large optimization space potentially allows an advantage of Quantum methods over classical algorithms, but at the price of problems in the optimization process -just to name a few, the presence of local minima and the low convergence speeds-.This project explores the application of reinforcement learning techniques to control and optimize the operation of a quantum computer solving a hard combinatorial problem, namely the MaxCut problem.Our methods use a collection of quantum observables to feed the state of the agent, inspired on a similar classical approach to the problem applied to the antiferromagnetic Ising model lattice.Our results show how an agent using these observables can optimally control the operation of a quantum device to obtain quasi-optimal solutions when in combination with the Quantum Approximate Optimization Algorithm (QAOA).v

Between science and economics•Apr 1, 2020•9 citations

Reinforcement Learning Quantum Local Search

Chen-Yu Liu, Hsi‐Sheng Goan

Quantum Local Search (QLS) is a promising approach that employs small-scale quantum computers to tackle large combinatorial optimization problems through local search on quantum hardware. However, the random selection of the sub-problem to solve in QLS may not be efficient. In this study, we propose a reinforcement learning (RL) based approach for training an agent to improve sub-problem selection in QLS beyond random selection. Our results demonstrate that the RL agent effectively enhances the average approximation ratio of QLS on fully-connected random Ising problems, indicating the potential of combining RL techniques with Noisy Intermediate-scale Quantum (NISQ) algorithms. This research opens a promising direction for integrating RL into quantum computing to enhance the performance of optimization tasks.

arXiv:2304.06473•IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 17, 2023•11 citations
Quantum hardware and quantum-inspired algorithms are becoming increasingly popular for combinatorial optimization. However, these algorithms may require careful hyperparameter tuning for each problem instance. We use a reinforcement learning agent in conjunction with a quantum-inspired algorithm to solve the Ising energy minimization problem, which is equivalent to the Maximum Cut problem. The agent controls the algorithm by tuning one of its parameters with the goal of improving recently seen solutions. We propose a new Rescaled Ranked Reward (R3) method that enables a stable single-player version of self-play training and helps the agent escape local optima. The training on any problem instance can be accelerated by applying transfer learning from an agent trained on randomly generated problems. Our approach allows sampling high quality solutions to the Ising problem with high probability and outperforms both baseline heuristics and a black-box hyperparameter optimization approach.
Variational Quantum Algorithms have emerged as promising tools for solving optimization problems on quantum computers. These algorithms leverage a parametric quantum circuit called ansatz, where its parameters are adjusted by a classical optimizer with the goal of optimizing a certain cost function. However, a significant challenge lies in designing effective circuits for addressing specific problems. In this study, we leverage the powerful and flexible Reinforcement Learning paradigm to train an agent capable of autonomously generating quantum circuits that can be used as ansatzes in variational algorithms to solve optimization problems. The agent is trained on diverse problem instances, including Maximum Cut, Maximum Clique and Minimum Vertex Cover, built from different graph topologies and sizes. Our analysis of the circuits generated by the agent and the corresponding solutions shows that the proposed method is able to generate effective ansatzes. While our goal is not to propose any new specific ansatz, we observe how the agent has discovered a novel family of ansatzes effective for Maximum Cut problems, which we call RyzR_{yz}Ryz​-connected. We study the characteristics of one of these ansatzes by comparing it against state-of-the-art quantum algorithms across instances of varying graph topologies, sizes, and problem types. Our results indicate that the RyzR_{yz}Ryz​-connected circuit achieves high approximation ratios for Maximum Cut problems, further validating our proposed agent. In conclusion, our study highlights the potential of Reinforcement Learning techniques in assisting researchers to design effective quantum circuits which could have applications in a wide number of tasks.
We develop a general method for incentive-based programming of hybrid quantum-classical computing systems using reinforcement learning, and apply this to solve combinatorial optimization problems on both simulated and real gate-based quantum computers. Relative to a set of randomly generated problem instances, agents trained through reinforcement learning techniques are capable of producing short quantum programs which generate high quality solutions on both types of quantum resources. We observe generalization to problems outside of the training set, as well as generalization from the simulated quantum resource to the physical quantum resource.
Quantum computing exploits basic quantum phenomena such as state superposition and entanglement to perform computations. The Quantum Approximate Optimization Algorithm (QAOA) is arguably one of the leading quantum algorithms that can outperform classical state-of-the-art methods in the near term. QAOA is a hybrid quantum-classical algorithm that combines a parameterized quantum state evolution with a classical optimization routine to approximately solve combinatorial problems. The quality of the solution obtained by QAOA within a fixed budget of calls to the quantum computer depends on the performance of the classical optimization routine used to optimize the variational parameters. In this work, we propose an approach based on reinforcement learning (RL) to train a policy network that can be used to quickly find high-quality variational parameters for unseen combinatorial problem instances. The RL agent is trained on small problem instances which can be simulated on a classical computer, yet the learned RL policy is generalizable and can be used to efficiently solve larger instances. Extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our trained RL policy can reduce the optimality gap by a factor up to 8.61 compared with other off-the-shelf optimizers tested.
In recent years, variational quantum algorithms such as the Quantum Approximation Optimization Algorithm (QAOA) have gained popularity as they provide the hope of using NISQ devices to tackle hard combinatorial optimization problems. It is, however, known that at low depth, certain locality constraints of QAOA limit its performance. To go beyond these limitations, a non-local variant of QAOA, namely recursive QAOA (RQAOA), was proposed to improve the quality of approximate solutions. The RQAOA has been studied comparatively less than QAOA, and it is less understood, for instance, for what family of instances it may fail to provide high-quality solutions. However, as we are tackling NP-hard problems (specifically, the Ising spin model), it is expected that RQAOA does fail, raising the question of designing even better quantum algorithms for combinatorial optimization. In this spirit, we identify and analyze cases where (depth-1) RQAOA fails and, based on this, propose a reinforcement learning enhanced RQAOA variant (RL-RQAOA) that improves upon RQAOA. We show that the performance of RL-RQAOA improves over RQAOA: RL-RQAOA is strictly better on these identified instances where RQAOA underperforms and is similarly performing on instances where RQAOA is near-optimal. Our work exemplifies the potentially beneficial synergy between reinforcement learning and quantum (inspired) optimization in the design of new, even better heuristics for complex problems.
Quantum Machine Learningby Jordi RIU This project applies reinforcement learning techniques to optimize a quantum algorithm.The upcoming quantum computers are capable of optimization tasks in an exponentially large parameter space.Having access to such a large optimization space potentially allows an advantage of Quantum methods over classical algorithms, but at the price of problems in the optimization process -just to name a few, the presence of local minima and the low convergence speeds-.This project explores the application of reinforcement learning techniques to control and optimize the operation of a quantum computer solving a hard combinatorial problem, namely the MaxCut problem.Our methods use a collection of quantum observables to feed the state of the agent, inspired on a similar classical approach to the problem applied to the antiferromagnetic Ising model lattice.Our results show how an agent using these observables can optimally control the operation of a quantum device to obtain quasi-optimal solutions when in combination with the Quantum Approximate Optimization Algorithm (QAOA).v
Quantum Local Search (QLS) is a promising approach that employs small-scale quantum computers to tackle large combinatorial optimization problems through local search on quantum hardware. However, the random selection of the sub-problem to solve in QLS may not be efficient. In this study, we propose a reinforcement learning (RL) based approach for training an agent to improve sub-problem selection in QLS beyond random selection. Our results demonstrate that the RL agent effectively enhances the average approximation ratio of QLS on fully-connected random Ising problems, indicating the potential of combining RL techniques with Noisy Intermediate-scale Quantum (NISQ) algorithms. This research opens a promising direction for integrating RL into quantum computing to enhance the performance of optimization tasks.