Implementació del QAOA en un algoritme de aprenentatge per reforç quàntic aplicat a les línies de feix al CERN

Nicolau Orell, Sebastià

Published Sep 14, 2023• 0 citations

Abstract

Quantum reinforcement learning (QRL), a cutting-edge field at the intersection of quantum computing and artificial intelligence, has the potential to revolutionize domains like chemistry and autonomous systems with its computational advantages. The free energy-based reinforcement learning algorithm (FERL) is a QRL algorithm inspired by the classical deep Q-learning algorithm (DQN) which uses a quantum Boltzmann machine (QBM) as the Q-function with quantum annealing (QA) as the solver. It has demonstrated considerable advantage compared to its classical counterpart DQN. The aim of this project is exploring the replacement of QA with the quantum approximate optimization algorithm (QAOA) and its recursive variant (RQAOA). We focus on the one-dimensional beam target steering control task based on the beam optics of the TT24-T4 transfer line at CERN. We perform and compare simulations of QA and QAOA, obtaining successful results for both QAOA and RQAOA, noticing that the performance of FERL using RQAOA for p=1 as a solver is very similar to the obtained through simulated QA for this scenario. The success of this experiment might lead the application of the QAOA solver to the hybrid actor-critic algorithm in development at CERN which allows to tackle control tasks using a continuous action space.

Related papers

Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Michael Schenk, Elías F. Combarro, Michele Grossi +4 more

Abstract Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic (A-C) scheme for continuous state-action spaces is developed based on the deep deterministic policy gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing (QA), both simulated and with D-Wave QA hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research. Among others, the hybrid A-C agent is evaluated on the actual electron beam line of the Advanced Wakefield Experiment (AWAKE).

arXiv:2209.11044Quantum Science and TechnologyFeb 5, 202419 citations

Free energy-based reinforcement learning using a quantum processor

Anna Levit, Daniel Crawford, Navid Ghadermarzy +3 more

Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks. In this paper, we introduce free energy-based reinforcement learning (FERL) as an application of quantum hardware. We propose a method for processing a quantum annealer's measured qubit spin configurations in approximating the free energy of a quantum Boltzmann machine (QBM). We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer. The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks.

arXiv:1706.00074May 29, 201716 citations

Reinforcement Learning via Replica Stacking of Quantum Measurements for the Training of Quantum Boltzmann Machines

Anna Levit, Daniel Crawford, Navid Ghadermarzy +3 more

Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks. In this paper, we introduce free-energy-based reinforcement learning (FERL) as an application of quantum hardware. We propose a method for processing a quantum annealer’s measured qubit spin configurations in approximating the free energy of a quantum Boltzmann machine (QBM). We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer. The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks.

Feb 15, 20180 citations

Quantum Reinforcement Learning

Daniel Crawford, Anna Levit, Navid Ghadermarzy +2 more

We investigate whether quantum annealers with select chip layouts can outperform classical computers in reinforcement learning tasks. We associate a transverse field Ising spin Hamiltonian with a layout of qubits similar to that of a deep Boltzmann machine (DBM) and use simulated quantum annealing (SQA) to numerically simulate quantum sampling from this system. We design a reinforcement learning algorithm in which the set of visible nodes representing the states and actions of an optimal policy are the first and last layers of the deep network. In absence of a transverse field, our simulations show that DBMs are trained more effectively than restricted Boltzmann machines (RBM) with the same number of nodes. We then develop a framework for training the network as a quantum Boltzmann machine (QBM) in the presence of a significant transverse field for reinforcement learning. This method also outperforms the reinforcement learning method that uses RBMs.

Dec 17, 20160 citations

Reinforcement learning assisted recursive QAOA

Yash J. Patel, Sofiène Jerbi, Thomas Bäck +1 more

In recent years, variational quantum algorithms such as the Quantum Approximation Optimization Algorithm (QAOA) have gained popularity as they provide the hope of using NISQ devices to tackle hard combinatorial optimization problems. It is, however, known that at low depth, certain locality constraints of QAOA limit its performance. To go beyond these limitations, a non-local variant of QAOA, namely recursive QAOA (RQAOA), was proposed to improve the quality of approximate solutions. The RQAOA has been studied comparatively less than QAOA, and it is less understood, for instance, for what family of instances it may fail to provide high-quality solutions. However, as we are tackling NP-hard problems (specifically, the Ising spin model), it is expected that RQAOA does fail, raising the question of designing even better quantum algorithms for combinatorial optimization. In this spirit, we identify and analyze cases where (depth-1) RQAOA fails and, based on this, propose a reinforcement learning enhanced RQAOA variant (RL-RQAOA) that improves upon RQAOA. We show that the performance of RL-RQAOA improves over RQAOA: RL-RQAOA is strictly better on these identified instances where RQAOA underperforms and is similarly performing on instances where RQAOA is near-optimal. Our work exemplifies the potentially beneficial synergy between reinforcement learning and quantum (inspired) optimization in the design of new, even better heuristics for complex problems.

arXiv:2207.06294EPJ Quantum TechnologyJan 17, 202429 citations

Quantum Machine Learning

Quantum Machine Learningby Jordi RIU This project applies reinforcement learning techniques to optimize a quantum algorithm.The upcoming quantum computers are capable of optimization tasks in an exponentially large parameter space.Having access to such a large optimization space potentially allows an advantage of Quantum methods over classical algorithms, but at the price of problems in the optimization process -just to name a few, the presence of local minima and the low convergence speeds-.This project explores the application of reinforcement learning techniques to control and optimize the operation of a quantum computer solving a hard combinatorial problem, namely the MaxCut problem.Our methods use a collection of quantum observables to feed the state of the agent, inspired on a similar classical approach to the problem applied to the antiferromagnetic Ising model lattice.Our results show how an agent using these observables can optimally control the operation of a quantum device to obtain quasi-optimal solutions when in combination with the Quantum Approximate Optimization Algorithm (QAOA).v

Between science and economicsApr 1, 20209 citations