IFAC-PapersOnLine

Quantum reinforcement learning method and application based on value function

Yi-Pei Liu, Qing‐Shan Jia, Xu Wang

Published Jan 1, 2022• 8 citations

Abstract

The Multi-Arm bandit(MAB) problem is a classical problem in the field of reinforcement learning with only one state. The Grid problem is a multi-state problem for reinforcement learning. In this work, we focus on how to combine the classical value function method to quantum computation, and we propose three novel quantum reinforcement learning(QRL) algorithms for the MAB problem and one novel QRL algorithm, which is combined with the quantum random walk and Grover algorithm, for the Grid problem. From the experiments, the learning process is speed-up by combining the value function with quantum computation.

DOI: 10.1016/j.ifacol.2022.08.061

Related papers

Quantum Reinforcement Learning for Multi-Armed Bandits

Yi-Pei Liu, Kuo Li, Xi Cao +2 more

This work focuses on the multi-armed bandits (MAB) problem and proposes a quantum reinforcement learning (RL) algorithm for action selection. Existing quantum RL algorithms generally assume that some prior information about the optimal action is known, and initial probability is set unequally. Our algorithm can be executed with equal initial probability on each action, and can greatly accelerate the learning process.

Chinese Control Conference (CCC)Jul 25, 20221 citation

Quantum Reinforcement Learning with Quantum World Model

Peigen Zeng, Ying He, Fei Richard Yu +1 more

Quantum reinforcement learning (QRL) can outperform classical reinforcement learning (RL) by utilizing quantum parallel theory and quantum phenomena such as superposition and entanglement. Although some excellent work has been done on QRL, most existing works either fail to show the exponential advantage of quantum computation over classical computation in terms of performance or are too demanding on quantum devices. In this paper, we provide a novel perspective on combining quantum computing and RL with faster convergence speed and relatively relaxed demands on quantum devices. Specifically, we propose a method to construct a world model with quantum circuit that allows it to interact in a quantum way. In addition, we use Grover's algorithm to efficiently extract high-value information from the quantum world model. Extensive simulation results show that the proposed method can have superior performance compared to classical RL algorithms.

GlobecomDec 4, 20231 citation

Quantum Reinforcement Learning

Daoyi Dong, Chunlin Chen, Han‐Xiong Li +1 more

The key approaches for machine learning, particularly learning in unknown probabilistic environments, are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum parallelism, a framework of a value-updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state, and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is updated in parallel according to rewards. Some related characteristics of QRL such as convergence, optimality, and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speedup learning through the quantum parallelism. To evaluate the performance and practicability of QRL, several simulated experiments are given, and the results demonstrate the effectiveness and superiority of the QRL algorithm for some complex problems. This paper is also an effective exploration on the application of quantum computation to artificial intelligence.

arXiv:0810.3828IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Aug 4, 2008410 citations

An Introduction to Quantum Reinforcement Learning (QRL)

Samuel Yen-Chi Chen

Recent advancements in quantum computing (QC) and machine learning (ML) have sparked considerable interest in the integration of these two cutting-edge fields. Among the various ML techniques, reinforcement learning (RL) stands out for its ability to address complex sequential decision-making problems. RL has already demonstrated substantial success in the classical ML community. Now, the emerging field of Quantum Reinforcement Learning (QRL) seeks to enhance RL algorithms by incorporating principles from quantum computing. This paper offers an introduction to this exciting area for the broader AI and ML community.

arXiv:2409.05846International Conference on Information and Communication Technology Convergence (ICTC)Oct 16, 202414 citations

Quantum Reinforcement Learning Algorithm

Jincheng Zhang

Quantum Reinforcement Learning (QRL) represents a novel approach to reinforcement learning that leverages the unique capabilities of quantum computing to enhance learning efficiency and enable the exploration of complex behavioral rules. This paper explores the theoretical foundations, design principles, and potential applications of a QRL algorithm, focusing on its ability to learn intricate patterns and make superior decisions within challenging environments. The core mechanism centers around quantum superposition and entanglement, strategically employed to represent the state space and guide the agent's exploration, resulting in improved learning speed and robustness. We present a detailed framework for implementing this algorithm, discussing its potential impact on diverse reinforcement learning scenarios, particularly those encountered in edge applications. The paper concludes with a discussion of ongoing research directions and future prospects for QRL.

Sep 3, 20260 citations

Quantum Policy Iteration via Amplitude Estimation and Grover Search -- Towards Quantum Advantage for Reinforcement Learning

Simon M. Wiedemann, Daniel M. Hein, Steffen Udluft +1 more

We present a full implementation and simulation of a novel quantum reinforcement learning method. Our work is a detailed and formal proof of concept for how quantum algorithms can be used to solve reinforcement learning problems and shows that, given access to error-free, efficient quantum realizations of the agent and environment, quantum methods can yield provable improvements over classical Monte-Carlo based methods in terms of sample complexity. Our approach shows in detail how to combine amplitude estimation and Grover search into a policy evaluation and improvement scheme. We first develop quantum policy evaluation (QPE) which is quadratically more efficient compared to an analogous classical Monte Carlo estimation and is based on a quantum mechanical realization of a finite Markov decision process (MDP). Building on QPE, we derive a quantum policy iteration that repeatedly improves an initial policy using Grover search until the optimum is reached. Finally, we present an implementation of our algorithm for a two-armed bandit MDP which we then simulate.

arXiv:2206.04741Jun 9, 202210 citations