International Journal of Quantum Information

QUANTUM COMPUTATION FOR ACTION SELECTION USING REINFORCEMENT LEARNING

C. L. CHEN, D. Y. DONG, Zonghai Chen

Published Dec 1, 2006• 26 citations

Abstract

This paper proposes a novel action selection method based on quantum computation and reinforcement learning (RL). Inspired by the advantages of quantum computation, the state/action in a RL system is represented with quantum superposition state. The probability of action eigenvalue is denoted by probability amplitude, which is updated according to rewards. And the action selection is carried out by observing quantum state according to collapse postulate of quantum measurement. The results of simulated experiments show that quantum computation can be effectively used to action selection and decision making through speeding up learning. This method also makes a good tradeoff between exploration and exploitation for RL using probability characteristics of quantum theory.

DOI: 10.1142/s0219749906002419

Related papers

Quantum Reinforcement Learning

Daoyi Dong, Chunlin Chen, Han‐Xiong Li +1 more

The key approaches for machine learning, particularly learning in unknown probabilistic environments, are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum parallelism, a framework of a value-updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state, and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is updated in parallel according to rewards. Some related characteristics of QRL such as convergence, optimality, and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speedup learning through the quantum parallelism. To evaluate the performance and practicability of QRL, several simulated experiments are given, and the results demonstrate the effectiveness and superiority of the QRL algorithm for some complex problems. This paper is also an effective exploration on the application of quantum computation to artificial intelligence.

arXiv:0810.3828IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Aug 4, 2008410 citations

A Novel Multiagent Reinforcement Learning Algorithm Combination with Quantum Computation

Xiangping Meng, Yu Chen, Yuzhen Pi +1 more

A novel learning policy in multiagent reinforcement learning is presented, trying to find another tradeoff of exploration and exploitation efficiently, which is different from traditional greedy or softmax action selection method. The state and action of multiagent are represented with quantum superposition state, and probability amplitude is used to denote the probability of an action. Quantum search algorithm is adopted in multiagent action selection. The experiment results show that the new algorithm is effective and can help multiagent learn faster. This combination of quantum computing with multiagent reinforcement learning is an attempt, and the idea possibly brings more researches in multiagent reinforcement learning.

World Congress on Intelligent Control and AutomationJan 1, 20062 citations

Superposition-Inspired Reinforcement Learning and Quantum Reinforcement Learning

Chunlin Chen, Daoyi Dong

According to the existing problems in RL area, such as low learning speed and tradeoff between exploration and exploitation, SIRL and QRL methods are introduced based on the theory of RL and quantum computation in this chapter, which follows the developing roadmap from the superposition-inspired methods to the RL methods in quantum systems. Just as simulated annealing algorithm comes from mimicking the physical annealing process, quantum characteristics also broaden our mind and provide alternative approaches to novel RL methods. In this chapter, SIRL method emphasizes the exploration policy and uses a probabilistic action selection method that is inspired by the state superposition principle and collapse postulate. The experiments, which include a puzzle problem and a mobile robot navigation problem, demanstrate the effectiveness of SIRL algorithm and show that it is superior to basic TD algorithm with -greedy policy. As for QRL, the state/action value is represented with quantum superposition state and the action selection is carried out by observing quantum state according to quantum collapse postulate, which means a QRL system is designed for the real quantum system although it can also be simulated on a traditional computer. The results of simulated experiments verified its feasibility and effectiveness with two examples: Prisoner's Dilemma and the control of a five-qubit system. The contents presented in this chapter are mainly the basic ideas and methods related to the combination of RL theory and quantum computation. More theoretic research and applictions are to be investigated in the future.

Jan 1, 20088 citations

Quantum-inspired reinforcement learning for decision-making of Markovian state transition

Daoyi Dong, Chunlin Chen

A novel quantum-inspired reinforcement learning (QiRL) algorithm is proposed for decision-making of Markovian state transition. The QiRL algorithm adopts a probabilistic action selection policy to better balance the tradeoff between exploration and exploitation, which is inspired by the collapse phenomenon in quantum measurement. Several simulated experiments of Markovian state transition demonstrate that QiRL is more robust to learning rates and initial states than traditional reinforcement learning. The QiRL approach provides an effective method for complex decision-making problems.

International Conference on Intelligent Systems and Knowledge Engineering (ISKE)Nov 1, 20104 citations

Cooperative multi-agent reinforcement learning based on quantum computing

Xiangping Meng

Due to the interactions among the agents in the cooperative multi-agent systems,multi-agent learning problem complexity can rise rapidly with the number of agents or their behavioral sophistication.In order to converge to desirable equilibrium,agents generally require sufficient exploration of strategy space and coordinate their policies to achieve optimal equilibrium.A novel cooperative multi-agent learning method is proposed based on quantum theory and reinforcement learning.This method not only coordinates agents’ behaviors using quantum entanglement and helps agents make action selection under quantum superposition,but also adopts Grover’s searching algorithm which can speed up learning.This method also makes a good tradeoff between exploration and exploitation using probability characteristics of quantum theory.The results of simulated experiments show that quantum theory can be effectively used to cooperative multi-agent reinforcement learning.

Computer Engineering and Applications JournalJan 1, 20080 citations

Quantum-Inspired Algorithms for Reinforcement Learning

Jincheng Zhang

This paper investigates the application of quantum-inspired algorithms to the field of reinforcement learning (RL). Traditional RL methods often struggle with exploration challenges and can be computationally expensive, particularly in complex environments. We propose a novel RL algorithm that leverages concepts from quantum mechanics, specifically superposition and entanglement, to address these limitations. The core idea is to represent states, actions, and rewards using quantum-inspired representations, allowing the agent to explore the environment more efficiently and potentially converge to optimal policies faster. This work presents a detailed formulation of the algorithm, including the mathematical framework and key considerations for implementation. The potential benefits of this approach, including improved exploration and reduced computational complexity, are discussed. The algorithm is designed to iteratively refine its policy based on simulated quantum measurements, aiming to minimize the expected return. The exploration strategy is guided by the inherent probabilistic nature of quantum states, leading to a more diverse and potentially faster learning process. Further research is warranted to fully explore the capabilities and limitations of this approach across various RL benchmark problems.

Sep 1, 20260 citations