Artificial intelligence has permeated all aspects of our lives today. However, to make AI behave like real AI, the critical bottleneck lies in the speed of computing. Quantum computers employ the peculiar and unique properties of quantum states such as superposition, entanglement, and interference to process information in ways that classical computers cannot. As a new paradigm of computation, quantum computers are capable of performing tasks intractable for classical processors, thus providing a quantum leap in AI research and making the development of real AI a possibility. In this regard, quantum machine learning not only enhances the classical machine learning approach but more importantly it provides an avenue to explore new machine learning models that have no classical counterparts. The qubit-based quantum computers cannot naturally represent the continuous variables commonly used in machine learning, since the measurement outputs of qubit-based circuits are generally discrete. Therefore, a continuous-variable (CV) quantum architecture based on a photonic quantum computing model is selected for our study. In this work, we employ machine learning and optimization to create photonic quantum circuits that can solve the contextual multi-armed bandit problem, a problem in the domain of reinforcement learning, which demonstrates that quantum reinforcement learning algorithms can be learned by a quantum device.
Artificial intelligence has permeated all aspects of our lives today. However, to make AI behave like real AI, the critical bottleneck lies in the speed of computing. Quantum computers employ the peculiar and unique properties of quantum states such as superposition, entanglement, and interference to process information in ways that classical computers cannot. As a new paradigm of computation, quantum computers are capable of performing tasks intractable for classical processors, thus providing a quantum leap in AI research and making the development of real AI a possibility. In this regard, quantum machine learning not only enhances the classical machine learning approach but more importantly it provides an avenue to explore new machine learning models that have no classical counterparts. The qubit-based quantum computers cannot naturally represent the continuous variables commonly used in machine learning, since the measurement outputs of qubit-based circuits are generally discrete. Therefore, a continuous-variable (CV) quantum architecture based on a photonic quantum computing model is selected for our study. In this work, we employ machine learning and optimization to create photonic quantum circuits that can solve the contextual multi-armed bandit problem, a problem in the domain of reinforcement learning, which demonstrates that quantum reinforcement learning algorithms can be learned by a quantum device.
The advantage of quantum computers over classical computers fuels the recent trend of developing machine learning algorithms on quantum computers, which can potentially lead to breakthroughs and new learning models in this area. The aim of our study is to explore deep quantum reinforcement learning (RL) on photonic quantum computers, which can process information stored in the quantum states of light. These quantum computers can naturally represent continuous variables, making them an ideal platform to create quantum versions of neural networks. Using quantum photonic circuits, we implement Q learning and actor-critic algorithms with multilayer quantum neural networks and test them in the grid world environment. Our experiments show that 1) these quantum algorithms can solve the RL problem and 2) compared to one layer, using three layer quantum networks improves the learning of both algorithms in terms of rewards collected. In summary, our findings suggest that having more layers in deep quantum RL can enhance the learning outcome.
Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example of reinforcement learning. However, the bandit problem deals with a static environment where the agent's action does not influence the reward probabilities. This study aims to extend the conventional approach to a more general multi-agent reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of agents' actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, discontinuous bandit Q-learning, in view of a potential photonic implementation. Here, state-action pairs in the environment are regarded as slot machines in the context of the bandit problem and an updated amount of Q-value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a multi-agent architecture in which agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state-action pair selections among agents. We demonstrate that multi-agent reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.
Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example in reinforcement learning. However, the bandit problem deals with a static environment where the agent’s actions do not influence the reward probabilities. This study aims to extend the conventional approach to a more general type of parallel reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of the agent’s actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, a modified bandit Q-learning method, in view of a potential photonic implementation. Here, state–action pairs in the environment are regarded as slot machines in the context of the bandit problem and a change in Q -value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a parallel architecture in which multiple agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state–action pair selections among agents. We demonstrate that parallel reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.
The convergence of machine learning (ML) and quantum computing is quickly becoming a game-changing new frontier in computational science. The most intriguing area of overlap may be quantum optimization, which seeks to reduce the exponentially increasing complexity and computational cost of ML algorithms. As they grow larger and more complicated, traditional optimization methods, particularly in high-dimensional and non-convex space, will inevitably hit performance limits. Quantum optimization is the whole new paradigm that leverages quantum mechanical effects like superposition, entanglement, and tunnelling to explore solution spaces more efficiently. The chapter outlines the theory, practice, and potential of hybrid quantum-classical optimization algorithms for machine learning. The most brilliant research direction in the current NISQ (noisy intermediate-scale quantum) era is mixed systems leveraging classical computing capability augmented by quantum capacity. In such designs, computationally intensive subroutines like optimization for global optima or escape from local minima are left to quantum processors and data preprocessing, iterative refinement, and model testing are performed by classical processors. We start by defining the fundamental principles of quantum computing and acquainting readers with the fundamentals like qubits, quantum gates, and quantum circuits. These are the building blocks for quantum algorithms, which hopefully can be utilized for optimization, for example, the QAOA (quantum approximate optimization algorithm), the VQE (variational quantum Eigensolver), and quantum annealing. They all build to optimize the problems in some way differently: QAOA through parameterized quantum circuits, VQE through approximations of ground states of the Hamiltonians, and quantum annealing through quantum evolution to low-energy solutions. We then form a loop between these quantum algorithms and regular machine learning tasks. Optimization impacts virtually all ML phases, from training neural networks, where one reduces loss functions; feature selection, where one aims to discover the most relevant subset of input variables; and hyperparameter optimization, which often involves expensive search procedures over extremely large combinatorial spaces. Quantum optimization techniques can therefore, in principle, accelerate such procedures through computational complexity reduction and enhancing solutions over challenging landscapes. Realistic simulations and case studies emphasize the degree to which hybrid quantum-classical algorithms have been employed in solving ML problems, that is, QSVMs, quantum-enhanced clustering, and quantum reinforcement learning. Despite their experimental nature, these applications already show encouraging results in terms of improved performance metrics as well as reduction in time-to-solution compared to classical-only solutions. Notwithstanding that, this new field is not bereft of real challenges. Quantum hardware suffers from hard limits on qubit fidelity, decoherence, and scalability in the current times. Furthermore, quantum circuit integration into broad ML pipelines requires novel architectures and software platforms. As a counter to those issues, this chapter surveys recent advances in quantum programming languages, simulators, and hybrid platforms like PennyLane, Qiskit, and TensorFlow Quantum. We conclude by looking to the future possibility that quantum optimization will revolutionize the machine learning landscape. As hardware improves and algorithmic complexity increases, the possibility of quantum-powered learning systems solving hard problems becomes increasingly accessible. From speeding up deep network training to enabling new types of unsupervised learning, quantum optimization holds the potential to recast computational tractability in machine learning. This chapter is an all-encompassing introduction and state-of-the-art work for students, researchers, and practitioners interested in using quantum optimization for practical ML applications. It sets the foundation for understanding how to pursue hybrid approaches (how they may be applied today in the real world and bracing oneself for disruption in the future).
In the realm of artificial intelligence, deep rein-forcement learning (RL) agents struggle with generalizability and require substantial computational resources, unlike humans who easily adapt and generalize across tasks. To address these challenges, we introduce Quantum Rainbow, a hybrid algorithm that leverages the neural mechanisms of human decision-making and the efficiency of quantum computing. Quantum Rainbow combines variational quantum circuits with the Rainbow Deep Q-Network (DQN) model to create a novel approach in rein-forcement learning that integrates quantum principles into deep learning paradigms. We evaluate our model using behavioral experiments through the Iowa Gambling Task and 4-Armed Bandit Task. Our investigations reveal a significant relationship between the architecture of quantum circuits and the performance of quantum RL agents. Specifically, using causal discovery methods, we demonstrate the critical role of quantum entanglement in enhancing model performance. These findings not only show promising results but also pave the way for future explorations into optimizing quantum circuit architectures for reinforcement learning applications. This study underscores the potential of quantum-enhanced algorithms to achieve “quantum advantage” by addressing fundamental limitations in conventional deep RL methods.
The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep Q-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.