The power of classical computers is still insufficient for deep reinforcement learning (DRL) problems which has large state space. Thanks to entanglement and superposition, quantum computers have a high computational power. The concept of using this high computing power to solve problems that would be difficult for classical computers is fairly common. In this study, a hybrid approach is proposed to take advantage of the benefits of quantum computers. Deep Q-Network (DQN) algorithm for DRL, optimization operations, and storage operations are performed on the classical computer side of this hybrid approach. In the quantum side, a variational quantum circuit (VQC) is proposed. The proposed method is applied to a navigation problem. The proposed approach is evaluated in terms of target success rate, collision (going out) rate. The proposed approach is compared to DRL solutions in the literature for the navigation problem with classical computers. According to the number of parameters used, the proposed approach appears to be successful. As a result, the performance of the proposed approach has been validated.
The power of classical computers is still insufficient for deep reinforcement learning (DRL) problems which has large state space. Thanks to entanglement and superposition, quantum computers have a high computational power. The concept of using this high computing power to solve problems that would be difficult for classical computers is fairly common. In this study, a hybrid approach is proposed to take advantage of the benefits of quantum computers. Deep Q-Network (DQN) algorithm for DRL, optimization operations, and storage operations are performed on the classical computer side of this hybrid approach. In the quantum side, a variational quantum circuit (VQC) is proposed. The proposed method is applied to a navigation problem. The proposed approach is evaluated in terms of target success rate, collision (going out) rate. The proposed approach is compared to DRL solutions in the literature for the navigation problem with classical computers. According to the number of parameters used, the proposed approach appears to be successful. As a result, the performance of the proposed approach has been validated.
Abstract: Quantum deep reinforcement learning is an integration of the principles of quantum computing with deep reinforcement learning in an effort to advance robot navigation tasks. Due to the explosion of the state and action spaces, traditionally DRL methods are limited in scalability and efficiency. QDRL, however, uses quantum superposition and entanglement for more efficient processing of big amounts of information-it enables robots to learn optimal policies for navigating complex environments. We introduce here a new QDRL framework in which quantum neural networks encode policy functions and value estimates. We demonstrate how quantum circuits can take advantage of multi-dimensional state representations in order to strengthen the power of a robot in changing its environment over time. Benchmark experiments on navigation tasks reflect phenomenal acceleration in learning performance along with better decision-making accuracy compared to classical DRL methods. We then consider integrating such quantum algorithms, as Grover's search, to improve exploration strategies in sparse reward settings. The experiments show that QDRL speeds up convergence rates and provides robots with the possibility to react more rapidly to unexpected obstacles. It discusses new applications for quantum computing of autonomous systems, potentially applied to robotics, autonomous vehicles, and other environments which require complex decision making. Some lines of work in the future are optimization of the resources needed for quantum computers and hybrid architectures for the resolution of classical problems in navigation
The development of quantum computational techniques has advanced greatly in recent years, parallel to the advancements in techniques for deep reinforcement learning. This work explores the potential for quantum computing to facilitate reinforcement learning problems. Quantum computing approaches offer important potential improvements in time and space complexity over traditional algorithms because of its ability to exploit the quantum phenomena of superposition and entanglement. Specifically, we investigate the use of quantum variational circuits, a form of quantum machine learning. We present our techniques for encoding classical data for a quantum variational circuit, we further explore pure and hybrid quantum algorithms for DQN and Double DQN. Our results indicate both hybrid and pure quantum variational circuit have the ability to solve reinforcement learning tasks with a smaller parameter space. These comparison are conducted with two OpenAI Gym environments: CartPole and Blackjack, The success of this work is indicative of a strong future relationship between quantum machine learning and deep reinforcement learning.
The development of quantum computational techniques has advanced greatly in recent years, parallel to the advancements in techniques for deep reinforcement learning. This work explores the potential for quantum computing to facilitate reinforcement learning problems. Quantum computing approaches offer important potential improvements in time and space complexity over traditional algorithms because of its ability to exploit the quantum phenomena of superposition and entanglement. Specifically, we investigate the use of quantum variational circuits, a form of quantum machine learning. We present our techniques for encoding classical data for a quantum variational circuit, we further explore pure and hybrid quantum algorithms for DQN and Double DQN. Our results indicate both hybrid and pure quantum variational circuit have the ability to solve reinforcement learning tasks with a smaller parameter space. These comparison are conducted with two OpenAI Gym environments: CartPole and Blackjack, The success of this work is indicative of a strong future relationship between quantum machine learning and deep reinforcement learning.
Abstract The task of collision-free navigation (CFN) of self-driving cars is an NP-hard problem usually tackled using Deep Reinforcement Learning (DRL). While DRL methods have proven to be effective, their implementation requires substantial computing resources and extended training periods to develop a robust agent. On the other hand, quantum reinforcement learning has recently demonstrated faster convergence and improved stability in simple, non-real-world environments. In this work, we propose Nav-Q, the first quantum-supported DRL algorithm for CFN of self-driving cars, that leverages quantum computation for improving the training performance without the requirement for onboard quantum hardware. Nav-Q is based on the actor-critic approach, where the critic is implemented using a hybrid quantum-classical algorithm suitable for near-term quantum devices. We assess the performance of Nav-Q using the CARLA driving simulator, a de facto standard benchmark for evaluating state-of-the-art DRL methods. Our empirical evaluations showcase that Nav-Q surpasses its classical counterpart in terms of training stability and, in certain instances, with respect to the convergence rate. Furthermore, we assess Nav-Q in relation to effective dimension, unveiling that the incorporation of a quantum component results in a model with greater descriptive power compared to classical baselines. Finally, we evaluate the performance of Nav-Q using noisy quantum simulation, observing that the quantum noise deteriorates the training performances but enhances the exploratory tendencies of the agent during training.
We utilize hybrid quantum deep reinforcement learning to learn navigation tasks for a simple, wheeled robot in simulated environments of increasing complexity. For this, we train parameterized quantum circuits (PQCs) with two different encoding strategies in a hybrid quantum-classical setup as well as a classical neural network baseline with the double deep Q network (DDQN) reinforcement learning algorithm. Quantum deep reinforcement learning (QDRL) has previously been studied in several relatively simple benchmark environments, mainly from the OpenAI gym suite. However, scaling behavior and applicability of QDRL to more demanding tasks closer to real-world problems e.g., from the robotics domain, have not been studied previously. Here, we show that quantum circuits in hybrid quantum-classic reinforcement learning setups are capable of learning optimal policies in multiple robotic navigation scenarios with notably fewer trainable parameters compared to a classical baseline. Across a large number of experimental configurations, we find that the employed quantum circuits outperform the classical neural network baselines when equating for the number of trainable parameters. Yet, the classical neural network consistently showed better results concerning training times and stability, with at least one order of magnitude of trainable parameters more than the best-performing quantum circuits. However, validating the robustness of the learning methods in a large and dynamic environment, we find that the classical baseline produces more stable and better performing policies overall. For the two encoding schemes, we observed better results for consecutively encoding the classical state vector on each qubit compared to encoding each component on a separate qubit. Our findings demonstrate that current hybrid quantum machine-learning approaches can be scaled to simple robotic problems while yielding sufficient results, at least in an idealized simulated setting, but there are yet open questions regarding the application to considerably more demanding tasks. We anticipate that our work will contribute to introducing quantum machine learning in general and quantum deep reinforcement learning in particular to more demanding problem domains and emphasize the importance of encoding techniques for classic data in hybrid quantum-classical settings.
With the evolution of technology, the need to solve complex computational problems like machine learning and deep learning has shot up. However, even the most powerful classical supercomputers find it difficult to execute these tasks. Advancements in quantum computing are leading researchers and tech-giants who strive for better quantum circuits to do machine learning tasks. Current works on Quantum Machine Learning (QML) ensure less memory consumption and reduced model parameters. However, it is strenuous to simulate the classical deep learning approach on existing quantum computing devices due to the inflexibility of Deep quantum circuits. Consequently, designing viable quantum algorithms for QML for noisy intermediate-scale quantum (NISQ) devices is essential. The proposed work aims to explore Variational Quantum Circuits (VQC) for Deep Q network-based Reinforcement Learning by remodeling the target network and experience replay into a representation of VQC. In addition, to the reduction in model parameters, quantum information encoding schemes are used to achieve better results than classical neural networks. VQCs are employed for policyselection and decision-making reinforcement learning by approximating the deep Q-value function with target network and experience replay.