Deep Reinforcement Learning is emerging as a promising approach for the continuous control task of robotic arm movement. However, the challenges of learning robust and versatile control capabilities are still far from being resolved for real-world applications, mainly because of two common issues of this learning paradigm: the exploration strategy and the slow learning speed, sometimes known as "the curse of dimensionality". This work aims at exploring and assessing the advantages of the application of Quantum Computing to one of the state-of-art Reinforcement Learning techniques for continuous control - namely Soft Actor-Critic. Specifically, the performance of a Variational Quantum Soft Actor-Critic on the movement of a virtual robotic arm has been investigated by means of digital simulations of quantum circuits. A quantum advantage over the classical algorithm has been found in terms of a significant decrease in the amount of required parameters for satisfactory model training, paving the way for further promising developments.
Deep Reinforcement Learning is emerging as a promising approach for the continuous control task of robotic arm movement. However, the challenges of learning robust and versatile control capabilities are still far from being resolved for real-world applications, mainly because of two common issues of this learning paradigm: the exploration strategy and the slow learning speed, sometimes known as "the curse of dimensionality". This work aims at exploring and assessing the advantages of the application of Quantum Computing to one of the state-of-art Reinforcement Learning techniques for continuous control - namely Soft Actor-Critic. Specifically, the performance of a Variational Quantum Soft Actor-Critic on the movement of a virtual robotic arm has been investigated by means of digital simulations of quantum circuits. A quantum advantage over the classical algorithm has been found in terms of a significant decrease in the amount of required parameters for satisfactory model training, paving the way for further promising developments.
Quantum computing has a superior advantage in tackling specific problems, such as integer factorization and Simon's problem. For more general tasks in machine learning, by applying variational quantum circuits, more and more quantum algorithms have been proposed recently, especially in supervised learning and unsupervised learning. However, little work has been done in reinforcement learning, arguably more important and challenging. Previous work in quantum reinforcement learning mainly focuses on discrete control tasks where the action space is discrete. In this work, we develop a quantum reinforcement learning algorithm based on soft actor-critic -- one of the state-of-the-art methods for continuous control. Specifically, we use a hybrid quantum-classical policy network consisting of a variational quantum circuit and a classical artificial neural network. Tested in a standard reinforcement learning benchmark, we show that this quantum version of soft actor-critic is comparable with the original soft actor-critic, using much less adjustable parameters. Furthermore, we analyze the effect of different hyper-parameters and policy network architectures, pointing out the importance of architecture design for quantum reinforcement learning.
Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.
The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior than classical agents with similar model sizes and architectures.
Classical reinforcement learning (RL) methods often struggle in complex, high-dimensional environments because of their extensive parameter requirements and challenges posed by stochastic, non-deterministic settings. This study introduces quantum deep reinforcement learning (QDRL) to train humanoid agents efficiently. While previous quantum RL models focused on smaller environments, such as wheeled robots and robotic arms, our work pioneers the application of QDRL to humanoid robotics, specifically in environments with substantial observation and action spaces, such as MuJoCo's Humanoid-v4 and Walker2d-v4. Using parameterized quantum circuits, we explored a hybrid quantum-classical setup to directly navigate high-dimensional state spaces, bypassing traditional mapping and planning. By integrating quantum computing with deep RL, we aim to develop models that can efficiently learn complex navigation tasks in humanoid robots. We evaluated the performance of the Soft Actor-Critic (SAC) in classical RL against its quantum implementation. The results show that the quantum SAC achieves an 8% higher average return (246.40) than the classical SAC (228.36) after 92% fewer steps, highlighting the accelerated learning potential of quantum computing in RL tasks.
The convergence of quantum computing and robotics represents a paradigm shift in autonomous systems design, promising to overcome fundamental computational limitations that have constrained classical robotic capabilities. This comprehensive review surveys the rapidly evolving field of quantum machine learning (QML) for robotics, systematically analyzing 107 peer-reviewed contributions spanning quantum algorithms, machine learning techniques, and their robotic applications. We provide a detailed historical perspective on the field's development, from the early theoretical foundations established by Grover's search algorithm and quantum reinforcement learning proposals to current experimental implementations on noisy intermediate-scale quantum (NISQ) devices. The review systematically examines five major application domains: (1) quantum reinforcement learning for navigation and control, (2) quantum search and path planning algorithms, (3) quantum-enhanced localization and mapping, (4) quantum optimization for inverse kinematics and task allocation, and (5) multi-robot coordination and swarm intelligence. For each domain, we present detailed algorithmic formulations, computational complexity analyses, and critical assessments of experimental results. Our analysis reveals that while provable quantum advantage in robotics remains largely prospective, recent demonstrations show promising results: quantum deep reinforcement learning achieving 40\% parameter reduction in navigation tasks, Grover-based kinematic optimization demonstrating up to 93x speedups, and quantum annealing providing practical solutions for multi-robot routing. We identify key technical challenges, including barren plateaus in variational circuits, noise-induced performance degradation, and the quantum-classical integration bottleneck. Finally, we outline a strategic roadmap for quantum robotics research, distinguishing near-term opportunities in hybrid algorithms from long-term goals requiring fault-tolerant quantum computation. This paper serves as both an authoritative technical reference and a comprehensive guide for researchers working at the intersection of quantum computing and autonomous systems.
The convergence of quantum computing and robotics represents a paradigm shift in autonomous systems design, promising to overcome fundamental computational limitations that have constrained classical robotic capabilities. This comprehensive review surveys the rapidly evolving field of quantum machine learning (QML) for robotics, systematically analyzing 107 peer-reviewed contributions spanning quantum algorithms, machine learning techniques, and their robotic applications. We provide a detailed historical perspective on the field's development, from the early theoretical foundations established by Grover's search algorithm and quantum reinforcement learning proposals to current experimental implementations on noisy intermediate-scale quantum (NISQ) devices. The review systematically examines five major application domains: (1) quantum reinforcement learning for navigation and control, (2) quantum search and path planning algorithms, (3) quantum-enhanced localization and mapping, (4) quantum optimization for inverse kinematics and task allocation, and (5) multi-robot coordination and swarm intelligence. For each domain, we present detailed algorithmic formulations, computational complexity analyses, and critical assessments of experimental results. Our analysis reveals that while provable quantum advantage in robotics remains largely prospective, recent demonstrations show promising results: quantum deep reinforcement learning achieving 40\% parameter reduction in navigation tasks, Grover-based kinematic optimization demonstrating up to 93x speedups, and quantum annealing providing practical solutions for multi-robot routing. We identify key technical challenges including barren plateaus in variational circuits, noise-induced performance degradation, and the quantum-classical integration bottleneck. Finally, we outline a strategic roadmap for quantum robotics research, distinguishing near-term opportunities in hybrid algorithms from long-term goals requiring fault-tolerant quantum computation. This paper serves as both an authoritative technical reference and a comprehensive guide for researchers working at the intersection of quantum computing and autonomous systems.