In recent years, various limitations of conventional supervised learning have been highlighted, leading to the emergence of reinforcement learning -- and, further, quantum reinforcement learning that exploits quantum resources such as entanglement and superposition -- as promising alternatives. Among the various reinforcement learning methodologies, gradient-based approaches, particularly policy gradient methods, are considered to have many benefits. Moreover, in the quantum regime, they also have a profit in that they can be readily implemented through parameterized quantum circuits (PQCs). From the perspective of learning, two indicators can be regarded as most crucial: expressivity and, for gradient-based methods, trainability. While a number of attempts have been made to quantify the expressivity and trainability of PQCs, clear efforts in the context of reinforcement learning have so far been lacking. Therefore, in this study, we newly define the notion of expressivity suited to reinforcement learning and demonstrate that the mutual information between action distribution and reward-signal distribution can, in certain respects, indicate information about both expressivity and trainability. Such research is valuable in that it provides an easy criterion for choosing among various PQCs employed in reinforcement learning, and further, enables the indirect estimation of learning progress even in black-box settings where the agent's achievement aligned with the episodes cannot be explicitly evaluated.
In recent years, various limitations of conventional supervised learning have been highlighted, leading to the emergence of reinforcement learning -- and, further, quantum reinforcement learning that exploits quantum resources such as entanglement and superposition -- as promising alternatives. Among the various reinforcement learning methodologies, gradient-based approaches, particularly policy gradient methods, are considered to have many benefits. Moreover, in the quantum regime, they also have a profit in that they can be readily implemented through parameterized quantum circuits (PQCs). From the perspective of learning, two indicators can be regarded as most crucial: expressivity and, for gradient-based methods, trainability. While a number of attempts have been made to quantify the expressivity and trainability of PQCs, clear efforts in the context of reinforcement learning have so far been lacking. Therefore, in this study, we newly define the notion of expressivity suited to reinforcement learning and demonstrate that the mutual information between action distribution and reward-signal distribution can, in certain respects, indicate information about both expressivity and trainability. Such research is valuable in that it provides an easy criterion for choosing among various PQCs employed in reinforcement learning, and further, enables the indirect estimation of learning progress even in black-box settings where the agent's achievement aligned with the episodes cannot be explicitly evaluated.
This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.
Variational quantum circuits have recently gained popularity as quantum machine learning models. While considerable effort has been invested to train them in supervised and unsupervised learning settings, relatively little attention has been given to their potential use in reinforcement learning. In this work, we leverage the understanding of quantum policy gradient algorithms in a number of ways. First, we investigate how to construct and train reinforcement learning policies based on variational quantum circuits. We propose several designs for quantum policies, provide their learning algorithms, and test their performance on classical benchmarking environments. Second, we show the existence of task environments with a provable separation in performance between quantum learning agents and any polynomial-time classical learner, conditioned on the widely-believed classical hardness of the discrete logarithm problem. We also consider more natural settings, in which we show an empirical quantum advantage of our quantum policies over standard neural-network policies. Our results constitute a first step towards establishing a practical near-term quantum advantage in a reinforcement learning setting. Additionally, we believe that some of our design choices for variational quantum policies may also be beneficial to other models based on variational quantum circuits, such as quantum classifiers and quantum regression models.
Training quantum neural networks (QNNs) on near-term hardware remains hampered by two compounding difficulties: the exponential vanishing of gradient variance known as the barren plateau, and the O(L⋅2n) time and memory cost of differentiating through an n-qubit, L-layer circuit. We propose RLQ-Grad, a reinforcement-learning-based optimizer in which a classical policy πφ (a spectrally-normalized PPO agent) learns to propose parameter updates directly, conditioned on the QNN's current parameters, loss, accuracy, and previous update. Because the surrogate gradient is emitted by a classical network rather than obtained by differentiating through the unitary U(θ), its variance is not constrained by the barren plateau concentration bound, and its cost scales with the number of trainable parameters rather than the Hilbert-space dimension. We prove these properties formally and verify them on a hardware-efficient ansatz across four supervised benchmarks with up to n=20 qubits. RLQ-Grad preserves a near-flat gradient-variance curve where backpropagation, parameter-shift, and adjoint differentiation decay by 1 to 2 orders of magnitude. Accounting for the full training pipeline (PPO rollouts, actor-critic updates, and optimizer states), RLQ-Grad needs under 2 MB of memory and runs 2490×, 7876×, and 673× faster per iteration than these three methods at n=20. It improves top-1 accuracy by up to +10% over gradient-based baselines on circuits of up to 12 qubits, and matches dedicated barren plateau mitigation methods on CIFAR-10 at 14 to 20 qubits, where evolutionary and gradient-free optimizers collapse to chance.
Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.
This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables arelog(n)-local, the gradients vanish polynomially with the number of qubits providedO(logn)depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.
This paper investigates the application of quantum-inspired algorithms to the field of reinforcement learning (RL). Traditional RL methods often struggle with exploration challenges and can be computationally expensive, particularly in complex environments. We propose a novel RL algorithm that leverages concepts from quantum mechanics, specifically superposition and entanglement, to address these limitations. The core idea is to represent states, actions, and rewards using quantum-inspired representations, allowing the agent to explore the environment more efficiently and potentially converge to optimal policies faster. This work presents a detailed formulation of the algorithm, including the mathematical framework and key considerations for implementation. The potential benefits of this approach, including improved exploration and reduced computational complexity, are discussed. The algorithm is designed to iteratively refine its policy based on simulated quantum measurements, aiming to minimize the expected return. The exploration strategy is guided by the inherent probabilistic nature of quantum states, leading to a more diverse and potentially faster learning process. Further research is warranted to fully explore the capabilities and limitations of this approach across various RL benchmark problems.