Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.
Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.
This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables arelog(n)-local, the gradients vanish polynomially with the number of qubits providedO(logn)depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.
Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.
Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an ε-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly or even outperform typical classical neural networks used in standard benchmarking environments and in quantum control, using only a fraction of the parameters. Moreover, we study the Barren Plateau phenomenon in quantum policy gradients using the Fisher Information Matrix spectrum.
Abstract Parameterized quantum circuits (PQCs) are crucial for quantum machine learning and circuit synthesis, enabling the practical implementation of complex quantum tasks. However, PQC learning has been largely confined to classical optimization methods, which suffer from issues like gradient vanishing.In this work, we introduce a nested optimization model, a hybrid approach that leverages quantum gradients to improve PQC learning for arbitrary polynomial-type cost functions. The proposed approach decomposes the learning problem into multiple subproblems, each aimed at learning the state identified by the quantum gradient using current circuit synthesis methods.By leveraging quantum gradients, the optimization procedure is free of a specific type of gradient vanishing, such as unfavorable local stationary points and barren plateaus.Specifically, with the guidance of quantum gradient, unfavorable local stationary points can be overcome by increasing the depth of the ansatz, while barren plateaus can be mitigated by constraining the optimization region.Numerically, we demonstrate the feasibility of the approach on two tasks: the Max-Cut problem and polynomial optimization. Additionally, we analyze the overheads introduced by this approach, which are primarily polynomial in the number of qubits in the system and can be controlled by adjusting the learning rate.The method can generate circuits while avoiding gradient vanishing, thus effectively optimizing the cost function.From the perspective of quantum algorithms, our model improves quantum optimization for polynomial-type cost functions, addressing the challenge of exponential sample complexity growth.
Whether parameterized quantum circuits (PQCs) can be systematically constructed to be both trainable and expressive remains an open question. Highly expressive PQCs often exhibit barren plateaus, while several trainable alternatives admit efficient classical simulation. We address this question by deriving a finite-sample, dimension-independent concentration bound for estimating the variance of a PQC cost function, yielding explicit trainability guarantees. Across commonly used ansätze, we observe an anticorrelation between trainability and expressibility, consistent with theoretical insights. Building on this observation, we propose a property-based ansatz-search framework for identifying circuits that combine trainability and expressibility. We demonstrate its practical viability on a real quantum computer and apply it to variational quantum algorithms. We identify quantum neural network ansätze with improved effective dimension using over 6× fewer parameters, and for VQE on H2 we achieve UCCSD-like accuracy at substantially reduced circuit complexity.
This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.