Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

IEEE International Conference on Quantum Computing and Engineering (QCE)

Trainability Issues in Quantum Policy Gradients with Softmax Activations

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Published Sep 15, 2024• 0 citations
Publisher

Abstract

This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables are$\log(n)$-local, the gradients vanish polynomially with the number of qubits provided$\mathrm{O}(\log n)$depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.

DOI
10.1109/qce60285.2024.10302

Related papers

Trainability issues in quantum policy gradients

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.

arXiv:2406.09614•Machine Learning Science and Technology•Jul 26, 2024•3 citations

Data re-uploading and expressivity/trainability trade-off on quantum policy gradients

Bernardo Manuel Barbosa Soares

This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.

•Dec 1, 2024•0 citations

Tight and Efficient Gradient Bounds for Parameterized Quantum Circuits

Alistair Letcher, Stefan Woerner, Christa Zoufal

The training of a parameterized model largely depends on the landscape of the underlying loss function. In particular, vanishing gradients are a central bottleneck in the scalability of variational quantum algorithms (VQAs), and are known to arise in various ways. However, a caveat of most existing gradient bound results is the requirement of t-design circuit assumptions that are typically not satisfied in practice. In this work, we loosen these assumptions altogether and derive tight upper and lower bounds on loss and gradient concentration for a large class of parameterized quantum circuits and arbitrary observables, which are significantly stronger than prior work. Moreover, we show that these bounds, as well as the variance of the loss itself, can be estimated efficiently and classically-providing practical tools to study the loss landscapes of VQA models, including verifying whether or not a circuit/observable induces barren plateaus. In particular, our results can readily be leveraged to rule out barren plateaus for a realistic class of ansätze and mixed observables, namely, observables containing a non-vanishing local term. This insight has direct implications for hybrid Quantum Generative Adversarial Networks (qGANs). We prove that designing the discriminator appropriately leads to 1-local weights that stay constant in the number of qubits, regardless of discriminator depth. This implies that qGANs with appropriately chosen generators do not suffer from barren plateaus even at scale-making them a promising candidate for applications in generative quantum machine learning. We demonstrate this result by training a qGAN to learn a 2D mixture of Gaussian distributions with up to 16 qubits, and provide numerical evidence that global contributions to the gradient, while initially exponentially small, may kick in substantially over the course of training.

arXiv:2309.12681•Quantum•Sep 25, 2024•28 citations

Policy Gradients using Variational Quantum Circuits

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an $ε$-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly or even outperform typical classical neural networks used in standard benchmarking environments and in quantum control, using only a fraction of the parameters. Moreover, we study the Barren Plateau phenomenon in quantum policy gradients using the Fisher Information Matrix spectrum.

arXiv:2203.10591•Mar 20, 2022•1 citation

Policy gradients using variational quantum circuits

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.

Quantum Machine Intelligence•Apr 11, 2023•29 citations

Gradient usability in few-qubit quantum Neural Networks: A signal-to-noise ratio framework for evaluating mitigation strategies

Brandon Shen, Karena Ling

The barren-plateau problem—the exponential flattening of optimization landscapes—severely limits the trainability of quantum neural networks (QNNs) on near-term hardware. This review examines ansatz-design and complementary mitigation strategies for few-qubit QNNs under finite-shot noisy intermediate-scale quantum (NISQ) constraints. Its primary conceptual contribution is a literature-grounded reframing of gradient-based trainability around pointwise finite-shot gradient signal-to-noise ratio (SNR) while retaining exact-gradient landscape variance as a complementary theoretical diagnostic. This reframing follows because gradient-based optimization acts on finite-shot gradient estimates whose statistical resolvability directly determines whether the optimizer receives a reliably distinguishable update signal, whereas numerical studies restricted to 2–10 qubits generally cannot establish asymptotic exponential or polynomial gradient-decay laws without theoretical support. Estimator SNR therefore addresses the immediate experimental question of whether the update signal is distinguishable from its sampling uncertainty, whereas exact-gradient variance characterizes landscape concentration. The review surveys ansatz design, parameter initialization, cost-function selection, and error management, and develops a conditional design heuristic: problem-inspired ansätze are preferred when the task maps onto circuit families with demonstrated trainability properties; otherwise, shallow hardware-efficient ansätze provide controlled baselines. As a secondary contribution, the review proposes a complete eight-configuration 2 3 factorial framework for evaluating interactions among restricted entanglement, local cost functions, and active residual shortcuts under matched measurement budgets. An exact identity motivates testing the interaction between restricted entanglement and local costs at the exact-gradient-signal level, but the interaction is tested two-sided; all estimator-SNR interactions remain open empirical questions. The framework is presented as an implementable research agenda enabled by the SNR perspective rather than as evidence that any interaction has been established.

Journal of High School Science•Aug 1, 2026•0 citations
This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables arelog⁡(n)\log(n)log(n)-local, the gradients vanish polynomially with the number of qubits providedO(log⁡n)\mathrm{O}(\log n)O(logn)depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.
Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.
This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.
The training of a parameterized model largely depends on the landscape of the underlying loss function. In particular, vanishing gradients are a central bottleneck in the scalability of variational quantum algorithms (VQAs), and are known to arise in various ways. However, a caveat of most existing gradient bound results is the requirement of t-design circuit assumptions that are typically not satisfied in practice. In this work, we loosen these assumptions altogether and derive tight upper and lower bounds on loss and gradient concentration for a large class of parameterized quantum circuits and arbitrary observables, which are significantly stronger than prior work. Moreover, we show that these bounds, as well as the variance of the loss itself, can be estimated efficiently and classically-providing practical tools to study the loss landscapes of VQA models, including verifying whether or not a circuit/observable induces barren plateaus. In particular, our results can readily be leveraged to rule out barren plateaus for a realistic class of ansätze and mixed observables, namely, observables containing a non-vanishing local term. This insight has direct implications for hybrid Quantum Generative Adversarial Networks (qGANs). We prove that designing the discriminator appropriately leads to 1-local weights that stay constant in the number of qubits, regardless of discriminator depth. This implies that qGANs with appropriately chosen generators do not suffer from barren plateaus even at scale-making them a promising candidate for applications in generative quantum machine learning. We demonstrate this result by training a qGAN to learn a 2D mixture of Gaussian distributions with up to 16 qubits, and provide numerical evidence that global contributions to the gradient, while initially exponentially small, may kick in substantially over the course of training.
Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an εεε-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly or even outperform typical classical neural networks used in standard benchmarking environments and in quantum control, using only a fraction of the parameters. Moreover, we study the Barren Plateau phenomenon in quantum policy gradients using the Fisher Information Matrix spectrum.
Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.
The barren-plateau problem—the exponential flattening of optimization landscapes—severely limits the trainability of quantum neural networks (QNNs) on near-term hardware. This review examines ansatz-design and complementary mitigation strategies for few-qubit QNNs under finite-shot noisy intermediate-scale quantum (NISQ) constraints. Its primary conceptual contribution is a literature-grounded reframing of gradient-based trainability around pointwise finite-shot gradient signal-to-noise ratio (SNR) while retaining exact-gradient landscape variance as a complementary theoretical diagnostic. This reframing follows because gradient-based optimization acts on finite-shot gradient estimates whose statistical resolvability directly determines whether the optimizer receives a reliably distinguishable update signal, whereas numerical studies restricted to 2–10 qubits generally cannot establish asymptotic exponential or polynomial gradient-decay laws without theoretical support. Estimator SNR therefore addresses the immediate experimental question of whether the update signal is distinguishable from its sampling uncertainty, whereas exact-gradient variance characterizes landscape concentration. The review surveys ansatz design, parameter initialization, cost-function selection, and error management, and develops a conditional design heuristic: problem-inspired ansätze are preferred when the task maps onto circuit families with demonstrated trainability properties; otherwise, shallow hardware-efficient ansätze provide controlled baselines. As a secondary contribution, the review proposes a complete eight-configuration 2 3 factorial framework for evaluating interactions among restricted entanglement, local cost functions, and active residual shortcuts under matched measurement budgets. An exact identity motivates testing the interaction between restricted entanglement and local costs at the exact-gradient-signal level, but the interaction is tested two-sided; all estimator-SNR interactions remain open empirical questions. The framework is presented as an implementable research agenda enabled by the SNR perspective rather than as evidence that any interaction has been established.