Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2611.10793

Trainability-Simulability Tradeoff in Parameterized IQP-based Reinforcement Learning Policies

André Sequeira

Published Nov 7, 2026• 0 citations
arXivHTMLPDF

Abstract

Parameterized instantaneous quantum polynomial-time (IQP) circuits generate distributions conjectured to be hard to sample classically. As reinforcement learning (RL) policies, their $2^n$ measurement outcomes are typically mapped to a smaller action space by a classical decoder. This aggregation can mitigate initialization concentration while making the policy classically simulable. We characterize this tradeoff through the decoder's Fourier--Pauli expansion. Under independent uniform initialization, we express policy variance as an exact sum over groups of nonconstant Pauli components with identical active-generator signatures. Each group contributes its squared aggregate coefficient times $2^{-r}$, where $r$ is the binary rank of its active generators. An inverse-polynomial aggregate coefficient with logarithmic rank prevents exponential policy concentration. Sparse Pauli expansions and contiguous action bins permit efficient classical additive estimation of action probabilities. We introduce a majority decoder combining protected low-weight components with exponentially large exact Fourier support, preserving initialization nonconcentration while excluding polynomial-size enumeration of the exact expansion. Numerical experiments corroborate the variance laws. In ten-qubit CartPole experiments, majority achieves higher returns than recursive parity and the tested local readouts, while higher-weight Pauli sectors contribute an increasing relative share of gradient magnitude during training. A complementary 21-qubit block-chain policy outperforms an extensive pool of fully connected classical neural networks in episode efficiency, even though its bounded-width structure permits efficient classical simulation. These results guide decoder design for quantum and quantum-inspired RL, distinguishing policy nonconcentration, learning performance, and classical simulability.

DOI
10.48550/arxiv.2611.10793

Related papers

Trainability issues in quantum policy gradients

André Sequeira, Luís Paulo Santos, Luís Soares Barbosa

Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.

arXiv:2406.09614•Machine Learning Science and Technology•Jul 26, 2024•5 citations

Trainability Issues in Quantum Policy Gradients with Softmax Activations

André Sequeira, Luís Paulo Santos, Luís Soares Barbosa

This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables are$\log(n)$-local, the gradients vanish polynomially with the number of qubits provided$\mathrm{O}(\log n)$depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.

IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 15, 2024•1 citation

Characterizing Trainability of Instantaneous Quantum Polynomial Circuit Born Machines

Kevin Shen, Susanne Pielawa, Vedran Dunjko +1 more

Instantaneous quantum polynomial quantum circuit Born machines (IQP-QCBMs) have been proposed as quantum generative models with a classically tractable training objective based on the maximum mean discrepancy (MMD) and a potential quantum advantage motivated by sampling-complexity arguments, making them an exciting model worth deeper investigation. While recent works have further proven the universality of a (slightly generalized) model, the next immediate question pertains to its trainability, i.e., whether it suffers from the exponentially vanishing loss gradients, known as the barren plateau issue, preventing effective use, and how regimes of trainability overlap with regimes of possible quantum advantage. Here, we provide significant strides in these directions. To study the trainability at initialization, we analytically derive closed-form expressions for the variances of the partial derivatives of the MMD loss function and provide general upper and lower bounds. With uniform initialization, we show that barren plateaus depend on the generator set and the spectrum of the chosen kernel. We identify regimes in which low-weight-biased kernels avoid exponential gradient suppression in structured topologies. Also, we prove that a small-variance Gaussian initialization ensures polynomial scaling for the gradient under mild conditions. As for the potential quantum advantage, we further argue, based on previous complexity-theoretic arguments, that sparse IQP families can output a probability distribution family that is classically intractable, and that this distribution remains trainable at initialization at least at lower-weight frequencies.

quant-pharXiv:2602.11042•Feb 12, 2026•14 citations

Data re-uploading and expressivity/trainability trade-off on quantum policy gradients

Bernardo Manuel Barbosa Soares

This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.

•Dec 1, 2024•0 citations

Discovery of connectivity-trainability trade-off of IQP circuits for Hamiltonian optimization

Quoc Chuong Nguyen, Dao Duy Tung, Vu Tuan Hai

Instantaneous Quantum Polynomial-time (IQP) circuits are promising candidates for near-term quantum computing due to the conjectured classical hardness of their sampling task and their shallow, highly parallelizable structure. Despite extensive studies of their complexity-theoretic properties, the role of circuit connectivity in their trainability as variational ansatze remains poorly understood. In this work, we present an investigation of IQP circuits for Hamiltonian optimization that traces a mechanistic chain linking circuit connectivity, entanglement generation, and gradient scaling: increasing connectivity drives the circuit output toward Haar-like statistics and volume-law entanglement, and this increase in entanglement is consistently associated with an exponential decay of the gradient variance and degraded optimization performance, while sparser architectures retain stable gradients at the cost of global expressibility and entanglement capacity. This connectivity-entanglement-gradient mechanism produces a nontrivial connectivity-trainability trade-off, demonstrating that effective IQP-based Hamiltonian optimization requires a balance between global expressibility and trainability rather than maximal expressivity alone. In particular, on our benchmarks, a circuit generating no entanglement already matches the optimization performance of more highly connected circuits, and a circular topology retains entanglement capacity while preserving trainability, making it a broadly effective choice among the architectures considered.

cs.SICross-listedarXiv:2606.24264•Physica Scripta•Jun 24, 2026•0 citations

Embedding-Dependent Performance of Variational Quantum Reinforcement Learning for Intrusion Detection Under Dimensionality Constraints

Raid Anis Kerkatou, Hacene Belhadef, Aicha Eutamene +1 more

Network intrusion detection systems (IDS) operate in high-dimensional feature spaces under evolving attack patterns and asymmetric misclassification costs, where false negatives represent a critical security risk. Reinforcement learning (RL) offers a natural mechanism for encoding domain-specific misclassification costs directly into the learning signal through reward shaping, enabling cost-sensitive policy optimization in adaptive streaming environments. However, the integration of variational quantum models into RL-based IDS remains insufficiently explored. This work investigates a variational quantum reinforcement learning (VQRL) framework for intrusion detection, in which parameterized quantum circuits are employed to model the policy function. We adopt an RL formulation primarily as a principled cost-sensitive optimization approach rather than to exploit sequential state dependencies, and we employ Instantaneous Quantum Polynomial (IQP) embedding as a quantum feature encoding strategy. The study analyzes how embedding expressivity interacts with varying levels of dimensionality reduction via principal component analysis (PCA) on the CICIDS2017 dataset. Experiments demonstrate that VQRL-IQP achieves high recall and reduces false negative rates in moderately high-dimensional feature spaces compared to a classical RL baseline. This improvement is accompanied by an increase in false positive rates, reflecting a trade-off shaped jointly by the reward structure and the structural properties of IQP encoding. Statistical validation across five independent runs confirms the consistency of these trends. Importantly, no general quantum advantage in accuracy or computational efficiency is claimed; rather, the results indicate that VQRL-IQP offers a distinct error trade-off that is operationally valuable in security-critical scenarios where minimizing missed attacks is the primary objective.

Electronics•Jun 30, 2026•1 citation
Parameterized instantaneous quantum polynomial-time (IQP) circuits generate distributions conjectured to be hard to sample classically. As reinforcement learning (RL) policies, their 2n2^n2n measurement outcomes are typically mapped to a smaller action space by a classical decoder. This aggregation can mitigate initialization concentration while making the policy classically simulable. We characterize this tradeoff through the decoder's Fourier--Pauli expansion. Under independent uniform initialization, we express policy variance as an exact sum over groups of nonconstant Pauli components with identical active-generator signatures. Each group contributes its squared aggregate coefficient times 2−r2^{-r}2−r, where rrr is the binary rank of its active generators. An inverse-polynomial aggregate coefficient with logarithmic rank prevents exponential policy concentration. Sparse Pauli expansions and contiguous action bins permit efficient classical additive estimation of action probabilities. We introduce a majority decoder combining protected low-weight components with exponentially large exact Fourier support, preserving initialization nonconcentration while excluding polynomial-size enumeration of the exact expansion. Numerical experiments corroborate the variance laws. In ten-qubit CartPole experiments, majority achieves higher returns than recursive parity and the tested local readouts, while higher-weight Pauli sectors contribute an increasing relative share of gradient magnitude during training. A complementary 21-qubit block-chain policy outperforms an extensive pool of fully connected classical neural networks in episode efficiency, even though its bounded-width structure permits efficient classical simulation. These results guide decoder design for quantum and quantum-inspired RL, distinguishing policy nonconcentration, learning performance, and classical simulability.
Abstract This research explores the trainability of Parameterized Quantum Circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and the mapping of these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.
This research addresses the trainability of Parameterized Quantum Circuit-based Softmax policies in Reinforcement Learning. We assess the trainability of these policies by examining the scaling of the expected value of the partial derivative of the log policy objective function. Here, we assume the hardware-efficient ansatz with blocks forming local 2-designs. In this setting, we show that if each expectation value representing the action's numerical preference is composed of a global observable, it leads to exponentially vanishing gradients. In contrast, for n-qubit systems, if the observables arelog⁡(n)\log(n)log(n)-local, the gradients vanish polynomially with the number of qubits providedO(log⁡n)\mathrm{O}(\log n)O(logn)depth. We also show that the expectation of the gradient of the log policy objective depend on the entire action space. Thus, even though global observables lead to concentration, the gradient signal can still be propagated in the presence of at least a single local observable. We validate the theoretical predictions in a series of ansatze and evaluate the performance of local and global observables in a multi-armed bandit setting.
Instantaneous quantum polynomial quantum circuit Born machines (IQP-QCBMs) have been proposed as quantum generative models with a classically tractable training objective based on the maximum mean discrepancy (MMD) and a potential quantum advantage motivated by sampling-complexity arguments, making them an exciting model worth deeper investigation. While recent works have further proven the universality of a (slightly generalized) model, the next immediate question pertains to its trainability, i.e., whether it suffers from the exponentially vanishing loss gradients, known as the barren plateau issue, preventing effective use, and how regimes of trainability overlap with regimes of possible quantum advantage. Here, we provide significant strides in these directions. To study the trainability at initialization, we analytically derive closed-form expressions for the variances of the partial derivatives of the MMD loss function and provide general upper and lower bounds. With uniform initialization, we show that barren plateaus depend on the generator set and the spectrum of the chosen kernel. We identify regimes in which low-weight-biased kernels avoid exponential gradient suppression in structured topologies. Also, we prove that a small-variance Gaussian initialization ensures polynomial scaling for the gradient under mild conditions. As for the potential quantum advantage, we further argue, based on previous complexity-theoretic arguments, that sparse IQP families can output a probability distribution family that is classically intractable, and that this distribution remains trainable at initialization at least at lower-weight frequencies.
This thesis investigates the integration of quantum computing into machine learning, specifically focusing on reinforcement learning. The study focuses on the building of a code library to enable the exploration of the use of variational quantum circuits as function approximators within reinforcement learning algorithms, specifically policy gradient methods. A comprehensive framework is developed to analyze the performance and trainability of variational quantum circuit-based reinforcement learning models, supported by this library, to understand how different quantum circuit designs and data re-uploading techniques might affect the performance and trainability. To demonstrate the library’s capabilities, experiments were conducted in classic control environments like CartPole-v1 and Acrobot-v1, evaluating the performance (measured as average return) and trainability of three quantum models. These experiments examined the effects of data re-uploading, classical postprocessing, and inverse temperature parameter scheduling on both metrics. The empirical results show that variational quantum circuits are effective function approximators, with data re-uploading significantly enhancing model performance. Additionally, the globality of the measured observables plays a critical role, with higher globality values leading to better performance. Despite increasing circuit expressiveness, the gradient magnitudes and variances remain substantial throughout training. Furthermore, the tested inverse temperature schedules, used to adjust the policy’s level of exploration, did not show any significant advantage, indicating the challenge in finding optimal settings for these schedules. The study also introduces the universal quantum classifier adapted for reinforcement learning. Contrary to the barren plateau phenomenon, it has been empirically demonstrated that although increasing the number of qubits leads to performance drops in the Acrobot environment, it does not lead to vanishing gradients during the agents’ training process. Furthermore, under specific sets of hyperparameters, all three quantum models demonstrate strong and similar performance levels in both environments.
Instantaneous Quantum Polynomial-time (IQP) circuits are promising candidates for near-term quantum computing due to the conjectured classical hardness of their sampling task and their shallow, highly parallelizable structure. Despite extensive studies of their complexity-theoretic properties, the role of circuit connectivity in their trainability as variational ansatze remains poorly understood. In this work, we present an investigation of IQP circuits for Hamiltonian optimization that traces a mechanistic chain linking circuit connectivity, entanglement generation, and gradient scaling: increasing connectivity drives the circuit output toward Haar-like statistics and volume-law entanglement, and this increase in entanglement is consistently associated with an exponential decay of the gradient variance and degraded optimization performance, while sparser architectures retain stable gradients at the cost of global expressibility and entanglement capacity. This connectivity-entanglement-gradient mechanism produces a nontrivial connectivity-trainability trade-off, demonstrating that effective IQP-based Hamiltonian optimization requires a balance between global expressibility and trainability rather than maximal expressivity alone. In particular, on our benchmarks, a circuit generating no entanglement already matches the optimization performance of more highly connected circuits, and a circular topology retains entanglement capacity while preserving trainability, making it a broadly effective choice among the architectures considered.
Network intrusion detection systems (IDS) operate in high-dimensional feature spaces under evolving attack patterns and asymmetric misclassification costs, where false negatives represent a critical security risk. Reinforcement learning (RL) offers a natural mechanism for encoding domain-specific misclassification costs directly into the learning signal through reward shaping, enabling cost-sensitive policy optimization in adaptive streaming environments. However, the integration of variational quantum models into RL-based IDS remains insufficiently explored. This work investigates a variational quantum reinforcement learning (VQRL) framework for intrusion detection, in which parameterized quantum circuits are employed to model the policy function. We adopt an RL formulation primarily as a principled cost-sensitive optimization approach rather than to exploit sequential state dependencies, and we employ Instantaneous Quantum Polynomial (IQP) embedding as a quantum feature encoding strategy. The study analyzes how embedding expressivity interacts with varying levels of dimensionality reduction via principal component analysis (PCA) on the CICIDS2017 dataset. Experiments demonstrate that VQRL-IQP achieves high recall and reduces false negative rates in moderately high-dimensional feature spaces compared to a classical RL baseline. This improvement is accompanied by an increase in false positive rates, reflecting a trade-off shaped jointly by the reward structure and the structural properties of IQP encoding. Statistical validation across five independent runs confirms the consistency of these trends. Importantly, no general quantum advantage in accuracy or computational efficiency is claimed; rather, the results indicate that VQRL-IQP offers a distinct error trade-off that is operationally valuable in security-critical scenarios where minimizing missed attacks is the primary objective.