Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2401.08307 · IEEE Transactions on Quantum Engineering

On Quantum Natural Policy Gradients

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Published Jan 1, 2024• 2 citations
arXivHTMLPDF

Abstract

This article delves into the role of the quantum Fisher information matrix (FIM) in enhancing the performance of parameterized quantum circuit (PQC)-based reinforcement learning agents. While previous studies have highlighted the effectiveness of PQC-based policies preconditioned with the quantum FIM in contextual bandits, its impact in broader reinforcement learning contexts, such as Markov decision processes, is less clear. Through a detailed analysis of Löwner inequalities between quantum and classical FIMs, this study uncovers the nuanced distinctions and implications of using each type of FIM. Our results indicate that a PQC-based agent using the quantum FIM without additional insights typically incurs a larger approximation error and does not guarantee improved performance compared to the classical FIM. Empirical evaluations in classic control benchmarks suggest even though quantum FIM preconditioning outperforms standard gradient ascent, in general, it is not superior to classical FIM preconditioning.

DOI
10.1109/tqe.2024.3418094

Related papers

Policy Gradients using Variational Quantum Circuits

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an $ε$-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly or even outperform typical classical neural networks used in standard benchmarking environments and in quantum control, using only a fraction of the parameters. Moreover, we study the Barren Plateau phenomenon in quantum policy gradients using the Fisher Information Matrix spectrum.

arXiv:2203.10591•Mar 20, 2022•1 citation

Policy gradients using variational quantum circuits

André Sequeira, Luís Paulo Santos, Lu­ís Soares Barbosa

Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.

Quantum Machine Intelligence•Apr 11, 2023•29 citations

Quantum Natural Policy Gradients: Towards Sample-Efficient Reinforcement Learning

Nico Meyer, Daniel D. Scherer, Axel Plinge +2 more

Reinforcement learning is a growing field in AI with a lot of potential. Intelligent behavior is learned automatically through trial and error in interaction with the environment. However, this learning process is often costly. Using variational quantum circuits as function approximators potentially can reduce this cost. In order to implement this, we propose the quantum natural policy gradient (QNPG) algorithm - a second-order gradient-based routine that takes advantage of an efficient approximation of the quantum Fisher information matrix. We experimentally demonstrate that QNPG outperforms first-order based training on different Contextual Bandits environments regarding convergence speed and stability and moreover reduces the sample complexity. Furthermore, we provide evidence for the practical feasibility of our approach by training on a 12-qubit hardware device.

arXiv:2304.13571•IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 17, 2023•10 citations

Benchmarking Quantum Reinforcement Learning

Georg Kruse, Rodrigo Coelho, Andreas Roßkopf +2 more

Quantum Reinforcement Learning (QRL) has emerged as a promising research field, leveraging the principles of quantum mechanics to enhance the performance of reinforcement learning (RL) algorithms. However, despite its growing interest, QRL still faces significant challenges. It is still uncertain if QRL can show any advantage over classical RL beyond artificial problem formulations. Additionally, it is not yet clear which streams of QRL research show the greatest potential. The lack of a unified benchmark and the need to evaluate the reliance on quantum principles of QRL approaches are pressing questions. This work aims to address these challenges by providing a comprehensive comparison of three major QRL classes: Parameterized Quantum Circuit based QRL (PQC-QRL) (with one policy gradient (QPG) and one Q-Learning (QDQN) algorithm), Free Energy based QRL (FE-QRL), and Amplitude Amplification based QRL (AA-QRL). We introduce a set of metrics to evaluate the QRL algorithms on the widely applicable benchmark of gridworld games. Our results provide a detailed analysis of the strengths and weaknesses of the QRL classes, shedding light on the role of quantum principles in QRL and paving the way for future research in this field.

arXiv:2502.04909•International Conference on Agents and Artificial Intelligence (ICAART)•Jan 1, 2025•3 citations

Quantum natural gradient with thermal-state initialization

Michele Minervini, Dhrumil Patel, Mark M. Wilde

Parameterized quantum circuits (PQCs) are central to variational quantum algorithms (VQAs), yet their performance is hindered by complex loss landscapes that make their trainability challenging. Quantum natural gradient descent, which leverages the geometry of the parameterized space through quantum generalizations of the Fisher information matrix, offers a promising solution but has been largely limited to pure-state scenarios, with only approximate methods available for mixed-state settings. This paper addresses this question, originally posed in [Stokes et al., Quantum 4, 269 (2020)], by providing exact methods to compute three quantum generalizations of the Fisher information matrix-the Fisher-Bures, Wigner-Yanase, and Kubo-Mori information matrices-for PQCs initialized with thermal states. We prove that these matrix elements can be estimated using quantum algorithms combining the Hadamard test, classical random sampling, and Hamiltonian simulation. By broadening the set of quantum generalizations of Fisher information and realizing their unbiased estimation, our results enable the implementation of quantum natural gradient descent algorithms for mixed-state PQCs, thereby enhancing the flexibility of optimization when using VQAs. Another immediate consequence of our findings is to establish fundamental limitations on the ability to estimate the parameters of a state generated by an unknown PQC, when given sample access to such a state.

arXiv:2502.19487•Physical Review A•Jul 7, 2025•2 citations

Hybrid Quantum-Classical Policy Gradient for Adaptive Control of Cyber-Physical Systems: A Comparative Study of VQC vs. MLP

Aueaphum Aueawatthanaphisut, Nyi Wunna Tun

The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.

arXiv:2510.06010•Oct 7, 2025•0 citations
This article delves into the role of the quantum Fisher information matrix (FIM) in enhancing the performance of parameterized quantum circuit (PQC)-based reinforcement learning agents. While previous studies have highlighted the effectiveness of PQC-based policies preconditioned with the quantum FIM in contextual bandits, its impact in broader reinforcement learning contexts, such as Markov decision processes, is less clear. Through a detailed analysis of Löwner inequalities between quantum and classical FIMs, this study uncovers the nuanced distinctions and implications of using each type of FIM. Our results indicate that a PQC-based agent using the quantum FIM without additional insights typically incurs a larger approximation error and does not guarantee improved performance compared to the classical FIM. Empirical evaluations in classic control benchmarks suggest even though quantum FIM preconditioning outperforms standard gradient ascent, in general, it is not superior to classical FIM preconditioning.
Variational Quantum Circuits are being used as versatile Quantum Machine Learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to Reinforcement Learning, less is known. In this work, we considered a Variational Quantum Circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a Reinforcement Learning agent. We show that an εεε-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly or even outperform typical classical neural networks used in standard benchmarking environments and in quantum control, using only a fraction of the parameters. Moreover, we study the Barren Plateau phenomenon in quantum policy gradients using the Fisher Information Matrix spectrum.
Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.
Reinforcement learning is a growing field in AI with a lot of potential. Intelligent behavior is learned automatically through trial and error in interaction with the environment. However, this learning process is often costly. Using variational quantum circuits as function approximators potentially can reduce this cost. In order to implement this, we propose the quantum natural policy gradient (QNPG) algorithm - a second-order gradient-based routine that takes advantage of an efficient approximation of the quantum Fisher information matrix. We experimentally demonstrate that QNPG outperforms first-order based training on different Contextual Bandits environments regarding convergence speed and stability and moreover reduces the sample complexity. Furthermore, we provide evidence for the practical feasibility of our approach by training on a 12-qubit hardware device.
Quantum Reinforcement Learning (QRL) has emerged as a promising research field, leveraging the principles of quantum mechanics to enhance the performance of reinforcement learning (RL) algorithms. However, despite its growing interest, QRL still faces significant challenges. It is still uncertain if QRL can show any advantage over classical RL beyond artificial problem formulations. Additionally, it is not yet clear which streams of QRL research show the greatest potential. The lack of a unified benchmark and the need to evaluate the reliance on quantum principles of QRL approaches are pressing questions. This work aims to address these challenges by providing a comprehensive comparison of three major QRL classes: Parameterized Quantum Circuit based QRL (PQC-QRL) (with one policy gradient (QPG) and one Q-Learning (QDQN) algorithm), Free Energy based QRL (FE-QRL), and Amplitude Amplification based QRL (AA-QRL). We introduce a set of metrics to evaluate the QRL algorithms on the widely applicable benchmark of gridworld games. Our results provide a detailed analysis of the strengths and weaknesses of the QRL classes, shedding light on the role of quantum principles in QRL and paving the way for future research in this field.
Parameterized quantum circuits (PQCs) are central to variational quantum algorithms (VQAs), yet their performance is hindered by complex loss landscapes that make their trainability challenging. Quantum natural gradient descent, which leverages the geometry of the parameterized space through quantum generalizations of the Fisher information matrix, offers a promising solution but has been largely limited to pure-state scenarios, with only approximate methods available for mixed-state settings. This paper addresses this question, originally posed in [Stokes et al., Quantum 4, 269 (2020)], by providing exact methods to compute three quantum generalizations of the Fisher information matrix-the Fisher-Bures, Wigner-Yanase, and Kubo-Mori information matrices-for PQCs initialized with thermal states. We prove that these matrix elements can be estimated using quantum algorithms combining the Hadamard test, classical random sampling, and Hamiltonian simulation. By broadening the set of quantum generalizations of Fisher information and realizing their unbiased estimation, our results enable the implementation of quantum natural gradient descent algorithms for mixed-state PQCs, thereby enhancing the flexibility of optimization when using VQAs. Another immediate consequence of our findings is to establish fundamental limitations on the ability to estimate the parameters of a state generated by an unknown PQC, when given sample access to such a state.
The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.