Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2304.09718 · Physical Review Research

Sample-efficient model-based reinforcement learning for quantum control

Irtaza Khalid, Carrie Ann Weidner, EDMOND A. JONCKHEERE, S. G. Schirmer, Frank C. Langbein

Published Oct 2, 2023• 15 citations
arXivHTMLPDF

Abstract

We propose a model-based reinforcement learning (RL) approach for noisy time-dependent gate optimization with reduced sample complexity over model-free RL. Sample complexity is defined as the number of controller interactions with the physical system. Leveraging an inductive bias, inspired by recent advances in neural ordinary differential equations (ODEs), we use an autodifferentiable ODE, parametrized by a learnable Hamiltonian ansatz, to represent the model approximating the environment, whose time-dependent part, including the control, is fully known. Control alongside Hamiltonian learning of continuous time-independent parameters is addressed through interactions with the system. We demonstrate an order of magnitude advantage in sample complexity of our method over standard model-free RL in preparing some standard unitary gates with closed and open system dynamics, in realistic computational experiments incorporating single-shot measurements, arbitrary Hilbert space truncations, and uncertainty in Hamiltonian parameters. Also, the learned Hamiltonian can be leveraged by existing control methods like GRAPE (gradient ascent pulse engineering) for further gradient-based optimization with the controllers found by RL as initializations. Our algorithm, which we apply to nitrogen vacancy (NV) centers and transmons, is well suited for controlling partially characterized one- and two-qubit systems.

DOI
10.1103/physrevresearch.5.043002

Related papers

Robust quantum control using reinforcement learning from demonstration

Shengyong Li, Yidian Fan, Xiangyang Li +6 more

Quantum control requires high-precision and robust control pulses to ensure optimal system performance. However, control sequences generated with a system model may suffer from model bias, leading to low fidelity. While model-free reinforcement learning (RL) methods have been developed to avoid such biases, training an RL agent from scratch can be time-consuming, often taking hours to gather enough samples for convergence. This challenge has hindered the broad application of RL techniques to larger and more complex quantum control issues, limiting their adaptability. In this work, we use Reinforcement Learning from Demonstration (RLfD) to leverage the control sequences generated with system models and further optimize them with RL to avoid model bias. By avoiding learning from scratch and starting with reasonable control pulse shapes, this approach can increase sample efficiency by reducing the number of samples, which can significantly reduce the training time. Thus, this method can effectively handle pulse shapes that are discretized into more than 1000 pieces without compromising final fidelity. We have simulated the preparation of several high-fidelity non-classical states using the RLfD method. We also find that the training process is more stable when using RLfD. In addition, this method is suitable for fast gate calibration using reinforcement learning.

arXiv:2503.21085•npj Quantum Information•Jul 25, 2025•12 citations

Uncovering Latent Structures in Robust Pulse Sequences: A Model-Based Reinforcement Learning Approach for Adaptable Quantum Control

Tobias Kiermeyer, Thomas Heydenreich, Léo Van Damme +3 more

Real-time adaptive control of quantum systems requires rapid generation of robust, high-fidelity pulses across a continuous range of operating conditions. Standard optimization algorithms such as gradient-ascent pulse engineering (GRAPE) solve each instance independently, discarding information between runs and requiring costly reinitialization when parameters change. We present an approach to robust optimal quantum control based on model-based reinforcement learning, in which a single neural network -- embedding the Hamiltonian directly into the training pipeline -- generates robust gates across an entire family of gate configurations, without pre-computed training data. Demonstrated on a single-spin (two-level) system, the trained networks produce pulses for arbitrary rotation angles over a range of pulse durations, detunings, and field inhomogeneities in milliseconds, at fidelities comparable to multi-seed GRAPE. The framework is inherently adaptable: any parameter entering the Hamiltonian can serve as a network input, extending the approach to different systems and control settings. Beyond speed, the network reveals structure in the control landscape: it discovers the same structured phase profiles that appear in GRAPE solutions -- made identifiable through fidelity-invariant symmetry transformations -- but more consistently than independent optimization. This consistency enables smooth interpolation across the entire trained parameter space.

quant-pharXiv:2606.24507•Jun 24, 2026•0 citations

Quantum control of solid state qubits via Reinforcement Learning

Dennis Herb, Mirko Rossini, Sabine Wölk +3 more

The use of reinforcement learning (RL) in the design of quantum control strategies for pulse level control of NISQ era quantum devices is attracting growing interest. Model-free RL, not requiring preexisting knowledge of the physical system, acts as a black-box optimization tool well suited for noisy devices. In this work, we provide a simulation-based analysis of the performance of model-free RL for state preparation and qubit gate design in a challenging setup: indirect control of NV nuclear spins in a spin-bath. In particular, the spin-bath is modeled via the generalized cluster correlation expansion accounting for magnetic impurities and intra-bath interactions. We analyze the performance of the RL agents by performing simulations with minimally observable environments and noisy, sparse rewards as introduced in (Phys. Rev. X 12, 011059), and compare them to other state-of-the-art black-box optimization tools.

quant-pharXiv:2610.21987•Oct 16, 2026•0 citations

Non-Markovian Quantum Control via Model Maximum Likelihood Estimation and Reinforcement Learning

Tanmay Neema, Susmit Jha, Tuhin Sahai

Reinforcement Learning (RL) techniques have been increasingly applied in optimizing control systems. However, their application in quantum systems is hampered by the challenge of performing closed-loop control due to the difficulty in measuring these systems. This often leads to reliance on assumed models, introducing model bias, a problem that is exacerbated in open quantum dynamics where Markovian approximations are not valid. To address these challenges, we propose a novel approach that incorporates the non-Markovian nature of the environment into a low-dimensional effective reservoir. By initially employing a series of measurements as a 'dataset', we utilize machine learning techniques to learn the effective quantum dynamics more efficiently than traditional tomographic methods. Our methodology aims to demonstrates that by integrating reinforcement learning with model learning, it is possible to devise control policies and models that can counteract decoherence in a spin-boson system. This approach may not only mitigates the issues of model bias but also provides a more accurate representation of quantum dynamics, paving the way for more effective quantum control strategies.

arXiv:2402.05084•Feb 7, 2024•0 citations

A differentiable programming method for quantum control

Frank Schäfer, Michal Kloc, Christoph Bruder +1 more

Abstract Optimal control is highly desirable in many current quantum systems, especially to realize tasks in quantum information processing. We introduce a method based on differentiable programming to leverage explicit knowledge of the differential equations governing the dynamics of the system. In particular, a control agent is represented as a neural network that maps the state of the system at a given time to a control pulse. The parameters of this agent are optimized via gradient information obtained by direct differentiation through both the neural network and the differential equation of the system. This fully differentiable reinforcement learning approach ultimately yields time-dependent control parameters optimizing a desired figure of merit. We demonstrate the method’s viability and robustness to noise in eigenstate preparation tasks for three systems: a single qubit, a chain of qubits, and a quantum parametric oscillator.

arXiv:2002.08376•Machine Learning Science and Technology•May 29, 2020•50 citations

Model-aware reinforcement learning for high-performance Bayesian experimental design in quantum metrology

Federico Belliardo, Fabio Zoratti, Florian Marquardt +1 more

Quantum sensors offer control flexibility during estimation by allowing manipulation by the experimenter across various parameters. For each sensing platform, pinpointing the optimal controls to enhance the sensor's precision remains a challenging task. While an analytical solution might be out of reach, machine learning offers a promising avenue for many systems of interest, especially given the capabilities of contemporary hardware. We have introduced a versatile procedure capable of optimizing a wide range of problems in quantum metrology, estimation, and hypothesis testing by combining model-aware reinforcement learning (RL) with Bayesian estimation based on particle filtering. To achieve this, we had to address the challenge of incorporating the many non-differentiable steps of the estimation in the training process, such as measurements and the resampling of the particle filter. Model-aware RL is a gradient-based method, where the derivatives of the sensor's precision are obtained through automatic differentiation (AD) in the simulation of the experiment. Our approach is suitable for optimizing both non-adaptive and adaptive strategies, using neural networks or other agents. We provide an implementation of this technique in the form of a Python library called qsensoropt, alongside several pre-made applications for relevant physical platforms, namely NV centers, photonic circuits, and optical cavities. This library will be released soon on PyPI. Leveraging our method, we've achieved results for many examples that surpass the current state-of-the-art in experimental design. In addition to Bayesian estimation, leveraging model-aware RL, it is also possible to find optimal controls for the minimization of the Cramér-Rao bound, based on Fisher information.

arXiv:2312.16985•Quantum•Dec 10, 2024•6 citations
We propose a model-based reinforcement learning (RL) approach for noisy time-dependent gate optimization with reduced sample complexity over model-free RL. Sample complexity is defined as the number of controller interactions with the physical system. Leveraging an inductive bias, inspired by recent advances in neural ordinary differential equations (ODEs), we use an autodifferentiable ODE, parametrized by a learnable Hamiltonian ansatz, to represent the model approximating the environment, whose time-dependent part, including the control, is fully known. Control alongside Hamiltonian learning of continuous time-independent parameters is addressed through interactions with the system. We demonstrate an order of magnitude advantage in sample complexity of our method over standard model-free RL in preparing some standard unitary gates with closed and open system dynamics, in realistic computational experiments incorporating single-shot measurements, arbitrary Hilbert space truncations, and uncertainty in Hamiltonian parameters. Also, the learned Hamiltonian can be leveraged by existing control methods like GRAPE (gradient ascent pulse engineering) for further gradient-based optimization with the controllers found by RL as initializations. Our algorithm, which we apply to nitrogen vacancy (NV) centers and transmons, is well suited for controlling partially characterized one- and two-qubit systems.
Quantum control requires high-precision and robust control pulses to ensure optimal system performance. However, control sequences generated with a system model may suffer from model bias, leading to low fidelity. While model-free reinforcement learning (RL) methods have been developed to avoid such biases, training an RL agent from scratch can be time-consuming, often taking hours to gather enough samples for convergence. This challenge has hindered the broad application of RL techniques to larger and more complex quantum control issues, limiting their adaptability. In this work, we use Reinforcement Learning from Demonstration (RLfD) to leverage the control sequences generated with system models and further optimize them with RL to avoid model bias. By avoiding learning from scratch and starting with reasonable control pulse shapes, this approach can increase sample efficiency by reducing the number of samples, which can significantly reduce the training time. Thus, this method can effectively handle pulse shapes that are discretized into more than 1000 pieces without compromising final fidelity. We have simulated the preparation of several high-fidelity non-classical states using the RLfD method. We also find that the training process is more stable when using RLfD. In addition, this method is suitable for fast gate calibration using reinforcement learning.
Real-time adaptive control of quantum systems requires rapid generation of robust, high-fidelity pulses across a continuous range of operating conditions. Standard optimization algorithms such as gradient-ascent pulse engineering (GRAPE) solve each instance independently, discarding information between runs and requiring costly reinitialization when parameters change. We present an approach to robust optimal quantum control based on model-based reinforcement learning, in which a single neural network -- embedding the Hamiltonian directly into the training pipeline -- generates robust gates across an entire family of gate configurations, without pre-computed training data. Demonstrated on a single-spin (two-level) system, the trained networks produce pulses for arbitrary rotation angles over a range of pulse durations, detunings, and field inhomogeneities in milliseconds, at fidelities comparable to multi-seed GRAPE. The framework is inherently adaptable: any parameter entering the Hamiltonian can serve as a network input, extending the approach to different systems and control settings. Beyond speed, the network reveals structure in the control landscape: it discovers the same structured phase profiles that appear in GRAPE solutions -- made identifiable through fidelity-invariant symmetry transformations -- but more consistently than independent optimization. This consistency enables smooth interpolation across the entire trained parameter space.
The use of reinforcement learning (RL) in the design of quantum control strategies for pulse level control of NISQ era quantum devices is attracting growing interest. Model-free RL, not requiring preexisting knowledge of the physical system, acts as a black-box optimization tool well suited for noisy devices. In this work, we provide a simulation-based analysis of the performance of model-free RL for state preparation and qubit gate design in a challenging setup: indirect control of NV nuclear spins in a spin-bath. In particular, the spin-bath is modeled via the generalized cluster correlation expansion accounting for magnetic impurities and intra-bath interactions. We analyze the performance of the RL agents by performing simulations with minimally observable environments and noisy, sparse rewards as introduced in (Phys. Rev. X 12, 011059), and compare them to other state-of-the-art black-box optimization tools.
Reinforcement Learning (RL) techniques have been increasingly applied in optimizing control systems. However, their application in quantum systems is hampered by the challenge of performing closed-loop control due to the difficulty in measuring these systems. This often leads to reliance on assumed models, introducing model bias, a problem that is exacerbated in open quantum dynamics where Markovian approximations are not valid. To address these challenges, we propose a novel approach that incorporates the non-Markovian nature of the environment into a low-dimensional effective reservoir. By initially employing a series of measurements as a 'dataset', we utilize machine learning techniques to learn the effective quantum dynamics more efficiently than traditional tomographic methods. Our methodology aims to demonstrates that by integrating reinforcement learning with model learning, it is possible to devise control policies and models that can counteract decoherence in a spin-boson system. This approach may not only mitigates the issues of model bias but also provides a more accurate representation of quantum dynamics, paving the way for more effective quantum control strategies.
Abstract Optimal control is highly desirable in many current quantum systems, especially to realize tasks in quantum information processing. We introduce a method based on differentiable programming to leverage explicit knowledge of the differential equations governing the dynamics of the system. In particular, a control agent is represented as a neural network that maps the state of the system at a given time to a control pulse. The parameters of this agent are optimized via gradient information obtained by direct differentiation through both the neural network and the differential equation of the system. This fully differentiable reinforcement learning approach ultimately yields time-dependent control parameters optimizing a desired figure of merit. We demonstrate the method’s viability and robustness to noise in eigenstate preparation tasks for three systems: a single qubit, a chain of qubits, and a quantum parametric oscillator.
Quantum sensors offer control flexibility during estimation by allowing manipulation by the experimenter across various parameters. For each sensing platform, pinpointing the optimal controls to enhance the sensor's precision remains a challenging task. While an analytical solution might be out of reach, machine learning offers a promising avenue for many systems of interest, especially given the capabilities of contemporary hardware. We have introduced a versatile procedure capable of optimizing a wide range of problems in quantum metrology, estimation, and hypothesis testing by combining model-aware reinforcement learning (RL) with Bayesian estimation based on particle filtering. To achieve this, we had to address the challenge of incorporating the many non-differentiable steps of the estimation in the training process, such as measurements and the resampling of the particle filter. Model-aware RL is a gradient-based method, where the derivatives of the sensor's precision are obtained through automatic differentiation (AD) in the simulation of the experiment. Our approach is suitable for optimizing both non-adaptive and adaptive strategies, using neural networks or other agents. We provide an implementation of this technique in the form of a Python library called qsensoropt, alongside several pre-made applications for relevant physical platforms, namely NV centers, photonic circuits, and optical cavities. This library will be released soon on PyPI. Leveraging our method, we've achieved results for many examples that surpass the current state-of-the-art in experimental design. In addition to Bayesian estimation, leveraging model-aware RL, it is also possible to find optimal controls for the minimization of the Cramér-Rao bound, based on Fisher information.