arXiv:2311.00756 · SciPost Physics Core

The quantum cartpole: A benchmark environment for non-linear reinforcement learning

Kai Meinerz, Simon Trebst, Mark Spencer Rudner, Evert van Nieuwenburg

Published May 7, 2024• 0 citations

Abstract

Feedback-based control is the de-facto standard when it comes to controlling classical stochastic systems and processes. However, standard feedback-based control methods are challenged by quantum systems due to measurement induced backaction and partial observability. Here we remedy this by using weak quantum measurements and model-free reinforcement learning agents to perform quantum control. By comparing control algorithms with and without state estimators to stabilize a quantum particle in an unstable state near a local potential energy maximum, we show how a trade-off between state estimation and controllability arises. For the scenario where the classical analogue is highly nonlinear, the reinforcement learned controller has an advantage over the standard controller. Additionally, we demonstrate the feasibility of using transfer learning to develop a quantum control agent trained via reinforcement learning on a classical surrogate of the quantum control problem. Finally, we present results showing how the reinforcement learning control strategy differs from the classical controller in the non-linear scenarios.

DOI: 10.21468/scipostphyscore.7.2.026

Related papers

Deep Reinforcement Learning Control of Quantum Cartpoles

Zhikang T. Wang, Yuto Ashida, Masahito Ueda

We generalize a standard benchmark of reinforcement learning, the classical cartpole balancing problem, to the quantum regime by stabilizing a particle in an unstable potential through measurement and feedback. We use state-of-the-art deep reinforcement learning to stabilize a quantum cartpole and find that our deep learning approach performs comparably to or better than other strategies in standard control theory. Our approach also applies to measurement-feedback cooling of quantum oscillators, showing the applicability of deep learning to general continuous-space quantum control.

arXiv:1910.09200Physical Review LettersSep 2, 202062 citations

Adaptive Quantum Control via Reinforcement Learning

Jincheng Zhang

This paper investigates the application of reinforcement learning (RL) for adaptive quantum control. Traditional quantum control methods often rely on pre-designed pulse shapes optimized for specific scenarios, lacking adaptability to variations in system parameters or environmental noise. This work proposes a novel framework where an RL agent learns to dynamically adjust quantum control pulses in real-time, maximizing the fidelity of quantum operations. The agent receives feedback from the quantum system, typically measured through state variables, and utilizes this information to optimize the control pulse sequence. This adaptive approach addresses the limitations of static control schemes and promises enhanced robustness and performance in complex quantum systems. The core claim centers on utilizing reinforcement learning to design optimal quantum control pulses. The central mechanism involves a trained controller dynamically adjusting pulses based on system feedback. The presented methodology offers a new approach to quantum control, particularly relevant in noisy environments and systems with parameter variations. The research demonstrates the potential of RL for achieving superior quantum control outcomes compared to conventional techniques. ---

Sep 4, 20260 citations

Towards Real-time Control of a CartPole System on a Quantum Computer

Nguyen Truong Thu Ngo, Väinö Mehtola, Jérome Lenssen +4 more

The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.

quant-pharXiv:2605.01716May 5, 20260 citations

Measurement-Based Feedback Quantum Control with Deep Reinforcement Learning for a Double-Well Nonlinear Potential

Sangkha Borah, Bijita Sarma, Michael J. Kewming +2 more

Closed loop quantum control uses measurement to control the dynamics of a quantum system to achieve either a desired target state or target dynamics. In the case when the quantum Hamiltonian is quadratic in x and p, there are known optimal control techniques to drive the dynamics toward particular states, e.g., the ground state. However, for nonlinear Hamiltonian such control techniques often fail. We apply deep reinforcement learning (DRL), where an artificial neural agent explores and learns to control the quantum evolution of a highly nonlinear system (double well), driving the system toward the ground state with high fidelity. We consider a DRL strategy which is particularly motivated by experiment where the quantum system is continuously but weakly measured. This measurement is then fed back to the neural agent and used for training. We show that the DRL can effectively learn counterintuitive strategies to cool the system to a nearly pure "cat" state, which has a high overlap fidelity with the true ground state.

arXiv:2104.11856Physical Review LettersNov 2, 202177 citations

Realizing a deep reinforcement learning agent discovering real-time feedback control strategies for a quantum system

Kevin Reuer, Jonas Landgraf, Thomas Fösel +10 more

To realize the full potential of quantum technologies, finding good strategies to control quantum information processing devices in real time becomes increasingly important. Usually these strategies require a precise understanding of the device itself, which is generally not available. Model-free reinforcement learning circumvents this need by discovering control strategies from scratch without relying on an accurate description of the quantum system. Furthermore, important tasks like state preparation, gate teleportation and error correction need feedback at time scales much shorter than the coherence time, which for superconducting circuits is in the microsecond range. Developing and training a deep reinforcement learning agent able to operate in this real-time feedback regime has been an open challenge. Here, we have implemented such an agent in the form of a latency-optimized deep neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit into a target state. To train the agent, we use model-free reinforcement learning that is based solely on measurement data. We study the agent's performance for strong and weak measurements, and for three-level readout, and compare with simple strategies based on thresholding. This demonstration motivates further research towards adoption of reinforcement learning for real-time feedback control of quantum devices and more generally any physical system requiring learnable low-latency feedback control.

arXiv:2210.16715Oct 30, 20223 citations

Entanglement engineering of optomechanical systems by reinforcement learning

Li-Li Ye, Christian Arenz, Joseph M. Lukens +1 more

Entanglement is fundamental to quantum information science and technology, yet controlling and manipulating entanglement -- so-called entanglement engineering -- for arbitrary quantum systems remains a formidable challenge. There are two difficulties: the fragility of quantum entanglement and its experimental characterization. We develop a model-free deep reinforcement-learning (RL) approach to entanglement engineering, in which feedback control together with weak continuous measurement and partial state observation is exploited to generate and maintain desired entanglement. We employ quantum optomechanical systems with linear or nonlinear photon-phonon interactions to demonstrate the workings of our machine-learning-based entanglement engineering protocol. In particular, the RL agent sequentially interacts with one or multiple parallel quantum optomechanical environments, collects trajectories, and updates the policy to maximize the accumulated reward to create and stabilize quantum entanglement over an arbitrary amount of time. The machine-learning-based model-free control principle is applicable to the entanglement engineering of experimental quantum systems in general.

arXiv:2406.04550Jun 6, 20241 citation