Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Machine Learning Science and Technology

Towards real-time control of a CartPole system on a quantum computer

Nguyen Truong Thu Ngo, Väinö Mehtola, Jerome Lenssen, Peiyong Wang, F. Cosco, Tien‐Fu Lu, James Q. Quach

Published Jul 21, 2026• 0 citations
Publisher

Abstract

Abstract The application of quantum reinforcement learning to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum–classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit. We demonstrate that a single-qubit agent acts as an effective learning model, and its performance is functionally comparable to a classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on noisy intermediate-scale quantum hardware. By bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables, we achieve more than an order-of-magnitude improvement in execution speed on the VTT Q5 processor. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.

DOI
10.1088/2632-2153/ae8e2c

Related papers

Towards Real-time Control of a CartPole System on a Quantum Computer

Nguyen Truong Thu Ngo, Väinö Mehtola, Jérome Lenssen +4 more

The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.

quant-pharXiv:2605.01716•May 5, 2026•0 citations

Realizing a deep reinforcement learning agent discovering real-time feedback control strategies for a quantum system

Kevin Reuer, Jonas Landgraf, Thomas Fösel +10 more

To realize the full potential of quantum technologies, finding good strategies to control quantum information processing devices in real time becomes increasingly important. Usually these strategies require a precise understanding of the device itself, which is generally not available. Model-free reinforcement learning circumvents this need by discovering control strategies from scratch without relying on an accurate description of the quantum system. Furthermore, important tasks like state preparation, gate teleportation and error correction need feedback at time scales much shorter than the coherence time, which for superconducting circuits is in the microsecond range. Developing and training a deep reinforcement learning agent able to operate in this real-time feedback regime has been an open challenge. Here, we have implemented such an agent in the form of a latency-optimized deep neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit into a target state. To train the agent, we use model-free reinforcement learning that is based solely on measurement data. We study the agent's performance for strong and weak measurements, and for three-level readout, and compare with simple strategies based on thresholding. This demonstration motivates further research towards adoption of reinforcement learning for real-time feedback control of quantum devices and more generally any physical system requiring learnable low-latency feedback control.

arXiv:2210.16715•Oct 30, 2022•3 citations

Hybrid Quantum-Classical Policy Gradient for Adaptive Control of Cyber-Physical Systems: A Comparative Study of VQC vs. MLP

Aueaphum Aueawatthanaphisut, Nyi Wunna Tun

The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.

arXiv:2510.06010•Oct 7, 2025•0 citations

Quantum Advantage Actor-Critic for Reinforcement Learning

Michael Kölle, Mohamad Hgog, Fabian Ritz +4 more

Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.

arXiv:2401.07043•International Conference on Agents and Artificial Intelligence (ICAART)•Jan 1, 2024•17 citations

Enhancement of Machine Learning for Adaptive Quantum Computing Control: The Optimization Performance and Prevention of Real Time Error

V. Saravanan, Ganesan M, R. Balasubramaniyan +3 more

Adaptive quantum computing control is important for the stability and efficiency of quantum systems, especially in the presence of dynamic environmental noise and hardware fluctuations. We present a unified strategy for machine learning (ML) techniques to improve adaptive quantum control in two main directions: optimization speed and out-of-sample error. Here, we propose a hybrid learning method combining reinforcement learning (RL), deep neural networks (DNNs), and quantum-inspired models to actively and adaptively update quantum gate parameters and error mitigation strategies on the fly. Embedding of quantum noise models directly into the learning allows the system to predict and correct for decoherence, gate errors and measurement inaccuracy before they affect computational fidelity. Using extensive simulations as well as experimental validation on noisy intermediate-scale quantum (NISQ) processors, our improved ML model achieves a substantially higher fidelity and robustness compared to standard model-free and static control approaches. We introduce a novel feedback-loop mechanism based on online learning algorithms enabling the system to adjust for changing noise profiles and hardware drift without retraining. The optimization part uses metaheuristic methods to quickly converge to control parameters that perform well, and the error prevention mechanism features anomaly detection and prediction correction layers to minimize the effects of cumulative decoherence. The results demonstrate that our adaptive control framework can be used to not only speed up the convergence in training quantum algorithms, but also sustain the performance of gates against error at a variety of error rates with the reduction of the real-time impact of the error up to 35 %. This work highlights the promise of machine learningaugmented quantum control for improving reliability and scalability in quantum computing and serves as a stepping stone for future incorporation with fault-tolerant designs.

International Conference on Electronics and Computing, Communication Networking Automation Technologies (ICEC2NT)•Sep 3, 2025•2 citations

Hardware-in-the-Loop Reinforcement Learning for Quantum Error Correction: A Systems-Engineering Approach to Latency-Aware Decoding

Yamamoto Kenji, Tanaka Hiroshi, Takeshi Saito

The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a $42\%$ reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.

•Jul 21, 2026•0 citations
Abstract The application of quantum reinforcement learning to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum–classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit. We demonstrate that a single-qubit agent acts as an effective learning model, and its performance is functionally comparable to a classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on noisy intermediate-scale quantum hardware. By bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables, we achieve more than an order-of-magnitude improvement in execution speed on the VTT Q5 processor. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.
The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.
To realize the full potential of quantum technologies, finding good strategies to control quantum information processing devices in real time becomes increasingly important. Usually these strategies require a precise understanding of the device itself, which is generally not available. Model-free reinforcement learning circumvents this need by discovering control strategies from scratch without relying on an accurate description of the quantum system. Furthermore, important tasks like state preparation, gate teleportation and error correction need feedback at time scales much shorter than the coherence time, which for superconducting circuits is in the microsecond range. Developing and training a deep reinforcement learning agent able to operate in this real-time feedback regime has been an open challenge. Here, we have implemented such an agent in the form of a latency-optimized deep neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit into a target state. To train the agent, we use model-free reinforcement learning that is based solely on measurement data. We study the agent's performance for strong and weak measurements, and for three-level readout, and compare with simple strategies based on thresholding. This demonstration motivates further research towards adoption of reinforcement learning for real-time feedback control of quantum devices and more generally any physical system requiring learnable low-latency feedback control.
The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.
Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.
Adaptive quantum computing control is important for the stability and efficiency of quantum systems, especially in the presence of dynamic environmental noise and hardware fluctuations. We present a unified strategy for machine learning (ML) techniques to improve adaptive quantum control in two main directions: optimization speed and out-of-sample error. Here, we propose a hybrid learning method combining reinforcement learning (RL), deep neural networks (DNNs), and quantum-inspired models to actively and adaptively update quantum gate parameters and error mitigation strategies on the fly. Embedding of quantum noise models directly into the learning allows the system to predict and correct for decoherence, gate errors and measurement inaccuracy before they affect computational fidelity. Using extensive simulations as well as experimental validation on noisy intermediate-scale quantum (NISQ) processors, our improved ML model achieves a substantially higher fidelity and robustness compared to standard model-free and static control approaches. We introduce a novel feedback-loop mechanism based on online learning algorithms enabling the system to adjust for changing noise profiles and hardware drift without retraining. The optimization part uses metaheuristic methods to quickly converge to control parameters that perform well, and the error prevention mechanism features anomaly detection and prediction correction layers to minimize the effects of cumulative decoherence. The results demonstrate that our adaptive control framework can be used to not only speed up the convergence in training quantum algorithms, but also sustain the performance of gates against error at a variety of error rates with the reduction of the real-time impact of the error up to 35 %. This work highlights the promise of machine learningaugmented quantum control for improving reliability and scalability in quantum computing and serves as a stepping stone for future incorporation with fault-tolerant designs.
The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a 42%42\%42% reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.