Towards Real-time Control of a CartPole System on a Quantum Computer

Abstract

The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.

The paper investigates whether a minimal quantum reinforcement-learning policy can learn CartPole and approach the latency requirements of real-time control. It combines simulation-based training, deployment-robustness tests, and inference on a superconducting QPU. The hybrid actor–critic uses a single-qubit circuit with three state-encoding rotations and one trainable rotation, followed by small classical networks. Across 50 seeded runs, the analytical-gradient hybrid reaches the criterion of 100 consecutive perfect episodes in $161.6\pm56.3$ episodes, versus $429.4\pm117.9$ for a classical actor–critic with larger networks. Finite-shot parameter-shift training also converges faster than the classical baseline, although more slowly than analytical-gradient training. A noisy-simulator study sweeps training and inference frequencies from 20 to 100 Hz and inference budgets of 128–1024 shots. Higher inference frequency and shot count generally improve balancing; at 100 Hz, all tested shot budgets give near-maximal 10-second durations. Training frequency has no consistent effect. On hardware, directly programming control electronics rather than using the standard software stack increases inference rates from approximately 0.14 Hz to 2.71–6.23 Hz, an $18.8$–$43.3\times$ speedup. Offline simulated CartPole episodes reach average scores of approximately 143, 166, 474, and 500 at 128, 256, 512, and 1024 shots, respectively. The hardware tests do not demonstrate real-time physical control: environment updates assume 50 Hz while actual inference remains slower. Stated limitations include the single-qubit, unentangled architecture and lack of scaling tests; the learning result is not a complexity-theoretic quantum speedup.