We report an experimental implementation of deep reinforcement learning (RL) for optimizing the loading stage of a 6 Li magneto-optical trap (MOT) in a high-dimensional continuous control space. An off-policy actor-critic agent observes in-situ fluorescence images at 111 ms intervals and updates seven experimental parameters over 18 decision steps. Learning is performed directly on the apparatus using a sparse terminal reward R = N × A extracted from an absorption image, where N is the atom number, and A is the peak optical depth. We benchmark Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) across three independent 300-episode training runs under identical alignment. Both algorithms surpass the human-optimized (HO) baseline, but the central finding concerns reliability rather than peak performance. In deterministic replay validation, the three SAC waveforms show reward coefficients of variation of 9–14%, and the minimum reward of each waveform exceeds the HO mean reward. The DDPG waveforms show up to three times the dispersion, and even the best-performing DDPG run contains a catastrophic near-zero shot. The SAC solutions are nearly time-independent and consistently converge to a blue-detuned repump laser with an elevated field gradient, reminiscent of compressed-MOT operating conditions. Our results show that off-policy actor-critic RL can autonomously optimize high-dimensional laser-cooling sequences in an operating ultracold-atom apparatus, and that SAC has a reproducibility advantage over DDPG in a real noisy environment.
We report an experimental implementation of deep reinforcement learning (RL) for optimizing the loading stage of a 6 Li magneto-optical trap (MOT) in a high-dimensional continuous control space. An off-policy actor-critic agent observes in-situ fluorescence images at 111 ms intervals and updates seven experimental parameters over 18 decision steps. Learning is performed directly on the apparatus using a sparse terminal reward R = N × A extracted from an absorption image, where N is the atom number, and A is the peak optical depth. We benchmark Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) across three independent 300-episode training runs under identical alignment. Both algorithms surpass the human-optimized (HO) baseline, but the central finding concerns reliability rather than peak performance. In deterministic replay validation, the three SAC waveforms show reward coefficients of variation of 9–14%, and the minimum reward of each waveform exceeds the HO mean reward. The DDPG waveforms show up to three times the dispersion, and even the best-performing DDPG run contains a catastrophic near-zero shot. The SAC solutions are nearly time-independent and consistently converge to a blue-detuned repump laser with an elevated field gradient, reminiscent of compressed-MOT operating conditions. Our results show that off-policy actor-critic RL can autonomously optimize high-dimensional laser-cooling sequences in an operating ultracold-atom apparatus, and that SAC has a reproducibility advantage over DDPG in a real noisy environment.
Abstract We demonstrate a reinforcement learning (RL)-based control framework for optimizing evaporative cooling in the preparation of strongly interacting degenerate Fermi gases of 6Li. Using a Soft Actor-Critic (SAC) algorithm, the system autonomously explores a high-dimensional parameter space to learn optimal cooling trajectories. Compared to conventional exponential ramps, our method achieves up to 130% improvement in atomic density within 0.5 second, revealing non-trivial control strategies that balance fast evaporation and thermalization. While our current optimization focuses on the evaporation stage, future integration of other cooling stages, such as gray molasses cooling, could further extend RL to the full preparation pipeline. Our result highlights the promise of RL as a general tool for closed-loop quantum control and automated calibration in complex atomic physics experiments.
One of the multiple facets of quantum reinforcement learning (QRL) is enhancing reinforcement learning (RL) algorithms with quantum submodules, namely with variational quantum circuits (VQC) as function approx-imators. QRL solutions are empirically proven to require fewer training iterations or adjustable parameters than their classical counterparts, but are usually restricted to applications that have a discrete action space and thus limited industrial relevance. We propose a hybrid quantum-classical (HQC) deep deterministic policy gradient (DDPG) approach for a robot to navigate through a maze using continuous states, continuous actions and using local observations from the robot’s LiDAR sensors. We show that this HQC method can lead to models of comparable test results to the neural network (NN)-based DDPG algorithm, that need around 200 times fewer weights. We also study the scalability of our solution with respect to the number of VQC layers and qubits, and find that in general results improve as the layer and qubit counts increase. The best rewards among all similarly sized HQC and classical DDPG methods correspond to a VQC of 8 qubits and 5 layers with no other NN. This work is another step towards continuous QRL, where literature has been sparse.
Quantum optimal control in the presence of decoherence is difficult, particularly when not all Hamiltonian parameters are known precisely, as in quantum sensing applications. In this context, maximizing the sensitivity of the system is the objective, for which the optimal target state or unitary transformations are unknown, especially in the case of multiparameter estimation. Here we investigate the use of reinforcement learning (RL), specifically the soft actor-critic (SAC) algorithm, for problems in quantum optimal control. We adopt a spin-based magnetometer as a benchmarking system for the efficacy of the SAC algorithm. In such systems, the magnitude of a background magnetic field acting on a spin can be determined via projective measurements. The precision of the determined magnitude can be optimized by applying pulses of transverse fields with different strengths. We train an RL agent on numerical simulations of the spin system to determine a transverse-field pulse sequence that optimizes the precision and compare it to existing sensing strategies. We evaluate the agent's performance against various Hamiltonian parameters, including values not seen in training, to investigate the agent's ability to generalize to different situations. We find that the RL agents are sensitive to certain parameters of the system, such as the pulse duration and the purity of the initial state, but overall are able to generalize well, supporting the use of RL in quantum optimal control settings.
We present a Multi-task Soft Actor-Critic (SAC) Reinforcement Learning framework designed for open-system quantum control across diverse Hamiltonians, which learns optimal pulse sequences while simultaneously discovering problem-specific evolution time T and number of control pulse segments N. Experimental results across 51 Hamiltonian variations demonstrate that the multi-task SAC model is able to generate control pulses that can drive a system, under environment noise, from its initial state to its target state with high fidelities, establishing essential foundations for universal quantum control applicable to realistic noisy quantum devices. Through progressive expansion of the training Hamiltonian set, we investigate if a single multi-task model trained using a given number of sample Hamiltonians can successfully accomplish state-transfer tasks for Hamiltonians drawn from the same Hamiltonian space but not encountered during training. In addition, our Robustness Infidelity Measure (RIM) analysis reveals that SAC trained policies exhibit superior robustness to pulse amplitude perturbations and decoherence rate variations compared to GRAPE-optimized controls.
Abstract Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic (A-C) scheme for continuous state-action spaces is developed based on the deep deterministic policy gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing (QA), both simulated and with D-Wave QA hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research. Among others, the hybrid A-C agent is evaluated on the actual electron beam line of the Advanced Wakefield Experiment (AWAKE).
Classical reinforcement learning (RL) methods often struggle in complex, high-dimensional environments because of their extensive parameter requirements and challenges posed by stochastic, non-deterministic settings. This study introduces quantum deep reinforcement learning (QDRL) to train humanoid agents efficiently. While previous quantum RL models focused on smaller environments, such as wheeled robots and robotic arms, our work pioneers the application of QDRL to humanoid robotics, specifically in environments with substantial observation and action spaces, such as MuJoCo's Humanoid-v4 and Walker2d-v4. Using parameterized quantum circuits, we explored a hybrid quantum-classical setup to directly navigate high-dimensional state spaces, bypassing traditional mapping and planning. By integrating quantum computing with deep RL, we aim to develop models that can efficiently learn complex navigation tasks in humanoid robots. We evaluated the performance of the Soft Actor-Critic (SAC) in classical RL against its quantum implementation. The results show that the quantum SAC achieves an 8% higher average return (246.40) than the classical SAC (228.36) after 92% fewer steps, highlighting the accelerated learning potential of quantum computing in RL tasks.