The realization of large-scale quantum computers requires not only quantum error correction (QEC) but also fault-tolerant operations to handle errors that propagate into harmful errors. Recently, flag-based protocols have been introduced that use ancillary qubits to flag harmful errors. However, there is no clear recipe for finding a fault-tolerant quantum circuit with flag-based protocols, especially when we consider hardware constraints, such as qubit connectivity and available gate set. In this work, we propose and explore reinforcement learning (RL) to automatically discover compact and hardware-adapted fault-tolerant quantum circuits. We show that in the task of fault-tolerant logical state preparation, RL discovers circuits with fewer gates and ancillary qubits than published results without and with hardware constraints of up to 15 physical qubits. Furthermore, RL allows for straightforward exploration of different qubit connectivities and the use of transfer learning to accelerate the discovery. More generally, our work opens the door towards the use of RL for the discovery of fault-tolerant quantum circuits for addressing tasks beyond state preparation, including magic state preparation, logical gate synthesis, or syndrome measurement.
The realization of large-scale quantum computers requires not only quantum error correction (QEC) but also fault-tolerant operations to handle errors that propagate into harmful errors. Recently, flag-based protocols have been introduced that use ancillary qubits to flag harmful errors. However, there is no clear recipe for finding a fault-tolerant quantum circuit with flag-based protocols, especially when we consider hardware constraints, such as qubit connectivity and available gate set. In this work, we propose and explore reinforcement learning (RL) to automatically discover compact and hardware-adapted fault-tolerant quantum circuits. We show that in the task of fault-tolerant logical state preparation, RL discovers circuits with fewer gates and ancillary qubits than published results without and with hardware constraints of up to 15 physical qubits. Furthermore, RL allows for straightforward exploration of different qubit connectivities and the use of transfer learning to accelerate the discovery. More generally, our work opens the door towards the use of RL for the discovery of fault-tolerant quantum circuits for addressing tasks beyond state preparation, including magic state preparation, logical gate synthesis, or syndrome measurement.
The realization of large-scale quantum computers requires not only quantum error correction but also fault-tolerant (FT) operations to handle errors that propagate into harmful errors. Recently, flag-based protocols have been introduced that use ancillary qubits to flag harmful errors. However, there is no clear recipe for finding a FT quantum circuit with flag-based protocols, especially when we consider hardware constraints, such as the qubit connectivity and available gate set. This work presents a novel approach to automatically discover compact and hardware-adapted FT quantum circuits to make significant progress towards scalable FT quantum computing. We employ reinforcement learning (RL) as an enabling tool, leveraging a fast, parallelized stabilizer quantum circuit simulator and a nontrivial reward function specifically adapted to the problem. We show that, in the task of FT logical state preparation, RL discovers not only circuits with fewer gates and ancillary qubits than published results but also novel circuits without and with hardware constraints of up to distance-5 codes with 25 physical qubits, and they can be implemented directly in experiments. Furthermore, RL allows for straightforward exploration of different qubit connectivities and the use of transfer learning to accelerate the discovery. More generally, our work sets the framework towards the use of RL or other machine learning techniques for FT quantum circuit discovery with hardware constraints to make real progress towards the realization of large-scale quantum computers, addressing tasks beyond state preparation, including magic state preparation, logical gate synthesis, and syndrome measurement.
Abstract In the ongoing race towards experimental implementations of quantum error correction (QEC), finding ways to automatically discover codes and encoding strategies tailored to the qubit hardware platform is emerging as a critical problem. Reinforcement learning (RL) has been identified as a promising approach, but so far it has been severely restricted in terms of scalability. In this work, we significantly expand the power of RL approaches to QEC code discovery. Explicitly, we train an RL agent that automatically discovers both QEC codes and their encoding circuits for a given gate set, qubit connectivity and error model, from scratch. This is enabled by a reward based on the Knill-Laflamme conditions and a vectorized Clifford simulator, showing its effectiveness with up to 25 physical qubits and distance 5 codes, while presenting a roadmap to scale this approach to 100 qubits and distance 10 codes in the near future. We also introduce the concept of a noise-aware meta-agent, which learns to produce encoding strategies simultaneously for a range of noise models, thus leveraging transfer of insights between different situations. Our approach opens the door towards hardware-adapted accelerated discovery of QEC approaches across the full spectrum of quantum hardware platforms of interest.
Quantum error correction is widely thought to be the key to fault-tolerant quantum computation. However, determining the most suited encoding for unknown error channels or specific laboratory setups is highly challenging. Here, we present a reinforcement learning framework for optimizing and fault-tolerantly adapting quantum error correction codes. We consider a reinforcement learning agent tasked with modifying a family of surface code quantum memories until a desired logical error rate is reached. Using efficient simulations with about 70 data qubits with arbitrary connectivity, we demonstrate that such a reinforcement learning agent can determine near-optimal solutions, in terms of the number of data qubits, for various error models of interest. Moreover, we show that agents trained on one setting are able to successfully transfer their experience to different settings. This ability for transfer learning showcases the inherent strengths of reinforcement learning and the applicability of our approach for optimization from off-line simulations to on-line laboratory settings.
The practical implementation of quantum algorithms on noisy intermediate-scale quantum devices encounters operational limitations due to decoherence and other sources of noise inherent in real hardware. To mitigate these errors while preserving the original functionality of the algorithm, shorter quantum circuits are therefore preferred. This motivates the development of effective quantum circuit optimization algorithms. Learning-based approaches have emerged as a leading candidate, yet existing autonomous agents remain inefficient, spending most of their training capacity rediscovering elementary reductions that deterministic rule-based methods already handle reliably. To address this challenge, we propose a reinforcement learning framework that embeds a deterministic Commutation-and-Reduction (CR) algorithm directly into the training environment. After every agent action, the CR algorithm automatically resolves elementary commutations and cancellations, enabling the agent to focus its learning capacity on the non-trivial optimizations where reinforcement learning adds real value. Empirical evaluation across two gate sets, the universal Clifford+T basis and the CNOT+Pauli basis, shows that RL+CR produces shorter circuits than a standard RL agent at all tested scales. We demonstrate that RL trained on smaller quantum circuits can be applied to larger quantum circuits. On 20-qubit Clifford+T circuits, five times larger than the training circuits, RL+CR removes twice as many gates as standard RL. This work provides a robust approach that could accelerate the compilation and optimization processes for future fault-tolerant and utility-scale quantum systems.
Reinforcement Learning (RL) has established itself as a powerful tool for designing quantum circuits, which are essential for processing quantum information. RL applications have typically focused on circuits of small to intermediate complexity, as computation times tend to increase exponentially with growing circuit complexity. This computational explosion severely limits the scalability of RL and casts significant doubt on its broader applicability. In this paper, we propose a principled approach based on the systematic discovery and introduction of composite gates -- {\it gadgets}, that enables RL scalability, thereby expanding its potential applications. As a case study, we explore the discovery of Clifford encoders for Quantum Error Correction. We demonstrate that incorporating gadgets in the form of composite Clifford gates, in addition to standard CNOT and Hadamard gates, significantly enhances the efficiency of RL agents. Specifically, the computation speed increases (by one or even two orders of magnitude), enabling RL to discover highly complex quantum codes without previous knowledge. We illustrate this advancement with examples of QEC code discovery with parameters $ [[n,1,d]] $ for $ d \leq 7 $ and $ [[n,k,6]] $ for $ k \leq 7 $. We note that the most complicated circuits of these classes were not previously found. We highlight the advantages and limitations of the gadget-based approach. Our method paves the way for scaling the RL-based automatic discovery of complicated quantum circuits for various tasks, which may include designing logical operations between logical qubits or discovering quantum algorithms.
Quantum error correction (QEC) is a crucial element in the realization of fault-tolerant quantum computation. However, traditional QEC schemes often rely on pre-determined, static parameters, which may not be optimally suited for the specific noise characteristics present in a given quantum hardware. This work introduces a novel approach to QEC that leverages reinforcement learning (RL) to dynamically adapt the error correction protocol in real-time. The system learns to optimize the parameters of the QEC code, such as measurement schedules and decoding strategies, based on continuous feedback from the quantum hardware. This adaptive approach promises to significantly improve the performance and efficiency of QEC, reducing the overhead associated with error correction and enhancing the fidelity of quantum computations. The core claim of this paper is that quantum error correction is computationally intensive and requires precise control over quantum systems. The central mechanism involves employing reinforcement learning to optimize the parameters of quantum error correction protocols in real-time, adapting to the specific noise characteristics of the quantum hardware. This represents a new automation of the QEC process.