IEEE International Conference on Quantum Software (QSW)

Deep Reinforcement Learning Strategies for Noise-Adaptive Qubit Routing

G. Indiveri A.M. Pascoal, João Paulo Fernandes, Rui Abreu

Published Jul 7, 2024• 3 citations

Abstract

Qubit routing is the process of transforming quantum circuits to ensure that all two-qubit operations obey the connectivity constraints of a target architecture. The severe limitations of NISQ computers make qubit routing a vital compilation step. We propose a method to route quantum circuits based on deep reinforcement learning (RL). Moreover, we augment our approach with device calibration data to enable routing decisions based on the reliability of individual gates and we improve upon a previous technique for encoding circuits as a numerical matrix. We also implement enhancements such as BRIDGE operations, qubit embeddings, and commutation analysis. Using proximal policy optimization (PPO), we trained agents for five IBM topologies with between 5 and 27 qubits. For random circuits, the proposed strategy reduced the number of additional two-qubit gates by up to 37.3% and improved estimated success probability by up to 26.8% compared to the best algorithm from the Qiskit framework. For a dataset of realistic circuits, we achieved up to 27.5% and 15.2% improvements in gate count and circuit fidelity, respectively. We found no significant advantage from using calibration-aware models for IBM architectures, which deserves further exploration. Our work helps cement RL as a valuable qubit routing strategy, serving as another stepping stone toward practical applications for quantum computing.

DOI: 10.1109/qsw62656.2024.00030

Related papers

Graph Reinforcement Learning for Calibration-Aware Quantum Circuit Routing

Yash Vardhan Tomar, Dheeraj Peddireddy

Quantum circuit routing is a key step in compiling programs for noisy intermediate-scale quantum processors, particularly superconducting devices whose sparse fixed coupling makes routing a central compilation cost. Routes that appear efficient by standard overhead metrics such as SWAP count, routed two-qubit count, and depth can still lose fidelity when they pass through poorly calibrated couplers. We study a calibration-aware graph reinforcement-learning router that uses same-day calibration data from superconducting IBM Heron r2 processors to choose hardware-edge SWAPs. We train the policy with proximal policy optimization and evaluate it with exact simulated fidelity across nine Munich Quantum Toolkit (MQT) Bench circuits and three calibration snapshots. Across these evaluations, pooled mean exact fidelity is 0.727, compared with 0.440 for SWAP-based bidirectional heuristic search (SABRE)-best20 and 0.481 for target-aware SABRE. We observe that fidelity gains come with higher routed two-qubit counts and are concentrated in 5 qubit and 8 qubit circuit families; under the fixed tree action graph, all 10 qubit families favor SABRE-best20. Overall, our results show that calibration-aware learned routing can improve fidelity beyond gate-count-driven compilation, by roughly 0.25 to 0.29 in absolute mean fidelity over the SABRE-family baselines.

quant-pharXiv:2606.12816Jun 12, 20260 citations

Rethinking Quantum Circuit Optimization as a Reinforcement Learning Problem

Arjun Mehta, Sven Richter

Quantum circuit optimization is a foundational challenge in realizing practical quantum computation, requiring the reduction of gate counts and circuit depth while preserving logical equivalence across noisy intermediate-scale quantum (NISQ) devices. Conventional rule-based and heuristic compilation approaches demonstrate diminishing effectiveness as circuit complexity grows, motivating the search for adaptive, data-driven alternatives. This paper reformulates quantum circuit optimization as a Markov decision process (MDP), enabling deep reinforcement learning (RL) agents to autonomously discover efficient transformation strategies. A graph neural network (GNN)-based actor-critic architecture is developed to encode circuit topology and guide transformation selection. Drawing on empirical evidence from phase-structured quantum control tasks, neural feedback architectures, and convergence behavior in active learning agents, we demonstrate that the proposed framework achieves an average two-qubit gate count reduction of 31.4% and a circuit depth reduction of 27.8% across standard benchmark circuits, outperforming established compilation baselines. This work provides a principled theoretical foundation for the RL-as-optimizer paradigm and contributes an extensible experimental platform suited to NISQ-era constraints.

American Journal Of Big DataApr 1, 20260 citations

Quantum Architecture Search via Continual Reinforcement Learning

Esther Ye, Samuel Yen-Chi Chen

Quantum computing has promised significant improvement in solving difficult computational tasks over classical computers. Designing quantum circuits for practical use, however, is not a trivial objective and requires expert-level knowledge. To aid this endeavor, this paper proposes a machine learning-based method to construct quantum circuit architectures. Previous works have demonstrated that classical deep reinforcement learning (DRL) algorithms can successfully construct quantum circuit architectures without encoded physics knowledge. However, these DRL-based works are not generalizable to settings with changing device noises, thus requiring considerable amounts of training resources to keep the RL models up-to-date. With this in mind, we incorporated continual learning to enhance the performance of our algorithm. In this paper, we present the Probabilistic Policy Reuse with deep Q-learning (PPR-DQL) framework to tackle this circuit design challenge. By conducting numerical simulations over various noise patterns, we demonstrate that the RL agent with PPR was able to find the quantum gate sequence to generate the two-qubit Bell state faster than the agent that was trained from scratch. The proposed framework is general and can be applied to other quantum gate synthesis or control problems -- including the automatic calibration of quantum devices.

arXiv:2112.05779Dec 10, 202115 citations

Quantum circuit optimization using deep reinforcement learning: Applications across multiple gate sets

Khoa Dang Tao, Sumin Jin, Muhammad Raza +1 more

The practical implementation of quantum algorithms on noisy intermediate-scale quantum devices encounters operational limitations due to decoherence and other sources of noise inherent in real hardware. To mitigate these errors while preserving the original functionality of the algorithm, shorter quantum circuits are therefore preferred. This motivates the development of effective quantum circuit optimization algorithms. Learning-based approaches have emerged as a leading candidate, yet existing autonomous agents remain inefficient, spending most of their training capacity rediscovering elementary reductions that deterministic rule-based methods already handle reliably. To address this challenge, we propose a reinforcement learning framework that embeds a deterministic Commutation-and-Reduction (CR) algorithm directly into the training environment. After every agent action, the CR algorithm automatically resolves elementary commutations and cancellations, enabling the agent to focus its learning capacity on the non-trivial optimizations where reinforcement learning adds real value. Empirical evaluation across two gate sets, the universal Clifford+T basis and the CNOT+Pauli basis, shows that RL+CR produces shorter circuits than a standard RL agent at all tested scales. We demonstrate that RL trained on smaller quantum circuits can be applied to larger quantum circuits. On 20-qubit Clifford+T circuits, five times larger than the training circuits, RL+CR removes twice as many gates as standard RL. This work provides a robust approach that could accelerate the compilation and optimization processes for future fault-tolerant and utility-scale quantum systems.

quant-pharXiv:2608.19103Aug 20, 20260 citations

Reinforcement Learning Based Quantum Circuit Optimization via ZX-Calculus

Jordi Riu, Jan Nogué, Gerard Vilaplana +2 more

We propose a novel Reinforcement Learning (RL) method for optimizing quantum circuits using graph-theoretic simplification rules of ZX-diagrams. The agent, trained using the Proximal Policy Optimization (PPO) algorithm, employs Graph Neural Networks to approximate the policy and value functions. We demonstrate the capacity of our approach by comparing it against the best performing ZX-Calculus-based algorithm for the problem in hand. After training on small Clifford+T circuits of 5-qubits and few tenths of gates, the agent consistently improves the state-of-the-art for this type of circuits, for at least up to 80-qubit and 2100 gates, whilst remaining competitive in terms of computational performance. Additionally, we illustrate the versatility of the agent by incorporating additional optimization routines on the workflow during training, improving the two-qubit gate count state-of-the-art on multiple structured quantum circuits for relevant applications of much larger dimension and different gate distributions than the circuits the agent trains on. This conveys the potential of tailoring the reward function to the specific characteristics of each application and hardware backend. Our approach is a valuable tool for the implementation of quantum algorithms in the near-term intermediate-scale range (NISQ).

arXiv:2312.11597QuantumMay 28, 202515 citations

Reinforcement Learning and DEAR Framework for Solving the Qubit Mapping Problem

ChingYao Huang, Chi-Hsiang Lien, Wai-Kei Mak

Quantum computing is gaining more and more attention due to its huge potential and the constant progress in quantum computer development. IBM and Google have released quantum architectures with more than 50 qubits. However, in these machines, the physical qubits are not fully connected so that two-qubit interaction can only be performed between specific pairs of the physical qubits. To execute a quantum circuit, it is necessary to transform it into a functionally equivalent one that respects the constraints imposed by the target architecture. Quantum circuit transformation inevitably introduces additional gates which reduces the fidelity of the circuit. Therefore, it is important that the transformation method completes the transformation with minimal overheads. It consists of two steps, initial mapping and qubit routing. Here we propose a reinforcement learning-based model to solve the initial mapping problem. Initial mapping is formulated as sequence-to-sequence learning and self-attention network is used to extract features from a circuit. For qubit routing, a DEAR (Dynamically-Extract-and-Route) framework is proposed. The framework iteratively extracts a subcircuit and uses A* search to determine when and where to insert additional gates. It helps to preserve the lookahead ability dynamically and to provide more accurate cost estimation efficiently during A* search. The experimental results show that our RL-model generates better initial mappings than the best known algorithms with 12% fewer additional gates in the qubit routing stage. Furthermore, our DEAR-framework outperforms the state-of-the-art qubit routing approach with 8.4% and 36.3% average reduction in the number of additional gates and execution time starting from the same initial mapping.

International Conference on Computer Aided DesignOct 30, 202215 citations