Quantum computing promises significant computational advantages over classical methods in certain applications but is currently too unstable for executing complex tasks. High-fidelity quantum circuits are critical to overcoming this challenge, yet they are computationally challenging to design. On current noisy intermediate-scale quantum (NISQ) hardware, shallower circuits with fewer two-qubit gates generally run with higher fidelity. Increased fidelity can be sought by sequentially rewriting the circuit, which involves a vast search space. This costly search space exploration could be circumvented by using reinforcement learning (RL) to teach an agent to rewrite the circuit in a sequential decision-making process. This thesis aims to implement a Deep Q-Network (DQN) algorithm to optimize BernsteinVazirani (BVZ) circuits. The DQN algorithm interacts with a Cirq-based environment. The environment provides valid rewrite actions to the DQN algorithm, which gains a scalar reward emphasizing depth and gate reduction. In the thesis we compare two different state encodings as input for the neural network: a compact state summarizing vector representation with three features, and an image-like encoding that maps circuit state topology with tokens through a fixed embedding. Both input encodings use the same action rewrite set and reward design. The network is implemented in PyTorch and trained with Adam optimizer, and runs on GPU via CUDA. Empirically, the agent shows promising initial optimizing performance. On several BVZ circuits with varying qubit count and topology, the agent drives the depth toward the theoretical optimum of three. The simplistic vector encoding already reaches reasonable circuit depth plateaus, with the image-like encoding together with hyperparameter tuning improving reliability, showing avenues for continued improvement. We conclude that DQN is a viable foundation for automated circuit optimization on BVZ circuits. To achieve reliable circuit optimization at scale, future research should investigate even richer encoding methods and reward function shaping. Finally, the results support applying the DQN to be applied to circuit families beyond BVZ.
Quantum computing promises significant computational advantages over classical methods in certain applications but is currently too unstable for executing complex tasks. High-fidelity quantum circuits are critical to overcoming this challenge, yet they are computationally challenging to design. On current noisy intermediate-scale quantum (NISQ) hardware, shallower circuits with fewer two-qubit gates generally run with higher fidelity. Increased fidelity can be sought by sequentially rewriting the circuit, which involves a vast search space. This costly search space exploration could be circumvented by using reinforcement learning (RL) to teach an agent to rewrite the circuit in a sequential decision-making process. This thesis aims to implement a Deep Q-Network (DQN) algorithm to optimize BernsteinVazirani (BVZ) circuits. The DQN algorithm interacts with a Cirq-based environment. The environment provides valid rewrite actions to the DQN algorithm, which gains a scalar reward emphasizing depth and gate reduction. In the thesis we compare two different state encodings as input for the neural network: a compact state summarizing vector representation with three features, and an image-like encoding that maps circuit state topology with tokens through a fixed embedding. Both input encodings use the same action rewrite set and reward design. The network is implemented in PyTorch and trained with Adam optimizer, and runs on GPU via CUDA. Empirically, the agent shows promising initial optimizing performance. On several BVZ circuits with varying qubit count and topology, the agent drives the depth toward the theoretical optimum of three. The simplistic vector encoding already reaches reasonable circuit depth plateaus, with the image-like encoding together with hyperparameter tuning improving reliability, showing avenues for continued improvement. We conclude that DQN is a viable foundation for automated circuit optimization on BVZ circuits. To achieve reliable circuit optimization at scale, future research should investigate even richer encoding methods and reward function shaping. Finally, the results support applying the DQN to be applied to circuit families beyond BVZ.
Quantum circuit optimization is a foundational challenge in realizing practical quantum computation, requiring the reduction of gate counts and circuit depth while preserving logical equivalence across noisy intermediate-scale quantum (NISQ) devices. Conventional rule-based and heuristic compilation approaches demonstrate diminishing effectiveness as circuit complexity grows, motivating the search for adaptive, data-driven alternatives. This paper reformulates quantum circuit optimization as a Markov decision process (MDP), enabling deep reinforcement learning (RL) agents to autonomously discover efficient transformation strategies. A graph neural network (GNN)-based actor-critic architecture is developed to encode circuit topology and guide transformation selection. Drawing on empirical evidence from phase-structured quantum control tasks, neural feedback architectures, and convergence behavior in active learning agents, we demonstrate that the proposed framework achieves an average two-qubit gate count reduction of 31.4% and a circuit depth reduction of 27.8% across standard benchmark circuits, outperforming established compilation baselines. This work provides a principled theoretical foundation for the RL-as-optimizer paradigm and contributes an extensible experimental platform suited to NISQ-era constraints.
A reinforcement learning (RL) framework is introduced for the efficient synthesis of quantum circuits that generate specified target quantum states from a fixed initial state, addressing a central challenge in both the Noisy Intermediate-Scale Quantum (NISQ) era and future fault-tolerant quantum computing. The approach utilizes tabular Q-learning, based on action sequences, within a discretized quantum state space, to effectively manage the exponential growth of the space dimension. The framework introduces a hybrid reward mechanism, combining a static, domain-informed reward that guides the agent toward the target state with customizable dynamic penalties that discourage inefficient circuit structures such as gate congestion and redundant state revisits. This is a circuit-aware reward, in contrast to the current trend of works on this topic, which are primarily fidelity-based. By leveraging sparse matrix representations and state-space discretization, the method enables practical navigation of high-dimensional environments while minimizing computational overhead. Benchmarking on graph-state preparation tasks for up to seven qubits, we demonstrate that the algorithm consistently discovers minimal-depth circuits with optimized gate counts. Moreover, extending the framework to a universal gate set still yields low depth circuits, highlighting the algorithm’s robustness and adaptability. The results confirm that this RL-driven approach, with our completely circuit-aware method, efficiently explores the complex quantum state space and synthesizes near-optimal quantum circuits, providing a resource-efficient foundation for quantum circuit optimization.
The practical implementation of quantum algorithms on noisy intermediate-scale quantum devices encounters operational limitations due to decoherence and other sources of noise inherent in real hardware. To mitigate these errors while preserving the original functionality of the algorithm, shorter quantum circuits are therefore preferred. This motivates the development of effective quantum circuit optimization algorithms. Learning-based approaches have emerged as a leading candidate, yet existing autonomous agents remain inefficient, spending most of their training capacity rediscovering elementary reductions that deterministic rule-based methods already handle reliably. To address this challenge, we propose a reinforcement learning framework that embeds a deterministic Commutation-and-Reduction (CR) algorithm directly into the training environment. After every agent action, the CR algorithm automatically resolves elementary commutations and cancellations, enabling the agent to focus its learning capacity on the non-trivial optimizations where reinforcement learning adds real value. Empirical evaluation across two gate sets, the universal Clifford+T basis and the CNOT+Pauli basis, shows that RL+CR produces shorter circuits than a standard RL agent at all tested scales. We demonstrate that RL trained on smaller quantum circuits can be applied to larger quantum circuits. On 20-qubit Clifford+T circuits, five times larger than the training circuits, RL+CR removes twice as many gates as standard RL. This work provides a robust approach that could accelerate the compilation and optimization processes for future fault-tolerant and utility-scale quantum systems.
A significant hurdle in the noisy intermediate-scale quantum (NISQ) era is identifying functional quantum circuits. These circuits must also adhere to the constraints imposed by current quantum hardware limitations. Variational quantum algorithms (VQAs), a class of quantum-classical optimization algorithms, were developed to address these challenges in the currently available quantum devices. However, the overall performance of VQAs depends on the initialization strategy of the variational circuit, the structure of the circuit (also known as ansatz), and the configuration of the cost function. Focusing on the structure of the circuit, in this thesis, we improve the performance of VQAs by automating the search for an optimal structure for the variational circuits using reinforcement learning (RL). Within the thesis, the optimality of a circuit is determined by evaluating its depth, the overall count of gates and parameters, and its accuracy in solving the given problem. The task of automating the search for optimal quantum circuits is known as quantum architecture search (QAS). The majority of research in QAS is primarily focused on a noiseless scenario. Yet, the impact of noise on the QAS remains inadequately explored. In this thesis, we tackle the issue by introducing a tensor-based quantum circuit encoding, restrictions on environment dynamics to explore the search space of possible circuits efficiently, an episode halting scheme to steer the agent to find shorter circuits, a double deep Q-network (DDQN) with an ε-greedy policy for better stability. The numerical experiments on noiseless and noisy quantum hardware show that in dealing with various VQAs, our RL-based QAS outperforms existing QAS. Meanwhile, the methods we propose in the thesis can be readily adapted to address a wide range of other VQAs.
We propose a novel Reinforcement Learning (RL) method for optimizing quantum circuits using graph-theoretic simplification rules of ZX-diagrams. The agent, trained using the Proximal Policy Optimization (PPO) algorithm, employs Graph Neural Networks to approximate the policy and value functions. We demonstrate the capacity of our approach by comparing it against the best performing ZX-Calculus-based algorithm for the problem in hand. After training on small Clifford+T circuits of 5-qubits and few tenths of gates, the agent consistently improves the state-of-the-art for this type of circuits, for at least up to 80-qubit and 2100 gates, whilst remaining competitive in terms of computational performance. Additionally, we illustrate the versatility of the agent by incorporating additional optimization routines on the workflow during training, improving the two-qubit gate count state-of-the-art on multiple structured quantum circuits for relevant applications of much larger dimension and different gate distributions than the circuits the agent trains on. This conveys the potential of tailoring the reward function to the specific characteristics of each application and hardware backend. Our approach is a valuable tool for the implementation of quantum algorithms in the near-term intermediate-scale range (NISQ).
This paper presents a novel approach to quantum circuit compilation leveraging reinforcement learning (RL). The core idea is to train an RL agent to dynamically optimize quantum circuit structures, aiming to minimize gate count and enhance overall circuit performance. Traditional quantum circuit compilation methods often rely on static heuristics and pre-defined optimization strategies, which may not be optimal for all quantum algorithms or hardware platforms. Our approach addresses this limitation by allowing the RL agent to learn a policy that adapts to the specific characteristics of the algorithm and the target quantum hardware. The agent learns by interacting with a simulated quantum circuit environment, receiving rewards based on circuit fidelity, depth, and gate count. The results demonstrate the potential of RL to significantly improve the efficiency and robustness of quantum computations. The key components of the system are: a high-level quantum algorithm representation, an RL agent, a quantum circuit simulator, and a reward function. The agent's policy is represented as a neural network, enabling it to generalize across different algorithms and hardware. We demonstrate the effectiveness of this approach through simulations, showcasing its ability to outperform conventional compilation methods in terms of reducing circuit depth and gate count while maintaining high fidelity.