Quantum computing promises advantages over classical computing. The manufacturing of quantum hardware is in the infancy stage, called the Noisy Intermediate-Scale Quantum (NISQ) era. A major challenge is automated quantum circuit design that map a quantum circuit to gates in a universal gate set. In this paper, we present a generic MDP modeling and employ Q-learning and DQN algorithms for quantum circuit design. By leveraging the power of deep reinforcement learning, we aim to provide an automatic and scalable approach over traditional hand-crafted heuristic methods.
Quantum computing promises advantages over classical computing. The manufacturing of quantum hardware is in the infancy stage, called the Noisy Intermediate-Scale Quantum (NISQ) era. A major challenge is automated quantum circuit design that map a quantum circuit to gates in a universal gate set. In this paper, we present a generic MDP modeling and employ Q-learning and DQN algorithms for quantum circuit design. By leveraging the power of deep reinforcement learning, we aim to provide an automatic and scalable approach over traditional hand-crafted heuristic methods.
Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (RL) to improve quantum circuit design (QCD), which we formalize by a set of generic objectives. Furthermore, we propose qcd-gym, a concrete framework formalized as a Markov decision process, to enable learning policies capable of controlling a universal set of continuously parameterized quantum gates. Finally, we provide benchmark comparisons to assess the shortcomings and strengths of current state-of-the-art RL algorithms.
Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.
With the evolution of technology, the need to solve complex computational problems like machine learning and deep learning has shot up. However, even the most powerful classical supercomputers find it difficult to execute these tasks. Advancements in quantum computing are leading researchers and tech-giants who strive for better quantum circuits to do machine learning tasks. Current works on Quantum Machine Learning (QML) ensure less memory consumption and reduced model parameters. However, it is strenuous to simulate the classical deep learning approach on existing quantum computing devices due to the inflexibility of Deep quantum circuits. Consequently, designing viable quantum algorithms for QML for noisy intermediate-scale quantum (NISQ) devices is essential. The proposed work aims to explore Variational Quantum Circuits (VQC) for Deep Q network-based Reinforcement Learning by remodeling the target network and experience replay into a representation of VQC. In addition, to the reduction in model parameters, quantum information encoding schemes are used to achieve better results than classical neural networks. VQCs are employed for policyselection and decision-making reinforcement learning by approximating the deep Q-value function with target network and experience replay.
Quantum computing promises significant computational advantages over classical methods in certain applications but is currently too unstable for executing complex tasks. High-fidelity quantum circuits are critical to overcoming this challenge, yet they are computationally challenging to design. On current noisy intermediate-scale quantum (NISQ) hardware, shallower circuits with fewer two-qubit gates generally run with higher fidelity. Increased fidelity can be sought by sequentially rewriting the circuit, which involves a vast search space. This costly search space exploration could be circumvented by using reinforcement learning (RL) to teach an agent to rewrite the circuit in a sequential decision-making process. This thesis aims to implement a Deep Q-Network (DQN) algorithm to optimize BernsteinVazirani (BVZ) circuits. The DQN algorithm interacts with a Cirq-based environment. The environment provides valid rewrite actions to the DQN algorithm, which gains a scalar reward emphasizing depth and gate reduction. In the thesis we compare two different state encodings as input for the neural network: a compact state summarizing vector representation with three features, and an image-like encoding that maps circuit state topology with tokens through a fixed embedding. Both input encodings use the same action rewrite set and reward design. The network is implemented in PyTorch and trained with Adam optimizer, and runs on GPU via CUDA. Empirically, the agent shows promising initial optimizing performance. On several BVZ circuits with varying qubit count and topology, the agent drives the depth toward the theoretical optimum of three. The simplistic vector encoding already reaches reasonable circuit depth plateaus, with the image-like encoding together with hyperparameter tuning improving reliability, showing avenues for continued improvement. We conclude that DQN is a viable foundation for automated circuit optimization on BVZ circuits. To achieve reliable circuit optimization at scale, future research should investigate even richer encoding methods and reward function shaping. Finally, the results support applying the DQN to be applied to circuit families beyond BVZ.
Quantum circuit optimization is a foundational challenge in realizing practical quantum computation, requiring the reduction of gate counts and circuit depth while preserving logical equivalence across noisy intermediate-scale quantum (NISQ) devices. Conventional rule-based and heuristic compilation approaches demonstrate diminishing effectiveness as circuit complexity grows, motivating the search for adaptive, data-driven alternatives. This paper reformulates quantum circuit optimization as a Markov decision process (MDP), enabling deep reinforcement learning (RL) agents to autonomously discover efficient transformation strategies. A graph neural network (GNN)-based actor-critic architecture is developed to encode circuit topology and guide transformation selection. Drawing on empirical evidence from phase-structured quantum control tasks, neural feedback architectures, and convergence behavior in active learning agents, we demonstrate that the proposed framework achieves an average two-qubit gate count reduction of 31.4% and a circuit depth reduction of 27.8% across standard benchmark circuits, outperforming established compilation baselines. This work provides a principled theoretical foundation for the RL-as-optimizer paradigm and contributes an extensible experimental platform suited to NISQ-era constraints.
The rapid advancement of quantum computing (QC) and machine learning (ML) has given rise to the burgeoning field of quantum machine learning (QML), aiming to capitalize on the strengths of quantum computing to propel ML forward. Despite its promise, crafting effective QML models necessitates profound expertise to strike a delicate balance between model intricacy and feasibility on Noisy Intermediate-Scale Quantum (NISQ) devices. While complex models offer robust representation capabilities, their extensive circuit depth may impede seamless execution on extant noisy quantum platforms. In this paper, we address this quandary of QML model design by employing deep reinforcement learning to explore proficient QML model architectures tailored for designated supervised learning tasks. Specifically, our methodology involves training an RL agent to devise policies that facilitate the discovery of QML models without predetermined ansatz. Furthermore, we integrate an adaptive mechanism to dynamically adjust the learning objectives, fostering continuous improvement in the agent's learning process. Through extensive numerical simulations, we illustrate the efficacy of our approach within the realm of classification tasks. Our proposed method successfully identifies VQC architectures capable of achieving high classification accuracy while minimizing gate depth. This pioneering approach not only advances the study of AI-driven quantum circuit design but also holds significant promise for enhancing performance in the NISQ era.