State preparation is currently the only means to provide input data for quantum algorithm, but finding the shortest possible sequence of gates to prepare a given state is not trivial. We approach this problem using reinforcement learning (RL), first on an agent that is trained to only prepare a single fixed quantum state. Despite the overhead of training a whole network to just produce one single data point, gradient-based backpropagation appears competitive to genetic algorithms in this scenario and single state preparation thus seems a worthwhile task. In a second case we then train a single network to prepare arbitrary quantum states to some degree of success, despite a complete lack of structure in the training data set. In both cases we find that training is severely improved by using QR decomposition to automatically map the agents' outputs to unitary operators to solve the problem of sparse rewards that usually makes this task challenging.
State preparation is currently the only means to provide input data for quantum algorithm, but finding the shortest possible sequence of gates to prepare a given state is not trivial. We approach this problem using reinforcement learning (RL), first on an agent that is trained to only prepare a single fixed quantum state. Despite the overhead of training a whole network to just produce one single data point, gradient-based backpropagation appears competitive to genetic algorithms in this scenario and single state preparation thus seems a worthwhile task. In a second case we then train a single network to prepare arbitrary quantum states to some degree of success, despite a complete lack of structure in the training data set. In both cases we find that training is severely improved by using QR decomposition to automatically map the agents' outputs to unitary operators to solve the problem of sparse rewards that usually makes this task challenging.
Quantum computers may solve some computing tasks more efficiently than classical computers, but to do so requires the design of appropriate quantum circuits. However, it is hard to design even simple quantum programs. Recent results have shown that deep learning augmented search has the potential to discover good circuits for a problem, but it is not yet sufficiently understood how well deep learning is able to model such a complex domain: the search space and output space grow exponentially. This work explores the ability of neural networks to encode quantum circuits for tasks that require knowledge about the unitary representation of the circuit. To this end, we trained neural networks to directly learn to predict the unitary of a circuit and applied reinforcement learning to train neural networks to solve circuit based quantum state preparation. This work finds that encoding quantum circuits is quite difficult especially with regards to the amount of entanglement applied in both application domains. The use of intermediate states and structure in quantum circuits combined with a reasonable inductive bias, where applicable, can alleviate these problems to some extent. The use of intermediate states also greatly improves the results of Deep Reinforcement Learning for quantum state preparation. Based on these results, we discuss the next set of challenges to address if we are to design neural network-based approaches for the automatic generation of quantum circuits. Add sentence on how it improves things in the end.
One of the most important initial steps in quantum computing is high-fidelity quantum state preparation, because errors in it can affect the final computational result. Therefore, one of the major challenges is to design a quantum circuit that can achieve the desired target state accurately with a smaller number of gates. Traditionally, circuits are designed using predefined rules and mathematical decompositions, but this makes it difficult to design circuits for different and complex target states. This paper presents TATVA, a reinforcement learning (RL) system for synthesizing quantum circuits. TATVA views circuit synthesis as a sequential decision problem in which the agents select one gate at a time and calculate the fidelity after each applied gate, with the gate producing the highest fidelity selected for the circuit. The system introduces a parallel architecture that uses two RL agents, Deep Q-Network (DQN) and Proximal Policy Optimization (PPO), along with Qiskit's statevector simulator. The system can synthesize up to 5 qubits while achieving a fidelity of 0.999999 and producing compact circuits. After achieving the desired fidelity, it performs a post-hoc optimization step that reduces the circuit depth while preserving the achieved fidelity. The circuit is finalized using fidelity, success rate, gate count, and circuit depth. Target states that are not used during training are used to test whether the system has learned effectively and can generalize beyond the training states. This approach aims to enable automatic quantum circuit synthesis with high fidelity using reinforcement learning.
Quantum computers promise significant computational advantages over classical computers for specific tasks. However, one of the challenges in the noisy intermediate-scale quantum (NISQ) era is the efficient compilation of quantum states and gates, which is crucial for the implementation of quantum algorithms. Given the high dimensionality of quantum systems, this compilation task is non-trivial. Reinforcement learning (RL) is a machine learning subfield focused on developing algorithms for decision-making in dynamic environments. An agent learns by interacting with its environment, how to act to optimize its performance on a certain task. Recently, the integration of deep learning has given rise to deep reinforcement learning (DRL), where deep neural networks approximate functions like value or policy, allowing agents to tackle high-dimensional and complex tasks with greater efficiency and success. Recently, deep reinforcement learning (DRL) has been proposed as a method to automate quantum compiling. In this thesis, we explore the application of RL for quantum state and unitary matrix synthesis, focusing on the use of two reinforcement learning algorithms Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO). Our results indicate that DRL shows promise in generalizing the task of quantum unitary synthesis and quantum state preparation. However, the current state-of-the-art is still limited to working in the no-volume law entangled part of the Hilbert space.
Current experimental quantum computing devices are limited by noise, mainly originating from entangling gates. If an efficient gate sequence for an operation is unknown, one often employs layered parameterized quantum circuits, especially hardware-efficient ansätze, with fixed entangling layer structures. We demonstrate a reinforcement learning algorithm to improve on these by optimizing the entangling gate sequence in the task of quantum state preparation. This allows us to restrict the required number of CNOT gates while taking the qubit connectivity architecture into account. Recent advancements using reinforcement learning have already demonstrated the power of this technique when optimizing the circuit for a sequence of non-parameterized gates. We extend this approach to parameterized gate sets by incorporating general single-qubit unitaries, thus allowing us to consistently reach higher state preparation fidelities at the same number of CNOT gates compared to a hardware-efficient ansatz.
The practical implementation of quantum algorithms on noisy intermediate-scale quantum devices encounters operational limitations due to decoherence and other sources of noise inherent in real hardware. To mitigate these errors while preserving the original functionality of the algorithm, shorter quantum circuits are therefore preferred. This motivates the development of effective quantum circuit optimization algorithms. Learning-based approaches have emerged as a leading candidate, yet existing autonomous agents remain inefficient, spending most of their training capacity rediscovering elementary reductions that deterministic rule-based methods already handle reliably. To address this challenge, we propose a reinforcement learning framework that embeds a deterministic Commutation-and-Reduction (CR) algorithm directly into the training environment. After every agent action, the CR algorithm automatically resolves elementary commutations and cancellations, enabling the agent to focus its learning capacity on the non-trivial optimizations where reinforcement learning adds real value. Empirical evaluation across two gate sets, the universal Clifford+T basis and the CNOT+Pauli basis, shows that RL+CR produces shorter circuits than a standard RL agent at all tested scales. We demonstrate that RL trained on smaller quantum circuits can be applied to larger quantum circuits. On 20-qubit Clifford+T circuits, five times larger than the training circuits, RL+CR removes twice as many gates as standard RL. This work provides a robust approach that could accelerate the compilation and optimization processes for future fault-tolerant and utility-scale quantum systems.
The quantum state generation problem is a major research goal of quantum control and quantum variational algorithms, which use iterative optimization methods to evolve the initial state to the target state. The Twin Delayed Deep Deterministic Policy (TD3) algorithm in reinforcement learning achieves high learning efficiency and better stability in continuous control tasks. Here, using the TD3, we propose a new quantum state preparation method that does not require case-by-case optimization, to find a suitable evolution path to obtain the desired state. Specifically, we input the initial state into the trained actor-network, which can output the parameters of the unitary gates step by step, thus gradually evolving the initial state to the fixed quantum state. According to the reversibility of the unitary transformation, we can obtain a sequence of unitary gates to evolve the fixed state to the desired state. To verify the effectiveness of the algorithm, we perform simulations for one-qubit, two-qubit, and four-qubit cases, and the results show that the trained actor-network can provide appropriate unitary transformations to obtain the fixed state.