Current experimental quantum computing devices are limited by noise, mainly originating from entangling gates. If an efficient gate sequence for an operation is unknown, one often employs layered parameterized quantum circuits, especially hardware-efficient ansätze, with fixed entangling layer structures. We demonstrate a reinforcement learning algorithm to improve on these by optimizing the entangling gate sequence in the task of quantum state preparation. This allows us to restrict the required number of CNOT gates while taking the qubit connectivity architecture into account. Recent advancements using reinforcement learning have already demonstrated the power of this technique when optimizing the circuit for a sequence of non-parameterized gates. We extend this approach to parameterized gate sets by incorporating general single-qubit unitaries, thus allowing us to consistently reach higher state preparation fidelities at the same number of CNOT gates compared to a hardware-efficient ansatz.
We extend directed quantum circuit synthesis (DQCS) with reinforcement learning from purely discrete gate selection to parameterized quantum state preparation with continuous single-qubit rotations
Rx,
Ry, and
Rz. We compare two training regimes: a one-stage agent that jointly selects the gate type, the affected qubit(s), and the rotation angle; and a two-stage variant that first proposes a discrete circuit and subsequently optimizes the rotation angles with Adam using parameter-shift gradients. Using Gymnasium and PennyLane, we evaluate Proximal Policy Optimization (PPO) and Advantage Actor--Critic (A2C) on systems comprising two to ten qubits and on targets of increasing complexity with
λ ranging from one to five. Whereas A2C does not learn effective policies in this setting, PPO succeeds under stable hyperparameters (one-stage: learning rate approximately
5×10−4 with a self-fidelity-error threshold of 0.01; two-stage: learning rate approximately
10−4). Both approaches reliably reconstruct computational basis states (between 83\% and 99\% success) and Bell states (between 61\% and 77\% success). However, scalability saturates for
λ of approximately three to four and does not extend to ten-qubit targets even at
λ=2. The two-stage method offers only marginal accuracy gains while requiring around three times the runtime. For practicality under a fixed compute budget, we therefore recommend the one-stage PPO policy, provide explicit synthesized circuits, and contrast with a classical variational baseline to outline avenues for improved scalability.