This paper investigates quantum control problems using tabular Q-learning. A modified tabular Q-learning algorithm based on dynamic greedy method is proposed and the proposed algorithm succeeds for finding control sequences to drive a two-qubit system to a given target state with high fidelity. The modified algorithm also shows improved performance over the traditional Q-learning for solving quantum control problems on continuous states space. Moreover, the modified tabular Q-learning algorithm is compared with stochastic gradient descent and Krotov algorithms for solving quantum control problems. Simulation results on a two-qubit system demonstrate the effectiveness of the proposed algorithm.
This paper investigates quantum control problems using tabular Q-learning. A modified tabular Q-learning algorithm based on dynamic greedy method is proposed and the proposed algorithm succeeds for finding control sequences to drive a two-qubit system to a given target state with high fidelity. The modified algorithm also shows improved performance over the traditional Q-learning for solving quantum control problems on continuous states space. Moreover, the modified tabular Q-learning algorithm is compared with stochastic gradient descent and Krotov algorithms for solving quantum control problems. Simulation results on a two-qubit system demonstrate the effectiveness of the proposed algorithm.
Quantum control refers to the manipulation of dynamical quantum systems to force them to complete given tasks such as preparing a desired state and tracking a designed trajectory. We consider the state preparation problem of a two-qubit closed quantum system from an initial to a desired state. The aim is to achieve a high fidelity in a given fixed time with limited control resources. The Deep Q-learning (DQL) for solving the quantum state preparation problem is explored in this paper. We propose a novel semi-Markov DQL algorithm based on a modified action selection procedure and improved replay memory to enhance performance of standard DQL algorithm. The proposed algorithm shows high performance for discovering high-fidelity control protocols and for converging to a good policy, compared with standard DQL. The proposed modifications enhance the exploration-exploitation ability for DQL agent and the robustness for solving quantum control problem with high-fidelity at different numbers of control steps. Numerical results on a two-qubit closed system show effectiveness of the proposed algorithm.
Abstract Reinforcement learning has been widely used in many problems, including quantum control of qubits. However, such problems can, at the same time, be solved by traditional, non-machine-learning methods, such as stochastic gradient descent and Krotov algorithms, and it remains unclear which one is most suitable when the control has specific constraints. In this work, we perform a comparative study on the efficacy of three reinforcement learning algorithms: tabular Q-learning, deep Q-learning, and policy gradient, as well as two non-machine-learning methods: stochastic gradient descent and Krotov algorithms, in the problem of preparing a desired quantum state. We found that overall, the deep Q-learning and policy gradient algorithms outperform others when the problem is discretized, e.g. allowing discrete values of control, and when the problem scales up. The reinforcement learning algorithms can also adaptively reduce the complexity of the control sequences, shortening the operation time and improving the fidelity. Our comparison provides insights into the suitability of reinforcement learning in quantum control problems.
Reinforcement learning (RL) is considered a powerful technology with the potential to revolutionize quantum control. However, the application effectiveness of traditional RL is often limited by some insurmountable experimental conditions. Thus, developing new RL algorithms that can efficiently manipulate the quantum system dynamics is a crucial task. Prior research has shown that incorporating quantum mechanical properties into RL can improve learning performance. In this article, we consider the quantum control problem where only the target state can be accurately identified and introduce a quantum-inspired RL (QiRL) method. In particular, we propose a quantum-inspired exploration strategy to replace a commonly used$\epsilon $-greedy strategy, as well as a quantum-inspired reward scheme to incentivize the learning agent. Numerical results on three quantum system control problems, i.e., one-qubit closed quantum system, two-level open quantum system, and many-qubit closed quantum system, verify the effectiveness of QiRL. Comparison results show that the proposed QiRL outperforms existing RL algorithms (deep Q-network and proximal policy optimization) in terms of stability and efficiency for solving quantum control problems.
The fast preparation of a quantum state with high fidelity is a key problem for a quantum system. We propose two modified algorithms based on the cutting-edge reinforcement learning methods for the flip from an initial state to a target one in a quantum spin system. The balance between exploration and exploitation and the size of a state space are key factors for reinforcement learning methods. In the first algorithm, we propose a modified ϵ-greedy strategy instead of the ε-greedy strategy to balance exploration and exploitation. Furthermore, we use the fidelity of the final state as the reward and leverage piecewise-constant driving protocols in every duration. The results show that the rate of correction of this algorithm is greater than that of Q-learning with ε-greedy strategy. However, it will still cost much time resource if the rotation angle from a state to the next state is too small. Thus, we initialize the state-action value table to reduce the state space in the second modified Q-learning algorithm (MQL), which can be found that the efficiency of learning is greatly improved.
Due to the large state space of the two-qubit system, and the adoption of ladder reward function in the existing quantum state preparation methods, the convergence speed is slow and it is difficult to prepare the desired target quantum state with high fidelity under limited conditions. To solve the above problems, a difference-driven reinforcement learning (RL) algorithm for quantum state preparation of two-qubit system is proposed by improving the reward function and action selection strategy. Firstly, a model is constructed for the problem of preparing quantum states of a two-qubit system, with restrictions on the type of quantum gates and the time for quantum state evolution. In the preparation process, a weighted differential dynamic reward function is designed to assist the algorithm quickly obtain the maximum expected cumulative reward. Then, an adaptive [Formula: see text]-greedy action selection strategy is adopted to achieve a balance between exploration and utilization to a certain extent, thereby improving the fidelity of the final quantum state. The simulation results show that the proposed algorithm can prepare quantum state with high fidelity under limited conditions. Compared with other algorithms, it has different degrees of improvement in convergence speed and fidelity of the final quantum state.
Reinforcement learning has been intensively applied to tackle complex quantum control problems owing to its adaptability in dynamic environments. However, commonly used reinforcement learning algorithms are restricted to selecting actions from a discrete action space, which may result in inaccurate control. In order to surpass this constraint, we propose an improved continuous reinforcement learning algorithm that can generate control policies in a continuous action space, enabling precise control of quantum systems. Moreover, a guided reward function design method is proposed to guide the learning process toward higher fidelity. Numerical results demonstrate that the proposed continuous reinforcement learning algorithm with a guided reward function can capably prepare states on one-qubit and two-qubit systems.