Hybrid Quantum Reinforcement Learning for Sequence Alignment (QLearning)

Annette Zapf, Sabine Wölk

Published Oct 6, 2025• 0 citations

Abstract

Pairwise sequence alignment is central in bioinformatics for analyzing DNA, RNA, and proteins, revealing similarities that inform evolutionary relationships, gene or protein functions, and disease mechanisms. Classical dynamic programming guarantees optimal alignments but scales poorly, while heuristics reduce runtime at the cost of optimality and still require high resources. Quantum algorithms exploit superposition and entanglement to accelerate optimization or find exact solutions faster. We present a hybrid quantum reinforcement learning (RL) agent based on (Hamann & Wölk, 2022), optimized for NISQ hardware. The agent learns a policy mapping states to action sequences from feedback of an environment. Using a Grover-like subroutine, it evaluates all sequences in parallel, reinforcing those leading to a solution, which reduces environment interactions and can achieve a quadratic speedup over classical RL. To enable execution on current hardware, we introduce a novel Parameterized Quantum Policy embedding entire action sequences, combined with a simplified Grover routine using a black-box oracle. This reduces resource demands but requires prior knowledge and retraining per alignment, limiting generalization. We also outline design ideas for a more general Grover oracle and circuit with broader generalization, which, while not NISQ-compatible, suggest a promising direction for future work.

Related papers

Symbiotic Quantum-Classical Hybrid Learning for Reinforcement Learning

Jincheng Zhang

This paper proposes a novel symbiotic learning framework that synergistically integrates quantum computing and classical reinforcement learning (RL). The core idea is to leverage the unique capabilities of quantum systems – specifically, their ability to efficiently represent and explore high-dimensional state spaces – alongside the established strengths of classical RL algorithms for policy optimization. We introduce a hybrid architecture where a quantum neural network (QNN) acts as a sophisticated state processor, generating candidate actions based on the current state, while a classical RL agent learns to refine and select the optimal actions. Our approach moves beyond quantum-inspired algorithms by establishing a genuine symbiotic relationship between quantum and classical components, leading to potentially faster learning rates and improved performance in complex RL environments. The framework is described mathematically, outlining the key components and interactions. We present a detailed mathematical formulation of the system, highlighting the contributions of both the quantum and classical components. The efficacy of the system is conceptually discussed, focusing on the benefits of combining quantum exploration with classical policy learning. This paper lays the groundwork for future research in this rapidly evolving field.

Sep 3, 20260 citations

Reinforcement-Learning-Based Variational Quantum Circuits Optimization for Combinatorial Problems

Sami Khairy, Ruslan Shaydulin, Łukasz Cincio +2 more

Quantum computing exploits basic quantum phenomena such as state superposition and entanglement to perform computations. The Quantum Approximate Optimization Algorithm (QAOA) is arguably one of the leading quantum algorithms that can outperform classical state-of-the-art methods in the near term. QAOA is a hybrid quantum-classical algorithm that combines a parameterized quantum state evolution with a classical optimization routine to approximately solve combinatorial problems. The quality of the solution obtained by QAOA within a fixed budget of calls to the quantum computer depends on the performance of the classical optimization routine used to optimize the variational parameters. In this work, we propose an approach based on reinforcement learning (RL) to train a policy network that can be used to quickly find high-quality variational parameters for unseen combinatorial problem instances. The RL agent is trained on small problem instances which can be simulated on a classical computer, yet the learned RL policy is generalizable and can be used to efficiently solve larger instances. Extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our trained RL policy can reduce the optimality gap by a factor up to 8.61 compared with other off-the-shelf optimizers tested.

arXiv:1911.04574Nov 11, 201921 citations

Performance analysis of a hybrid agent for quantum-accessible reinforcement learning

Arne Hamann, Sabine Wölk

Abstract In the last decade quantum machine learning has provided fascinating and fundamental improvements to supervised, unsupervised and reinforcement learning (RL). In RL, a so-called agent is challenged to solve a task given by some environment. The agent learns to solve the task by exploring the environment and exploiting the rewards it gets from the environment. For some classical task environments, an analogue quantum environment can be constructed which allows to find rewards quadratically faster by applying quantum algorithms. In this paper, we analytically analyze the behavior of a hybrid agent which combines this quadratic speedup in exploration with the policy update of a classical agent. This leads to a faster learning of the hybrid agent compared to the classical agent. We demonstrate that if the classical agent needs on average ⟨J⟩ rewards and ⟨T⟩cl epochs to learn how to solve the task, the hybrid agent will take ⟨ T ⟩ q ⩽ α s α o ⟨ T ⟩ c l ⟨ J ⟩ epochs on average. Here, α s and α o denote constants depending on details of the quantum search and are independent of the problem size. Additionally, we prove that if the environment allows for maximally α o k max sequential coherent interactions, e.g. due to noise effects, an improvement given by ⟨T⟩q ≈ α o ⟨T⟩cl/(4k max) is still possible.

arXiv:2107.14001New Journal of PhysicsMar 1, 202211 citations

Quantum-Inspired Algorithms for Reinforcement Learning Policy Optimization

Jincheng Zhang

This paper investigates the potential of quantum-inspired algorithms to improve reinforcement learning (RL) policy optimization. Traditional RL methods often struggle with slow convergence and exploration challenges, particularly in complex environments. We propose a novel approach that draws inspiration from quantum mechanics, specifically superposition and entanglement, to address these limitations. Our algorithms aim to exploit superposition to simultaneously evaluate multiple action possibilities and utilize entanglement to model correlations between states, thereby accelerating the learning process. We present a framework for incorporating these concepts into RL, focusing on algorithmic design rather than a direct quantum simulation. The theoretical analysis suggests that our approach could significantly enhance exploration efficiency and, consequently, policy convergence rates. This work represents a direct application of quantum-inspired ideas to RL policy optimization, offering a potentially transformative pathway for developing more efficient and robust RL agents.

Sep 7, 20260 citations

Hybrid Action Reinforcement Learning for Quantum Architecture Search

Jiayang Niu, Yan Wang, Jie Li +4 more

Reinforcement learning-based Quantum Architecture Search (QAS) offers a promising avenue for automating the design of variational quantum circuits, but existing methods typically decouple discrete structure search from continuous parameter optimization, resulting in inefficient or brittle solutions. We propose HyRLQAS (Hybrid-Action Reinforcement Learning for Quantum Architecture Search), a unified reinforcement learning framework that jointly learns gate placement and parameter initialization within a hybrid discrete-continuous action space, while enabling dynamic refinement of previously placed gates. Trained in a variational quantum eigensolver setting, the agent constructs circuits that directly optimize molecular ground-state energies. Across multiple molecular benchmarks, HyRLQAS demonstrates strong and competitive performance against state-of-the-art QAS methods, achieving lower energy errors with fewer gates. Notably, HyRLQAS reaches chemical-accuracy-level convergence down to 1e-8 energy error after classical optimization, and policy-guided initialization reduces the iteration count of downstream classical optimizers. These results demonstrate that hybrid-action reinforcement learning provides a principled and effective mechanism for coupling circuit topology design with optimization-aware parameterization.

quant-pharXiv:2511.04967Nov 10, 20250 citations

Q-Policy: Quantum-Enhanced Policy Evaluation for Scalable Reinforcement Learning

Cherukuri, Kalyan, Lala, Aarav, Yash Yardi

We propose Q-Policy, a hybrid quantum-classical reinforcement learning (RL) framework that mathematically accelerates policy evaluation and optimization by exploiting quantum computing primitives. Q-Policy encodes value functions in quantum superposition, enabling simultaneous evaluation of multiple state-action pairs via amplitude encoding and quantum parallelism. We introduce a quantum-enhanced policy iteration algorithm with provable polynomial reductions in sample complexity for the evaluation step, under standard assumptions. To demonstrate the technical feasibility and theoretical soundness of our approach, we validate Q-Policy on classical emulations of small discrete control tasks. Due to current hardware and simulation limitations, our experiments focus on showcasing proof-of-concept behavior rather than large-scale empirical evaluation. Our results support the potential of Q-Policy as a theoretical foundation for scalable RL on future quantum devices, addressing RL scalability challenges beyond classical approaches.

arXiv:2505.11862May 17, 20250 citations