A hybrid quantum-classical learning agent based on projective simulation / Arne Hamann, BSc

Sabine Wölk

Oliver Sefrin, Sabine Wölk

Abstract The “hybrid agent for quantum-accessible reinforcement learning,” as defined in (Hamann and Wölk New J Phys 24:033044 2022), provides a proven quasi-quadratic speedup and is experimentally tested. However, the standard version can only be applied to episodic learning tasks with fixed episode length. In many real-world applications, the information about the necessary number of steps within an episode to reach a defined target is not available in advance and especially before reaching the target for the first time. Furthermore, in such scenarios, classical agents have the advantage of observing at which step they reach the target. How to best deal with an unknown target distance in classical and quantum reinforcement learning and whether the hybrid agent can provide an advantage in such learning scenarios is unknown so far. In this work, we introduce a hybrid agent with a stochastic episode length selection strategy to alleviate the need for knowledge about the necessary episode length. Through simulations, we test the adapted hybrid agent’s performance versus classical counterparts with and without similar episode selection strategies. Our simulations demonstrate a speedup in certain scenarios due to our developed episode length selection strategy for classical learning agents as well as an additional speedup for our resulting hybrid learning agent.

Youle Wang, Guangxi Li, Xin Wang

Sefrin, Oliver, Wölk, Sabine Esther

In Quantum Reinforcement Learning, the "hybrid agent for quantum-accessible reinforcement learning" (Hamann and Wölk, 2022) provides a quadratic speed-up in terms of sample complexity over classical algorithms. This hybrid agent may be used in deterministic and strictly episodic environments, for which the maze problem is a standard example. With the current algorithm, however, the episode length (i.e., the number of actions to be played in an episode) is a hyperparameter which needs to be set. For scenarios such as mazes with an unknown distance towards the goal, this poses a problem, since a feasible episode length value is not known initially. In this work, we propose an adaption to the hybrid algorithm that uses a variable episode length selection strategy, allowing its usage in a wider range of maze problem scenarios. We test our novel approach against classical agents in various maze scenarios. Finally, we reason about conditions for which a quantum advantage persists.

Lian Peng, Meikang Qiu A, Chong Li +1 more

Annette Zapf, Sabine Wölk

Pairwise sequence alignment is central in bioinformatics for analyzing DNA, RNA, and proteins, revealing similarities that inform evolutionary relationships, gene or protein functions, and disease mechanisms. Classical dynamic programming guarantees optimal alignments but scales poorly, while heuristics reduce runtime at the cost of optimality and still require high resources. Quantum algorithms exploit superposition and entanglement to accelerate optimization or find exact solutions faster. We present a hybrid quantum reinforcement learning (RL) agent based on (Hamann & Wölk, 2022), optimized for NISQ hardware. The agent learns a policy mapping states to action sequences from feedback of an environment. Using a Grover-like subroutine, it evaluates all sequences in parallel, reinforcing those leading to a solution, which reduces environment interactions and can achieve a quadratic speedup over classical RL. To enable execution on current hardware, we introduce a novel Parameterized Quantum Policy embedding entire action sequences, combined with a simplified Grover routine using a black-box oracle. This reduces resource demands but requires prior knowledge and retraining per alignment, limiting generalization. We also outline design ideas for a more general Grover oracle and circuit with broader generalization, which, while not NISQ-compatible, suggest a promising direction for future work.