Energies

On the Use of Quantum Reinforcement Learning in Energy-Efficiency Scenarios

Eva Andrés, Manuel Pegalájar Cuéllar, Gabriel Navarro

Published Aug 19, 2022• 28 citations

Abstract

In the last few years, deep reinforcement learning has been proposed as a method to perform online learning in energy-efficiency scenarios such as HVAC control, electric car energy management, or building energy management, just to mention a few. On the other hand, quantum machine learning was born during the last decade to extend classic machine learning to a quantum level. In this work, we propose to study the benefits and limitations of quantum reinforcement learning to solve energy-efficiency scenarios. As a testbed, we use existing energy-efficiency-based reinforcement learning simulators and compare classic algorithms with the quantum proposal. Results in HVAC control, electric vehicle fuel consumption, and profit optimization of electrical charging stations applications suggest that quantum neural networks are able to solve problems in reinforcement learning scenarios with better accuracy than their classical counterpart, obtaining a better cumulative reward with fewer parameters to be learned.

DOI: 10.3390/en15166034

Related papers

Reinforcement Learning with Deep Quantum Neural Networks

Wei Hu, James Lee Hu

The advantage of quantum computers over classical computers fuels the recent trend of developing machine learning algorithms on quantum computers, which can potentially lead to breakthroughs and new learning models in this area. The aim of our study is to explore deep quantum reinforcement learning (RL) on photonic quantum computers, which can process information stored in the quantum states of light. These quantum computers can naturally represent continuous variables, making them an ideal platform to create quantum versions of neural networks. Using quantum photonic circuits, we implement Q learning and actor-critic algorithms with multilayer quantum neural networks and test them in the grid world environment. Our experiments show that 1) these quantum algorithms can solve the RL problem and 2) compared to one layer, using three layer quantum networks improves the learning of both algorithms in terms of rewards collected. In summary, our findings suggest that having more layers in deep quantum RL can enhance the learning outcome.

Journal of Quantum Information ScienceJan 1, 201913 citations

A framework for deep energy-based reinforcement learning with quantum speed-up

Sofiène Jerbi, Hendrik Poulsen Nautrup, Lea M. Trenkwalder +2 more

In the past decade, deep learning methods have seen tremendous success in various supervised and unsupervised learning tasks such as classification and generative modeling. More recently, deep neural networks have emerged in the domain of reinforcement learning as a tool to solve decision-making problems of unprecedented complexity, e.g., navigation problems or game-playing AI. Despite the successful combinations of ideas from quantum computing with machine learning methods, there have been relatively few attempts to design quantum algorithms that would enhance deep reinforcement learning. This is partly due to the fact that quantum enhancements of deep neural networks, in general, have not been as extensively investigated as other quantum machine learning methods. In contrast, projective simulation is a reinforcement learning model inspired by the stochastic evolution of physical systems that enables a quantum speed-up in decision making. In this paper, we develop a unifying framework that connects deep learning and projective simulation, opening the route to quantum improvements in deep reinforcement learning. Our approach is based on so-called generative energy-based models to design reinforcement learning methods with a computational advantage in solving complex and large-scale decision-making problems.

Oct 28, 20194 citations

From Classical to Quantum: A Review of Recent Progress in Reinforcement Learning

Kunal Kashyap, Lalit, Daksh Shah +1 more

Standard machine learning algorithms have taken shape well due to increase in computational power. There is a special focus on reinforcement learning, which offers solutions to concepts like autonomous vehicles and robotics, among others. On the other hand, research in the fields of quantum computing and associated machine learning has catapulted in the last few years. This work analytically compares and contrasts between the classical and quantum approaches to two reinforcement learning problems, and tries to put focus on future works to advance both fields further.

International Conference for Emerging Technology (INCET)May 21, 20214 citations

Efficient Dimensionality Reduction Strategies for Quantum Reinforcement Learning

Eva Andrés, Manuel Pegalájar Cuéllar, Gabriel Navarro

Quantum neural networks constitute one of the most promising applications of Quantum Machine Learning, as they leverage both the capabilities of classical neural networks and the unique advantages of quantum mechanics. Moreover, quantum mechanics has demonstrated its ability to detect atypical patterns in data that are challenging for classical approaches to recognize. However, despite their potential, there are still open questions such as barren plateau phenomenon and the challenges of scalability and the curse of dimensionality, which become particularly relevant in Reinforcement Learning (RL) when working in environments with high-dimensional state and action spaces. This study delves into the critical realm of representing classical data as quantum states, a topic of keen interest across the scientific community. The aim is to construct streamlined circuits for efficient execution on quantum computers and simulators using minimal qubits and entanglement gates to evade barren plateau phenomena and reducing computational times. Our investigation examines and validates the efficacy of three strategies for data management and dimensionality reduction in real-world, large-scale environments for Quantum Reinforcement Learning, particularly in energy efficiency scenarios. The techniques encompass amplitude encoding, linear layer preprocessing, and data reuploading, supplemented by trainable parameters. This research sheds light on the potential of quantum machine learning in enhancing real-world environments, including energy efficiency scenarios and showcases the capabilities of quantum neural networks in the reinforcement learning landscape.

IEEE AccessJan 1, 202326 citations

Control of Qubit Dynamics Using Reinforcement Learning

Dimitris Koutromanos, Dionisis Stefanatos, Emmanuel Paspalakis

The progress in machine learning during the last decade has had a considerable impact on many areas of science and technology, including quantum technology. This work explores the application of reinforcement learning (RL) methods to the quantum control problem of state transfer in a single qubit. The goal is to create an RL agent that learns an optimal policy and thus discovers optimal pulses to control the qubit. The most crucial step is to mathematically formulate the problem of interest as a Markov decision process (MDP). This enables the use of RL algorithms to solve the quantum control problem. Deep learning and the use of deep neural networks provide the freedom to employ continuous action and state spaces, offering the expressivity and generalization of the process. This flexibility helps to formulate the quantum state transfer problem as an MDP in several different ways. All the developed methodologies are applied to the fundamental problem of population inversion in a qubit. In most cases, the derived optimal pulses achieve fidelity equal to or higher than 0.9999, as required by quantum computing applications. The present methods can be easily extended to quantum systems with more energy levels and may be used for the efficient control of collections of qubits and to counteract the effect of noise, which are important topics for quantum sensing applications.

InformationMay 11, 202411 citations

Brain-Inspired Agents for Quantum Reinforcement Learning

Eva Andrés, Manuel Pegalájar Cuéllar, Gabriel Navarro

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

MathematicsApr 19, 20248 citations