Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

International Green and Sustainable Computing Conference

Quantum Reinforcement Learning for Air Conditioning Control Based on Variational Quantum Circuits

Harshitta Gandhi, Zhipeng Deng

Published Jun 19, 2026• 0 citations
Publisher

Abstract

We present a novel approach integrating quantum neural networks with reinforcement learning (RL) for house air conditioning control. Our framework employs a quantum deep Q-network (QDQN) RL implemented with PennyLane, leveraging parameterized variational quantum circuits (VQC) as function approximators for Q-values. The agent interacts with simulated indoor thermodynamic modelling, incorporating thermal capacity, heat transfer, and outdoor temperature variations. The action space consists of discrete air conditioning power levels, and the reward function balances comfort, energy, and switching penalties. To stabilize training, we introduce a target network, experience replay, and epsilon-greedy exploration. QDQN encodes the state vector into quantum rotations and applies multi-layer entangling gates, providing a potentially richer representation of the environment compared to classical networks. The QDQN agent uses VQC architecture. It incorporates data re-uploading, parameterized rotations, and strongly entangled qubits to effectively capture inter-feature correlations. We demonstrate the algorithm on episodes with outdoor conditions in both winter and summer conditions, showing convergence to policies that maintain indoor temperatures within setpoint bounds with zero violation rate. To validate the framework in a real-world noise setting, the quantum pipeline is simulated on the IBM Marrakesh system, featuring a 156-qubit Heron r2 processor with a median two-qubit (CZ) error of approximately 2.59 × 10− 3. Our results highlight the promise of quantum RL for real-world controls.

DOI
10.1145/3797248.3816053

Related papers

Variational Quantum Circuit Design for Quantum Reinforcement Learning on Continuous Environments

Georg Kruse, Theodora-Augustina Drăgan, Robert Wille +1 more

Quantum Reinforcement Learning (QRL) emerged as a branch of reinforcement learning (RL) that uses quantum submodules in the architecture of the algorithm. One branch of QRL focuses on the replacement of neural networks (NN) by variational quantum circuits (VQC) as function approximators. Initial works have shown promising results on classical environments with discrete action spaces, but many of the proposed architectural design choices of the VQC lack a detailed investigation. Hence, in this work we investigate the impact of VQC design choices such as angle embedding, encoding block architecture and postprocessesing on the training capabilities of QRL agents. We show that VQC design greatly influences training performance and heuristically derive enhancements for the analyzed components. Additionally, we show how to design a QRL agent in order to solve classical environments with continuous action spaces and benchmark our agents against classical feed-forward NNs.

arXiv:2312.13798•International Conference on Agents and Artificial Intelligence (ICAART)•Jan 1, 2024•8 citations

Implementation of Quantum Deep Reinforcement Learning Using Variational Quantum Circuits

S Lokes, C. Sakthi Jay Mahenthar, S Parvatha Kumaran +2 more

With the evolution of technology, the need to solve complex computational problems like machine learning and deep learning has shot up. However, even the most powerful classical supercomputers find it difficult to execute these tasks. Advancements in quantum computing are leading researchers and tech-giants who strive for better quantum circuits to do machine learning tasks. Current works on Quantum Machine Learning (QML) ensure less memory consumption and reduced model parameters. However, it is strenuous to simulate the classical deep learning approach on existing quantum computing devices due to the inflexibility of Deep quantum circuits. Consequently, designing viable quantum algorithms for QML for noisy intermediate-scale quantum (NISQ) devices is essential. The proposed work aims to explore Variational Quantum Circuits (VQC) for Deep Q network-based Reinforcement Learning by remodeling the target network and experience replay into a representation of VQC. In addition, to the reduction in model parameters, quantum information encoding schemes are used to achieve better results than classical neural networks. VQCs are employed for policyselection and decision-making reinforcement learning by approximating the deep Q-value function with target network and experience replay.

International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT)•Oct 13, 2022•19 citations

Variational Quantum Circuits for Deep Reinforcement Learning

Samuel Yen-Chi Chen, Chao-Han Huck Yang, Jun Qi +3 more

The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep Q-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.

arXiv:1907.00397•IEEE Access•Jan 1, 2020•453 citations

Quantum-Inspired Reinforcement Learning Controller for DC–DC Power Converters Supplying Constant Power Loads

Ali Masoudian, Uffe Jakobsen, Mohammad Hassan Khooban

This paper proposes a quantum neural network (QNN)-based reinforcement learning (RL) controller for the real-time control of DC–DC buck converters supplying constant power loads (CPLs). Unlike conventional RL approaches that rely on classical neural networks, the proposed method employs a parameterized quantum circuit to represent the control policy, enabling a compact and expressive function approximator. The QNN is implemented as a classically simulated variational quantum circuit; no physical quantum hardware or quantum computational speedup is claimed. It is trained offline using a deterministic policy gradient framework and subsequently deployed in a real-time control loop under strict sampling constraints. The proposed 36-parameter QNN actor reduces trainable parameters by 85% versus a computation-matched classical RL baseline (245 parameters), while achieving sub-millisecond settling times and near-zero steady-state error across reference tracking, load disturbance, and supply variation scenarios validated on a dSPACE DS1202 hardware platform at 100-kHz switching frequency. Small-signal stability analysis at three tested operating points confirms phase margins exceeding 30° and a gain margin that is theoretically infinite at those points, as the linearized phase does not cross −180°. The results highlight the ability of the QNN to capture complex system dynamics with fewer trainable parameters while maintaining stable closed-loop operation, providing, to the best of the authors’ knowledge, a first step toward integrating quantum machine learning concepts into real-time power electronics control.

IEEE Access•Jan 1, 2026•1 citation

QCNN-Inspired Variational Circuits for Enhanced Noise Robustness in Quantum Deep Q-Learning

Louyang Yu, Wenbin Yu, Yadang Chen +1 more

Quantum reinforcement learning (QRL) is often evaluated under idealized, noiseless assumptions, yet realistic quantum devices inevitably introduce noise that can severely degrade performance. This paper improves the robustness of quantum deep Q-learning (QDQN) by redesigning the variational quantum circuit (VQC) used in its value-function approximator. Motivated by recent advances in quantum convolutional neural networks (QCNNs), we construct four QCNN-inspired VQC variants (Models A–D) by combining representative QCNN two-qubit building blocks with an explicit fully connected (all-to-all) layer. Using a 10-fold evaluation protocol at a fixed noise level p = 0.005, Model D achieves the best robustness, reducing the mean number of episodes required to reach a target reward from 1981 (baseline) to 1243. Under a stricter success criterion, Model D also doubles the empirically observed noise-tolerance boundary from 0.002 to 0.004. These results indicate that carefully chosen QCNN-style circuit components and connectivity can significantly improve the noise robustness of QDQN-like QRL agents.

Information•Mar 3, 2026•1 citation

Quantum Multi-Agent Reinforcement Learning via Variational Quantum Circuit Design

Won Joon Yun, Yunseok Kwak, Jae Pyoung Kim +4 more

In recent years, quantum computing (QC) has been getting a lot of attention from industry and academia. Especially, among various QC research topics, variational quantum circuit (VQC) enables quantum deep reinforcement learning (QRL). Many studies of QRL have shown that the QRL is superior to the classical reinforcement learning (RL) methods under the constraints of the number of training parameters. This paper extends and demonstrates the QRL to quantum multi-agent RL (QMARL). However, the extension of QRL to QMARL is not straightforward due to the challenge of the noise intermediate-scale quantum (NISQ) and the non-stationary properties in classical multi-agent RL (MARL). Therefore, this paper proposes the centralized training and decentralized execution (CTDE) QMARL framework by designing novel VQCs for the framework to cope with these issues. To corroborate the QMARL framework, this paper conducts the QMARL demonstration in a single-hop environment where edge agents offload packets to clouds. The extensive demonstration shows that the proposed QMARL framework enhances 57.7% of total reward than classical frameworks.

arXiv:2203.10443•IEEE International Conference on Distributed Computing Systems (ICDCS)•Jul 1, 2022•69 citations
We present a novel approach integrating quantum neural networks with reinforcement learning (RL) for house air conditioning control. Our framework employs a quantum deep Q-network (QDQN) RL implemented with PennyLane, leveraging parameterized variational quantum circuits (VQC) as function approximators for Q-values. The agent interacts with simulated indoor thermodynamic modelling, incorporating thermal capacity, heat transfer, and outdoor temperature variations. The action space consists of discrete air conditioning power levels, and the reward function balances comfort, energy, and switching penalties. To stabilize training, we introduce a target network, experience replay, and epsilon-greedy exploration. QDQN encodes the state vector into quantum rotations and applies multi-layer entangling gates, providing a potentially richer representation of the environment compared to classical networks. The QDQN agent uses VQC architecture. It incorporates data re-uploading, parameterized rotations, and strongly entangled qubits to effectively capture inter-feature correlations. We demonstrate the algorithm on episodes with outdoor conditions in both winter and summer conditions, showing convergence to policies that maintain indoor temperatures within setpoint bounds with zero violation rate. To validate the framework in a real-world noise setting, the quantum pipeline is simulated on the IBM Marrakesh system, featuring a 156-qubit Heron r2 processor with a median two-qubit (CZ) error of approximately 2.59 × 10− 3. Our results highlight the promise of quantum RL for real-world controls.
Quantum Reinforcement Learning (QRL) emerged as a branch of reinforcement learning (RL) that uses quantum submodules in the architecture of the algorithm. One branch of QRL focuses on the replacement of neural networks (NN) by variational quantum circuits (VQC) as function approximators. Initial works have shown promising results on classical environments with discrete action spaces, but many of the proposed architectural design choices of the VQC lack a detailed investigation. Hence, in this work we investigate the impact of VQC design choices such as angle embedding, encoding block architecture and postprocessesing on the training capabilities of QRL agents. We show that VQC design greatly influences training performance and heuristically derive enhancements for the analyzed components. Additionally, we show how to design a QRL agent in order to solve classical environments with continuous action spaces and benchmark our agents against classical feed-forward NNs.
With the evolution of technology, the need to solve complex computational problems like machine learning and deep learning has shot up. However, even the most powerful classical supercomputers find it difficult to execute these tasks. Advancements in quantum computing are leading researchers and tech-giants who strive for better quantum circuits to do machine learning tasks. Current works on Quantum Machine Learning (QML) ensure less memory consumption and reduced model parameters. However, it is strenuous to simulate the classical deep learning approach on existing quantum computing devices due to the inflexibility of Deep quantum circuits. Consequently, designing viable quantum algorithms for QML for noisy intermediate-scale quantum (NISQ) devices is essential. The proposed work aims to explore Variational Quantum Circuits (VQC) for Deep Q network-based Reinforcement Learning by remodeling the target network and experience replay into a representation of VQC. In addition, to the reduction in model parameters, quantum information encoding schemes are used to achieve better results than classical neural networks. VQCs are employed for policyselection and decision-making reinforcement learning by approximating the deep Q-value function with target network and experience replay.
The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep Q-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.
This paper proposes a quantum neural network (QNN)-based reinforcement learning (RL) controller for the real-time control of DC–DC buck converters supplying constant power loads (CPLs). Unlike conventional RL approaches that rely on classical neural networks, the proposed method employs a parameterized quantum circuit to represent the control policy, enabling a compact and expressive function approximator. The QNN is implemented as a classically simulated variational quantum circuit; no physical quantum hardware or quantum computational speedup is claimed. It is trained offline using a deterministic policy gradient framework and subsequently deployed in a real-time control loop under strict sampling constraints. The proposed 36-parameter QNN actor reduces trainable parameters by 85% versus a computation-matched classical RL baseline (245 parameters), while achieving sub-millisecond settling times and near-zero steady-state error across reference tracking, load disturbance, and supply variation scenarios validated on a dSPACE DS1202 hardware platform at 100-kHz switching frequency. Small-signal stability analysis at three tested operating points confirms phase margins exceeding 30° and a gain margin that is theoretically infinite at those points, as the linearized phase does not cross −180°. The results highlight the ability of the QNN to capture complex system dynamics with fewer trainable parameters while maintaining stable closed-loop operation, providing, to the best of the authors’ knowledge, a first step toward integrating quantum machine learning concepts into real-time power electronics control.
Quantum reinforcement learning (QRL) is often evaluated under idealized, noiseless assumptions, yet realistic quantum devices inevitably introduce noise that can severely degrade performance. This paper improves the robustness of quantum deep Q-learning (QDQN) by redesigning the variational quantum circuit (VQC) used in its value-function approximator. Motivated by recent advances in quantum convolutional neural networks (QCNNs), we construct four QCNN-inspired VQC variants (Models A–D) by combining representative QCNN two-qubit building blocks with an explicit fully connected (all-to-all) layer. Using a 10-fold evaluation protocol at a fixed noise level p = 0.005, Model D achieves the best robustness, reducing the mean number of episodes required to reach a target reward from 1981 (baseline) to 1243. Under a stricter success criterion, Model D also doubles the empirically observed noise-tolerance boundary from 0.002 to 0.004. These results indicate that carefully chosen QCNN-style circuit components and connectivity can significantly improve the noise robustness of QDQN-like QRL agents.
In recent years, quantum computing (QC) has been getting a lot of attention from industry and academia. Especially, among various QC research topics, variational quantum circuit (VQC) enables quantum deep reinforcement learning (QRL). Many studies of QRL have shown that the QRL is superior to the classical reinforcement learning (RL) methods under the constraints of the number of training parameters. This paper extends and demonstrates the QRL to quantum multi-agent RL (QMARL). However, the extension of QRL to QMARL is not straightforward due to the challenge of the noise intermediate-scale quantum (NISQ) and the non-stationary properties in classical multi-agent RL (MARL). Therefore, this paper proposes the centralized training and decentralized execution (CTDE) QMARL framework by designing novel VQCs for the framework to cope with these issues. To corroborate the QMARL framework, this paper conducts the QMARL demonstration in a single-hop environment where edge agents offload packets to clouds. The extensive demonstration shows that the proposed QMARL framework enhances 57.7% of total reward than classical frameworks.