arXiv:2508.09183 · IEEE International Conference on Quantum Artificial Intelligence (QAI)

Quantum-Efficient Reinforcement Learning Solutions for Last-Mile On-Demand Delivery

Farzan Moosavi, Bilal Farooq

Published Nov 2, 2025• 0 citations

Abstract

Quantum computation has demonstrated a promising alternative to solving the NP-hard combinatorial problems. Specifically, when it comes to optimization, classical approaches become intractable to account for large-scale solutions. Specifically, we investigate quantum computing to solve the large-scale Capacitated Pickup and Delivery Problem with Time Windows (CPDPTW). In this regard, a Reinforcement Learning (RL) framework augmented with a Parametrized Quantum Circuit (PQC) is designed to minimize the travel time in a realistic last-mile on-demand delivery. A novel problem-specific encoding quantum circuit with an entangling and variational layer is proposed. Moreover, Proximal Policy Optimization (PPO) and Quantum Singular Value Transformation (QSVT) are designed for comparison through numerical experiments, highlighting the superiority of the proposed method in terms of the scale of the solution and training complexity while incorporating the real-world constraints.

DOI: 10.1109/qai63978.2025.00069

Related papers

Reinforcement-Learning-Based Variational Quantum Circuits Optimization for Combinatorial Problems

Sami Khairy, Ruslan Shaydulin, Łukasz Cincio +2 more

Quantum computing exploits basic quantum phenomena such as state superposition and entanglement to perform computations. The Quantum Approximate Optimization Algorithm (QAOA) is arguably one of the leading quantum algorithms that can outperform classical state-of-the-art methods in the near term. QAOA is a hybrid quantum-classical algorithm that combines a parameterized quantum state evolution with a classical optimization routine to approximately solve combinatorial problems. The quality of the solution obtained by QAOA within a fixed budget of calls to the quantum computer depends on the performance of the classical optimization routine used to optimize the variational parameters. In this work, we propose an approach based on reinforcement learning (RL) to train a policy network that can be used to quickly find high-quality variational parameters for unseen combinatorial problem instances. The RL agent is trained on small problem instances which can be simulated on a classical computer, yet the learned RL policy is generalizable and can be used to efficiently solve larger instances. Extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our trained RL policy can reduce the optimality gap by a factor up to 8.61 compared with other off-the-shelf optimizers tested.

arXiv:1911.04574Nov 11, 201921 citations

Investigating Quantum Reinforcement Learning structure to the CartPole control task

Nguyen Truong Thu Ngo, Tien‐Fu Lu, James Q. Quach +1 more

In recent years, reinforcement learning (RL) has been proven to effectively provide solution to complex problems in various engineering problems such as self-driving cars, industry automation in production lines. These applications also extend to other fields including providing a platform for financial trading and healthcare. Reinforcement learning comes with the trade-offs between exploitation and exploration regardless of applied techniques and algorithms. In more complex engineering system, which often consists of large action and state-space. This space can increase exponentially, and most RL techniques fail to efficiently compute optimal policies to these problems. The inefficiency in RL models leads to the extensive requirement of computational process, which does not often come cheap and affordable. To tackle this challenge, quantum computational models were studied and achieved different levels of success. Since its first theory in the 1940s, quantum algorithm has been improved and advanced to provide exponential speed up compared to classical solutions. Quantum computation offers great potential improvements to traditional RL models due to its ability to create superpositions and entanglement. In past research, quantum variational circuits (QVC) were created as alternatives to neural networks commonly used in RL and experimented on several RL benchmarks in OpenAI Gym environments: CartPole, Acrobat and Lunar Landing. One of such QVCs were designed and tested to achieve greater efficiency in learning speed while offer reduced number of trainable parameter than classical RL. Our research aims to investigate the QVC to balance the CartPole problem running on both local PC simulator and quantum computer. The research signifies the first time a quantum RL could learn to obtain the optimal policy to acquire the maximum expected reward to a control problem and effectively apply the trainable parameter to balance the pole. The system is first tailored into a RL environment with state space and action space. For each time step, the input data from the environment (state space) is encoded into quantum states. Through the QVC, the algorithm learns to optimize the RL policy to calculate the probability of future action to obtain the optimal reward for each state-action pair. The trainable parameters from the QVC are again optimized using gradient descent. The QVC design is vital to the success of the RL model; therefore, we vary the QVC gates and investigate the model performance. It is expected that the Quantum RL model would outperform the classical RL in learning the optimal policy in term of speed and computational resources. The successful control of the pole on a quantum simulation would be proof that quantum models could offer real solution to future computation-intensive problems where classical solutions are unaffordable.

Asian Society for Precision Engineering and Nanotechnology (ASPEN 2022)Jan 1, 20220 citations

Reinforcement Learning Based Quantum Circuit Optimization via ZX-Calculus

Jordi Riu, Jan Nogué, Gerard Vilaplana +2 more

We propose a novel Reinforcement Learning (RL) method for optimizing quantum circuits using graph-theoretic simplification rules of ZX-diagrams. The agent, trained using the Proximal Policy Optimization (PPO) algorithm, employs Graph Neural Networks to approximate the policy and value functions. We demonstrate the capacity of our approach by comparing it against the best performing ZX-Calculus-based algorithm for the problem in hand. After training on small Clifford+T circuits of 5-qubits and few tenths of gates, the agent consistently improves the state-of-the-art for this type of circuits, for at least up to 80-qubit and 2100 gates, whilst remaining competitive in terms of computational performance. Additionally, we illustrate the versatility of the agent by incorporating additional optimization routines on the workflow during training, improving the two-qubit gate count state-of-the-art on multiple structured quantum circuits for relevant applications of much larger dimension and different gate distributions than the circuits the agent trains on. This conveys the potential of tailoring the reward function to the specific characteristics of each application and hardware backend. Our approach is a valuable tool for the implementation of quantum algorithms in the near-term intermediate-scale range (NISQ).

arXiv:2312.11597QuantumMay 28, 202515 citations

A Quantum Q-Learning Model for the Capacitated Vehicle Routing Problem

Junde Xu, Ningyi Xie, Xinwei Lee +3 more

The research on Variational Quantum Algorithms (VQAs) has gained significant momentum because of their promising practicality in the noisy intermediate-scale quantum (NISQ) era. Recent studies highlight the potential of VQAs in reinforcement learning (RL) by replacing classical components with parameterized quantum circuits (PQCs). Building on this, we propose a quantum Q-learning model for the Capacitated Vehicle Routing Problem (CVRP), combining a PQC with an RL training process. We present a PQC for embedding both static and dynamic states in RL environments. The proposed PQC approximates the Q-value function by using the expectation values of different observables. Experimental results demonstrate that the proposed PQC outperforms the existing PQC for the CVRP with fewer quantum resources. Furthermore, our method shows competitive performance compared to the classical counterpart.

IEEE International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC)Dec 16, 20242 citations

Optimization of Reinforcement Learning Using Quantum Computation

Roopa Ravish, Nischal R. Bhat, N. Nandakumar +3 more

Exploring the convergence of quantum computing and machine learning, this paper delves into Quantum Reinforcement Learning (QRL) with a specific focus on Variational Quantum Circuits (VQC). Alongside a comprehensive examination of the field, the study presents a concrete implementation of QRL tailored to foundational gym environments. Leveraging principles from Quantum Computing, experiments reveal QRL’s potential in enhancing reinforcement learning tasks within these environments, showcasing notable space optimization for RL models. Quantitative results indicate a reduction in the number of trainable parameters by up to 90% when compared to classical approaches, significantly improving efficiency, mainly due to quantum principles of superposition and entanglement. This highlights the promising implications of integrating quantum computing techniques to address challenges and advance capabilities in fundamental reinforcement learning scenarios. Furthermore, the study discusses the adaptability of QRL algorithms to diverse problem domains, suggesting their potential for scalability and applicability beyond simple environments. Such versatility underscores the robustness and practical relevance of QRL methodologies in real-world scenarios, positioning them as valuable tools for tackling complex reinforcement learning challenges.

IEEE AccessJan 1, 202410 citations

A Hybrid Quantum Classical Optimization Framework for Pickup and Delivery Problems with Parcel Lockers Using Quantum Graph Attention Networks

Aqdas Shehzad, Zhao Xu, Muhammad Aurangzeb +4 more

Abstract The significant growth of e-commerce has created a vast demand for last-mile parcel delivery, in which case, parcel lockers will be the future of home delivery. But the Pickup and Delivery Problem with Lockers (PDPL) is computationally hard as it is an NP-hard problem with locker capacities, time windows, and stochastic customer availability. In this paper, we present Q-PDPL, a novel quantum-assisted hybrid approach for optimization that employs Variation Quantum Eigensolver (VQE), Quantum Approximate Optimization Algorithm (QAOA), and Quantum Graph Attention Networks (Q-GAT) in a Dantzig-Wolfe decomposition (DW) scheme. The framework recasts the pricing problem as a ground state of the energy optimization problem, which is quantum efficiently solvable by means of parameterized quantum circuits. In a stochastic setting, a Q-GAT-enhanced deep reinforcement learning module is employed that delivers online routing decisions. Tests with Solomon VRPTW and Li and Lim PDPTW datasets over the SQL demonstrate that Q-PDPL is 18.7% cheaper than classical Adaptive Large Neighborhood Search, 23.4% better than Branch-and-Price, and handles over 500 customers. The quantum pricing sub problem has O(√N⋅"poly" (logN)) complexity (classical O(N 2 )) and a solution quality within 2.1% of the optimal bound. These results hold out promise for quantum-classical hybrid algorithms for large-scale logistics.

Aug 19, 20260 citations