Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

Abstract

Quantum cloud computing, delivered through the quantum-as-a-service (QaaS) model, provides access to quantum computing resources. However, applying uniform time-based pricing across fundamentally heterogeneous quantum resources significantly complicates task orchestration, particularly when addressing the tradeoff between execution costs and system performance. While heuristic methods rely on predefined scheduling rules, classical deep reinforcement learning (DRL) models may require more trainable parameters in this setting. Motivated by the potential of parameterised quantum circuits (PQCs) as compact function approximators, we propose QRLQ, a cost-delay-aware quantum cloud scheduling framework integrating PQCs with a dueling double deep Q-network (D3QN) to dynamically account for both cost and delay. Our simulation results show that QRLQ achieves lower mean cost and delay than the heuristic baselines, achieving a 5-11% lower mean cost relative to availability-based and rotation-based heuristics and reducing mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while retaining execution fidelity within 2% of a fidelity-greedy policy. Compared with the classical DRL baseline, QRLQ achieves comparable scheduling performance while using 72% fewer trainable parameters. This work explores the feasibility of using QRL for task orchestration in quantum cloud environments and demonstrates its potential for cost-delay-aware quantum resource management.

The paper studies online assignment of quantum-cloud jobs to heterogeneous QPUs under uniform execution-time pricing, where using faster devices reduces cost but can increase queueing. It formulates scheduling as an MDP with a reward combining normalized execution and waiting times, using equal objective weights and a circuit-complexity adjustment. The proposed QRLQ scheduler replaces the feature extractor in a duelling double deep Q-network with an eight-qubit, single-layer parameterized quantum circuit, followed by classical value and advantage heads. Evaluation uses classical circuit simulation, five Qiskit fake IBM backends, MQT Bench workloads, and stochastic arrivals. Execution times are estimated from transpiled-circuit critical paths; fidelity is approximated by multiplying operation-success probabilities. Training uses five random seeds; evaluation uses one trained seed on 21,000 unseen tasks across 100 episodes. At an assumed price of $\$1.60$ per execution second, QRLQ achieves mean cost $\$3.70$ and delay $29.99$ seconds, versus $\$3.80$ and $32.94$ seconds for classical D3QN. Relative to heuristic baselines, reported cost reductions are 5–11% and delay reductions 17–82%; the fastest-device heuristic remains cheaper at $\$2.83$, with $170.81$ seconds delay. Estimated fidelity is 0.612 versus 0.624 for fidelity-greedy scheduling. QRLQ uses 214 trainable parameters versus 758 for D3QN, approximately 72% fewer. Stated limitations include idealized simulation, serial shots without reset/readout overheads, and one task per QPU. Robustness tests perturb circuit parameters rather than execute on noisy hardware. Larger clusters, multiprogramming, distributed execution, iterative hybrid workloads, and physical-hardware validation remain future work.