The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present QOPS, a novel quantum software testing approach. QOPS introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. QOPS also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making QOPS practical for testing complex quantum programs in industrial settings. We empirically evaluate QOPS on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of QOPS by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.
The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present QOPS, a novel quantum software testing approach. QOPS introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. QOPS also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making QOPS practical for testing complex quantum programs in industrial settings. We empirically evaluate QOPS on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of QOPS by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.
Quantum software testing is important for reliable quantum software engineering. Despite recent advances, existing quantum software testing approaches rely on simple test inputs and statistical oracles, costly program specifications, and limited validation on real quantum computers. To address these challenges, we propose SB-QOPS, a search-based quantum program testing approach via commuting Pauli strings. SB-QOPS, as a direct extension to a previously proposed QOPS approach, redefines test cases in terms of Pauli strings and introduces a measurement-centric oracle that exploits their commutation properties, enabling effective testing of quantum programs while reducing the need for full program specifications. By systematically exploring the search space through an expectation-value-based fitness function, SB-QOPS improves test budget utilization and increases the likelihood of uncovering subtle faults. We conduct a large-scale empirical evaluation on quantum circuits of up to 29 qubits on real quantum computers and emulators. We assess three search strategies: Genetic Algorithm, Hill Climbing, and the (1+1) Evolutionary Algorithm, and evaluate SB-QOPS under both simulated and real noisy conditions. Experiments span three quantum computing platforms: IBM, IQM, and Quantinuum. Results show that SB-QOPS significantly outperforms QOPS, achieving a fault-detection score of 100% for circuits up to 29 qubits, and demonstrating portability across quantum platforms.
Quantum Computing (QC) promises computational speedup over classic computing. However, noise exists in near-term quantum computers. Quantum software testing (for gaining confidence in quantum software's correctness) is inevitably impacted by noise, i.e., it is impossible to know if a test case failed due to noise or real faults. Existing testing techniques test quantum programs without considering noise, i.e., by executing tests on ideal quantum computer simulators. Consequently, they are not directly applicable to test quantum software on real quantum computers or noisy simulators. Thus, we propose a noise-aware approach (namedQOIN) to alleviate the noise effect on test results of quantum programs.QOINemploys machine learning techniques (e.g., transfer learning) to learn the noise effect of a quantum computer and filter it from a program's outputs. Such filtered outputs are then used as the input to perform test case assessments (determining the passing or failing of a test case execution against a test oracle). We evaluatedQOINon IBM's 23 noise models, Google's two available noise models, and Rigetti's Quantum Virtual Machine, with six real-world and 800 artificial programs. We also generated faulty versions of these programs to check if a failing test case execution can be determined under noise. Results show thatQOINcan reduce the noise effect by more than80%on most noise models. We used an existing test oracle to evaluateQOIN's effectiveness in quantum software testing. The results showed thatQOINattained scores of99%,75%, and86%for precision, recall, and F1-score, respectively, for the test oracle across six real-world programs. For artificial programs,QOINachieved scores of93%,79%, and86%for precision, recall, and F1-score respectively. This highlightsQOIN's effectiveness in learning noise patterns for noise-aware quantum software testing.
Machine learning has been extensively applied for classical software testing activities such as test generation, minimization, and prioritization. Along the same lines, there has been interest in applying quantum machine learning to classical software testing. For example, Quantum Extreme Learning Machines (QELMs) were recently applied for testing classical software of industrial elevators. However, studies on QELMs, whether in software testing or other areas, used ideal simulators that fail to account for the noise in current quantum computers. While ideal simulations offer insight into QELM's theoretical capabilities, they do not enable studying their performance on current noisy quantum computers. To this end, we study how quantum noise affects QELM in three industrial classical software testing case studies, providing insights into QELMs' robustness to noise for software testing applications. Such insights assess QELMs potential as a viable solution for software testing problems in today's noisy quantum computing. Our results show that QELMs are significantly affected by quantum noise, with a performance drop of 250% in regression and 50% in classification software testing tasks. Quantum noise also increases uncertainty in QELM models, producing a saturation effect where larger qubit counts make the models increasingly random and unreliable. While error mitigation techniques can enhance noise resilience, achieving an average 3% performance drop in classification, their effectiveness varies by context. For classification tasks, QLEAR performs well, whereas Zero Noise Extrapolation is more effective for regression and smaller qubit counts. However, no single mitigation approach consistently reduces uncertainty across tasks or scales reliably as the number of qubits increases, highlighting the need for QELM-tailored strategies.
Quantum computers have the potential to outperform classical computers for some complex computational problems. However, current quantum computers (e.g., from IBM and Google) have inherent noise that results in errors in the outputs of quantum software executing on the quantum computers, affecting the reliability of quantum software development. The industry is increasingly interested in machine learning (ML)-based error mitigation techniques, given their scalability and practicality. However, existing ML-based techniques have limitations, such as only targeting specific noise types or specific quantum circuits. This paper proposes a practical ML-based approach, called Q-LEAR, with a novel feature set, to mitigate noise errors in quantum software outputs. We evaluated Q-LEAR on eight quantum computers and their corresponding noisy simulators, all from IBM, and compared Q-LEAR with a state-of-the-art ML-based approach taken as baseline. Results show that, compared to the baseline, Q-LEAR achieved a 25% average improvement in error mitigation on both real quantum computers and simulators. We also discuss the implications and practicality of Q-LEAR, which, we believe, is valuable for practitioners.
As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.
Quantum software testing is essential for the quality assurance of quantum programs. However, existing techniques face many challenges, such as constructing test oracles and distinguishing genuine faults from noise-induced errors. Propertybased testing is a promising solution for dealing with the test oracle issue, which verifies general properties rather than requiring inputs and outputs. However, its effectiveness in noisy quantum environments has not been studied. To this end, we evaluate the feasibility of applying property-based testing in noisy quantum computers by integrating it with a state-of-the-art machine learning-based noise mitigation method called QOIN. Our results show that on average, QOIN mitigates noise for individual circuits, though it may introduce outliers. Crucially, this does not guarantee property preservation: we show that some properties align better with QOIN, while others remain closer under unmitigated noise. To address this, we apply a hybrid approach that selectively applies QOIN during property-based testing. We show that this hybrid approach improves alignment with the ideal execution and significantly reduces false positives in assertion outcomes across most executed mutants. This provides a solid foundation for applying property-based testing to noisy quantum systems.