As quantum software engineering advances, testing techniques are required to assess the quality of quantum programs (QPs). In the test process, the test oracle is vital for determining whether the test result indicates a success or a failure. Most related works directly measure the output states and acquire the corresponding test results by comparing the output distribution with the expected one. While attention has been paid to the capability of fault detection, the guarantee for the correctness of the produced test results remains limited. Unlike classical programs (CPs), the output quantum states of QPs should be transformed into probabilistic classical outcomes through quantum measurement. This additional operation of measurement could cause a test oracle to yield the wrong test results. Especially for high-dimensional output spaces, numerous measurement outcomes are required to capture the distribution characteristics, threatening the effectiveness and cost-efficiency of test oracles. Hence, this paper proposes a novel specified test oracle DOSS employing a dynamic scheme to integrate a quantum algorithm (i.e., swap test) with the direct measurement mode. This innovative approach enables the validation of individual outputs rather than their distribution during the testing phase. Considering acceptable cost, DOSS decomposes the fully or partially separable output states to lower the dimensionality and simplify the quantum circuit for testing. Empirical studies demonstrate that DOSS generally gives more correct test results than baselines, and maintains reasonable cost on an ideal simulator. Besides, DOSS’s effectiveness with quantum noise involved is validated via three noisy simulators.
As quantum software engineering advances, testing techniques are required to assess the quality of quantum programs (QPs). In the test process, the test oracle is vital for determining whether the test result indicates a success or a failure. Most related works directly measure the output states and acquire the corresponding test results by comparing the output distribution with the expected one. While attention has been paid to the capability of fault detection, the guarantee for the correctness of the produced test results remains limited. Unlike classical programs (CPs), the output quantum states of QPs should be transformed into probabilistic classical outcomes through quantum measurement. This additional operation of measurement could cause a test oracle to yield the wrong test results. Especially for high-dimensional output spaces, numerous measurement outcomes are required to capture the distribution characteristics, threatening the effectiveness and cost-efficiency of test oracles. Hence, this paper proposes a novel specified test oracle DOSS employing a dynamic scheme to integrate a quantum algorithm (i.e., swap test) with the direct measurement mode. This innovative approach enables the validation of individual outputs rather than their distribution during the testing phase. Considering acceptable cost, DOSS decomposes the fully or partially separable output states to lower the dimensionality and simplify the quantum circuit for testing. Empirical studies demonstrate that DOSS generally gives more correct test results than baselines, and maintains reasonable cost on an ideal simulator. Besides, DOSS’s effectiveness with quantum noise involved is validated via three noisy simulators.
Quantum software testing is important for reliable quantum software engineering. Despite recent advances, existing quantum software testing approaches rely on simple test inputs and statistical oracles, costly program specifications, and limited validation on real quantum computers. To address these challenges, we propose SB-QOPS, a search-based quantum program testing approach via commuting Pauli strings. SB-QOPS, as a direct extension to a previously proposed QOPS approach, redefines test cases in terms of Pauli strings and introduces a measurement-centric oracle that exploits their commutation properties, enabling effective testing of quantum programs while reducing the need for full program specifications. By systematically exploring the search space through an expectation-value-based fitness function, SB-QOPS improves test budget utilization and increases the likelihood of uncovering subtle faults. We conduct a large-scale empirical evaluation on quantum circuits of up to 29 qubits on real quantum computers and emulators. We assess three search strategies: Genetic Algorithm, Hill Climbing, and the (1+1) Evolutionary Algorithm, and evaluate SB-QOPS under both simulated and real noisy conditions. Experiments span three quantum computing platforms: IBM, IQM, and Quantinuum. Results show that SB-QOPS significantly outperforms QOPS, achieving a fault-detection score of 100% for circuits up to 29 qubits, and demonstrating portability across quantum platforms.
Quantum computing has emerged as an innovative computational paradigm with great potential in various domains. As quantum computing advances, the development of high-quality quantum programs has become crucial, making it essential to ensure their reliability. Software testing plays a vital role in achieving the reliability and quality of software systems. Various testing strategies and tools have been proposed for traditional programs; however, research on testing methodologies for quantum programs is still in the early stages. Traditional testing techniques, while effective for classical programs, struggle to address the unique challenges posed by quantum programs, including inherent characteristics of quantum systems (such as superposition and entanglement), and the exponentially expanding input space as the number of qubits increases. Moreover, traditional testing strategies typically do not account for the hidden and non-deterministic failure patterns associated with input quantum bits (qubits), which, if recognized, could potentially lead to more efficient fault detection. Therefore, in this paper, we introduce a novel approach—Quantum Dynamic-Testing with Incremental Learning (QDT-IL), which aims to effectively capture and adapt to the failure patterns in quantum programs. QDT-IL employs an incremental learning model to learn from executed test cases and continuously updates its predictions on the failure tendencies for new test cases. By utilizing distance-based diversity metrics, QDT-IL strategically increases the variety of test cases, enabling targeted exploration of failure-prone regions in the input space. This combination of adaptive learning and diverse test case selection noticeably enhances the effectiveness and efficiency of quantum program testing. Experimental studies show that QDT-IL outperforms the baseline strategies, providing a more effective testing process for quantum programs.
The capability of Quantum Computing (QC) in solving complex problems has been increasingly recognized. However, similar to classical computing, to fully exploit QC's potential, it is important to ensure the correctness of quantum programs. Doing so via software testing is, however, very challenging because of QC's inherent properties: superposition and entanglement. Towards the direction of ensuring the correctness of quantum programs, we propose an approach called QuCAT (QUantum CombinAtorial Testing) for systematic and automated testing of quantum programs by benefiting from combinatorial testing, which has been proven to be cost-effective in testing classical programs. QuCAT supports two combinatorial test suite generation scenarios, i.e., generating combinatorial test suites of a given strength, and incrementally generating and executing combinatorial test suites of increasing strength until a fault is found. The approach employs two types of test oracles to assess test results. We performed an empirical study with 18 faulty versions of quantum programs to evaluate QuCAT with strengths of two, three, and four in the two test generation scenarios. We compare the cost-effectiveness of combinatorial testing of various strengths and random testing (taken as baseline approach). Results show that combinatorial testing always performs better than random testing with the same cost and finds faults more quickly (in terms of required number of test cases). In addition, in most cases, combinatorial testing with a higher strength outperforms the lower strength in terms of effectiveness.
As quantum computing continues to emerge, ensuring the quality of quantum programs has become increasingly critical. Quantum program testing has emerged as a prominent research area within the scope of quantum software engineering. While numerous approaches have been proposed to address quantum program quality assurance, our analysis reveals that most existing methods rely on measurement-based validation in practice. However, due to the inherently probabilistic nature of quantum programs, measurement-based validation methods face significant limitations.To investigate these limitations, we conducted an empirical study of recent research on quantum program testing, analyzing measurement-based validation methods in the literature. Our analysis categorizes existing measurement-based validation methods into two groups: distribution-level validation and output-value-level validation. We then compare measurement-based validation with statevector-based validation methods to evaluate their pros and cons. Our findings demonstrate that measurement-based validation is suitable for straightforward assessments, such as verifying the existence of specific output values, while statevector-based validation proves more effective for complicated tasks such as assessing the program behaviors.
Quantum software stacks (QSSs), which provide quantum circuit transformers and simulators, enable circuit transformations and the execution of circuits on classical computers. Despite their importance, they have not been effectively tested yet, leaving the correctness in question. The main obstacle to testing is the absence of a testing oracle, which checks the semantics-preservation of circuit transformations and the correctness of simulation results. While previous studies have employed differential and metamorphic testing to circumvent the necessity for an oracle, they have detected very few non-crash bugs. In this work, we address this gap by introducing QASMInfer, an exact inference system for quantum circuits, which computes the probability distribution of possible circuit outcomes. By supporting circuits written in OpenQASM, the de facto standard quantum assembly language used by most QSSs, QASMInfer acts as a unified testing oracle for multiple QSSs. Our design of QASMInfer achieves three key goals: (1) support for dynamic circuits, an important class of quantum circuits, (2) efficiency, and (3) reliability. For efficiency, we introduce two optimizations and an efficient matrix representation. For reliability, we prove physical consistency, ensuring that QASMInfer’s inference results adhere to the physical principles of quantum computing. To simplify the proof, we introduce OpenQASMCore, a core language for OpenQASM, and perform exact inference for OpenQASM by desugaring it to OpenQASMCore. Our implementation and proof are fully mechanized in the Coq proof assistant. Testing six real-world QSSs using QASMInfer revealed 31 bugs, including 20 non-crash bugs, demonstrating QASMInfer’s effectiveness as a testing oracle.
We present a new technique for accelerating quantum program testing. Given a quantum circuit with an input/output specification, our goal is to check whether executing the program on the input state produces the expected output. In quantum computing, however, it is impossible to directly check the equivalence of the two quantum states. Instead, we rely on statistical testing, which involves repeated program executions, state measurements, and subsequent comparisons with the specified output. To guarantee a high level of assurance, however, this method requires an extensive number of measurements. In this paper, we propose a solution to alleviate this challenge by adapting Fixed-Point Amplitude Amplification (FPAA) for quantum program testing. We formally present our technique, demonstrate its ability to reduce the required number of measurements as well as runtime cost without sacrificing the original statistical guarantee, and showcase its runtime effectiveness through case studies.