Testing quantum programs in a black-box setting is difficult because bug-relevant behavioral differences may be missed at two levels: the chosen inputs may fail to trigger faulty behavior, and even when faults are triggered, the chosen measurements may fail to make the resulting differences observable. Existing methods have mainly strengthened either input generation or measurement design in isolation, leaving their coupling insufficiently addressed. In this paper, we present QDoctor, a probe-based black-box testing framework that wraps the program under test with a variety of property-based probes, thereby effectively extracting program information and exploring its behavior. A probe integrates two components: test-case generation and the specification of the measurement basis. To reduce false positive rate, QDoctor also supports semantic-aware partial equivalence testing through constrained probe-oracle design. Experimental results on benchmark suites show that QDoctor achieves the best balanced performance among representative black-box baselines, reaching 99.89% recall, 98.23% precision, 99.06% F1, and 0.9988 AUC. Ablation shows that jointly strengthening inputs and measurements raises recall from 85.280% to 99.728% with zero false positives, while semantic probes drive the false-positive rate to zero on SWAP-induced equivalent mutants.
Testing quantum programs in a black-box setting is difficult because bug-relevant behavioral differences may be missed at two levels: the chosen inputs may fail to trigger faulty behavior, and even when faults are triggered, the chosen measurements may fail to make the resulting differences observable. Existing methods have mainly strengthened either input generation or measurement design in isolation, leaving their coupling insufficiently addressed. In this paper, we present QDoctor, a probe-based black-box testing framework that wraps the program under test with a variety of property-based probes, thereby effectively extracting program information and exploring its behavior. A probe integrates two components: test-case generation and the specification of the measurement basis. To reduce false positive rate, QDoctor also supports semantic-aware partial equivalence testing through constrained probe-oracle design. Experimental results on benchmark suites show that QDoctor achieves the best balanced performance among representative black-box baselines, reaching 99.89% recall, 98.23% precision, 99.06% F1, and 0.9988 AUC. Ablation shows that jointly strengthening inputs and measurements raises recall from 85.280% to 99.728% with zero false positives, while semantic probes drive the false-positive rate to zero on SWAP-induced equivalent mutants.
Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.
Quantum programs exhibit inherent non-deterministic behavior, which poses more significant challenges for error discovery compared to classical programs. While several testing methods have been proposed for quantum programs, they often overlook fundamental questions in black-box testing. In this paper, we bridge this gap by presenting three novel algorithms specifically designed to address the challenges of equivalence, identity, and unitarity checking in black-box testing of quantum programs. We also explore optimization techniques for these algorithms, including specialized versions for equivalence and unitarity checking, and provide valuable insights into parameter selection to maximize performance and effectiveness. To evaluate the effectiveness of our proposed methods, we conducted comprehensive experimental evaluations, which demonstrate that our methods can rigorously perform equivalence, identity, and unitarity checking, offering robust support for black-box testing of quantum programs.
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
Manually debugging quantum programs is a difficult and time-intensive process. In this paper, we introduce an automated debugging technique, based on delta debugging and property-based testing, for quantum programs. Our technique automatically identifies the changes made within an update to a quantum program that cause a property-based regression test to fail. To evaluate our technique, we inject faults and semantic preserving changes into three quantum algorithms. We discuss the viability and efficacy of our approach after measuring the percentage of faults and semantic preserving changes. Our results indicate that our method has a high true positive (called sensitivity) and true negative rate (called specificity) and is robust in terms of the amount of changes introduced to the program. Moreover, the sensitivity of the method increases significantly with the number of properties. While the specificity remains stable when increasing the number of properties and inputs.
Quantum software relies on compilers, intermediate representations, and execution frameworks that transform programs before execution. Faults in these components can change the behavior of otherwise correct quantum programs. Yet quantum testing techniques are often evaluated using mutation-generated faults and curated bug benchmarks. We investigate how well the faults represented by these artifacts reflect those encountered in modern quantum software. We audit four quantum mutation-testing tools and three curated bug benchmarks, and collect 410 verified bugs from ten quantum software repositories over two years. We classify faults from all three sources into a common taxonomy of 56 bug families and compare their representation across mutation operators, curated benchmarks, and recent real-world bugs. The mutation tools support 20 bug families, and curated benchmarks add 15 more. Yet only 56 of the 410 real-world bugs (13.7%) belong to families represented by either source; the remaining 354 (86.3%) belong to families absent from both. The largest unrepresented families concern compiler transformations (66 bugs), type handling (53), and input validation (47). Unlike local gate or statement changes, these faults often depend on intermediate representations, compiler pipelines, input types, or dependency versions. Existing mutation operators primarily modify circuits and program constructs, whereas many real-world faults arise in the compilers, representations, and framework interfaces that process them. Addressing this gap requires looking beyond circuit-level faults toward testing semantic preservation across compiler transformations, correctness at representation boundaries, and interactions between software components. Our study provides empirical evidence to guide fault models and benchmarks that capture these mechanisms and preserve the contexts needed to reproduce them.
Oracle quantum programs are a fundamental class of quantum programs that serve as a critical bridge between quantum computing and classical computing. Many important quantum algorithms are built upon oracle quantum programs, making it essential to ensure their correctness during development. Although software testing is a well-established approach for improving program reliability, no systematic method has been developed to test oracle quantum programs. This paper proposes a black-box testing framework designed for general oracle quantum programs. We formally define these programs, establish the foundational theory for their testing, and propose a detailed testing framework. We develop a prototype tool and conduct extensive experimental evaluations to evaluate the effectiveness of the framework. Our results demonstrate that the proposed framework significantly aids developers in testing oracle quantum programs, providing insights to enhance the reliability of quantum software.