Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2503.22641 · ACM Transactions on Quantum Computing

QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit

Gabriel Pontolillo, Mohammad Reza Mousavi, Marek Grzesiuk

Published May 8, 2026• 4 citations
arXivHTMLPDF

Abstract

Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.

DOI
10.1145/3815169

Related papers

Mutation Testing of Quantum Programs: A Case Study With Qiskit

Daniel Fortunato, José Campos, Rui Abreu

As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.

IEEE Transactions on Quantum Engineering•Jan 1, 2022•62 citations

From Ideal to Noisy: Adapting Property-Based Testing for Real-World Noisy Quantum Computers

Gabriel Pontolillo, Asmar Muqeet, Shaukat Ali +1 more

Quantum software testing is essential for the quality assurance of quantum programs. However, existing techniques face many challenges, such as constructing test oracles and distinguishing genuine faults from noise-induced errors. Propertybased testing is a promising solution for dealing with the test oracle issue, which verifies general properties rather than requiring inputs and outputs. However, its effectiveness in noisy quantum environments has not been studied. To this end, we evaluate the feasibility of applying property-based testing in noisy quantum computers by integrating it with a state-of-the-art machine learning-based noise mitigation method called QOIN. Our results show that on average, QOIN mitigates noise for individual circuits, though it may introduce outliers. Crucially, this does not guarantee property preservation: we show that some properties align better with QOIN, while others remain closer under unmitigated noise. To address this, we apply a hybrid approach that selectively applies QOIN during property-based testing. We show that this hybrid approach improves alignment with the ideal execution and significantly reduces false positives in assertion outcomes across most executed mutants. This provides a solid foundation for applying property-based testing to noisy quantum systems.

IEEE International Conference on Quantum Computing and Engineering (QCE)•Aug 30, 2025•0 citations

Muskit: A Mutation Analysis Tool for Quantum Software Testing

Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini +1 more

Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917

IEEE/ACM International Conference on Automated Software Engineering (ASE)•Nov 1, 2021•69 citations

QUTest: A Native Testing Framework for Quantum Programs

José Campos

Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.

quant-pharXiv:2605.19736•IEEE/ACM International Conference on Automated Software Engineering (ASE)•May 20, 2026•0 citations

Delta Debugging for Property-Based Regression Testing of Quantum Programs

Gabriel Pontolillo, Mohammad Reza Mousavi

Manually debugging quantum programs is a difficult and time-intensive process. In this paper, we introduce an automated debugging technique, based on delta debugging and property-based testing, for quantum programs. Our technique automatically identifies the changes made within an update to a quantum program that cause a property-based regression test to fail. To evaluate our technique, we inject faults and semantic preserving changes into three quantum algorithms. We discuss the viability and efficacy of our approach after measuring the percentage of faults and semantic preserving changes. Our results indicate that our method has a high true positive (called sensitivity) and true negative rate (called specificity) and is robust in terms of the amount of changes introduced to the program. Moreover, the sensitivity of the method increases significantly with the number of properties. While the specificity remains stable when increasing the number of properties and inputs.

•Apr 16, 2024•7 citations

Quantum Concolic Testing

Shangzhou Xia, Jianjun Zhao, Fuyuan Zhang +1 more

This paper presents the first concolic testing framework explicitly designed for quantum programs. The framework introduces quantum constraint generation methods for quantum control statements that quantify quantum states and offers a symbolization method for quantum variables. Based on this framework, we generate path constraints for each concrete execution path of a quantum program. These constraints guide the exploration of new paths, with a quantum constraint solver determining outcomes to create novel input samples, thereby enhancing branch coverage. Our framework has been implemented in Python and integrated with Qiskit for practical evaluation. Experimental results show that our concolic testing framework improves branch coverage, generates high-quality quantum input samples, and detects bugs, demonstrating its effectiveness and efficiency in quantum programming and bug detection. Regarding branch coverage, our framework achieves more than 74.27% on quantum programs with under 5 qubits.

arXiv:2405.04860•Proceedings of the ACM on software engineering.•Jun 22, 2025•9 citations
Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.
As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.
Quantum software testing is essential for the quality assurance of quantum programs. However, existing techniques face many challenges, such as constructing test oracles and distinguishing genuine faults from noise-induced errors. Propertybased testing is a promising solution for dealing with the test oracle issue, which verifies general properties rather than requiring inputs and outputs. However, its effectiveness in noisy quantum environments has not been studied. To this end, we evaluate the feasibility of applying property-based testing in noisy quantum computers by integrating it with a state-of-the-art machine learning-based noise mitigation method called QOIN. Our results show that on average, QOIN mitigates noise for individual circuits, though it may introduce outliers. Crucially, this does not guarantee property preservation: we show that some properties align better with QOIN, while others remain closer under unmitigated noise. To address this, we apply a hybrid approach that selectively applies QOIN during property-based testing. We show that this hybrid approach improves alignment with the ideal execution and significantly reduces false positives in assertion outcomes across most executed mutants. This provides a solid foundation for applying property-based testing to noisy quantum systems.
Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917
Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.
Manually debugging quantum programs is a difficult and time-intensive process. In this paper, we introduce an automated debugging technique, based on delta debugging and property-based testing, for quantum programs. Our technique automatically identifies the changes made within an update to a quantum program that cause a property-based regression test to fail. To evaluate our technique, we inject faults and semantic preserving changes into three quantum algorithms. We discuss the viability and efficacy of our approach after measuring the percentage of faults and semantic preserving changes. Our results indicate that our method has a high true positive (called sensitivity) and true negative rate (called specificity) and is robust in terms of the amount of changes introduced to the program. Moreover, the sensitivity of the method increases significantly with the number of properties. While the specificity remains stable when increasing the number of properties and inputs.
This paper presents the first concolic testing framework explicitly designed for quantum programs. The framework introduces quantum constraint generation methods for quantum control statements that quantify quantum states and offers a symbolization method for quantum variables. Based on this framework, we generate path constraints for each concrete execution path of a quantum program. These constraints guide the exploration of new paths, with a quantum constraint solver determining outcomes to create novel input samples, thereby enhancing branch coverage. Our framework has been implemented in Python and integrated with Qiskit for practical evaluation. Experimental results show that our concolic testing framework improves branch coverage, generates high-quality quantum input samples, and detects bugs, demonstrating its effectiveness and efficiency in quantum programming and bug detection. Regarding branch coverage, our framework achieves more than 74.27% on quantum programs with under 5 qubits.