arXiv:2605.19736 · IEEE/ACM International Conference on Automated Software Engineering (ASE)

QUTest: A Native Testing Framework for Quantum Programs

José Campos

quant-phcs.SESubmitted May 19, 2026• Announced May 20, 2026• 0 citations

Abstract

Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.

Related papers

QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit

Gabriel Pontolillo, Mohammad Reza Mousavi, Marek Grzesiuk

Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.

arXiv:2503.22641ACM Transactions on Quantum ComputingMay 8, 20264 citations

QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit

Daniel Fortunato, José Campos, Rui Abreu

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum gates, we describe our efforts in developing a tool, coined QMutPy, leveraging the well-known open-source mutation tool MutPy. We further discuss the design and implementation of QMutPy, and the usage of a novel set of mutation operators that generate mutants for qubit measurements and gates. To evaluate QMutPy’s performance, we conducted a preliminary study on 11 real quantum programs written in the IBM’s Qiskit library. QMutPy has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests. QMutPy is publicly available at https://github.com/danielfobooss/mutpy. Tool demo: https://youtu.be/fC4tOY5trqc.

ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)Jul 15, 202248 citations

Mutation Testing of Quantum Programs: A Case Study With Qiskit

Daniel Fortunato, José Campos, Rui Abreu

As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.

IEEE Transactions on Quantum EngineeringJan 1, 202262 citations

Muskit: A Mutation Analysis Tool for Quantum Software Testing

Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini +1 more

Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917

IEEE/ACM International Conference on Automated Software Engineering (ASE)Nov 1, 202169 citations

QuSBT: Search-Based Testing of Quantum Programs

Xinyi Wang, Paolo Arcaini, Tao Yue +1 more

Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we present a test generation tool for quantum programs based on a genetic algorithm, called QuSBT (Search-based Testing of Quantum Programs). QuSBT automates the testing of quantum programs, with the aim of finding a test suite having the maximum number of failing test cases. QuSBT utilizes IBM’s Qiskit as the simulation framework for quantum programs. We present the tool architecture in addition to the implemented methodology (i.e., the encoding of the search individual, the definition of the fitness function expressing the search problem, and the test assessment w.r.t. two types of failures). Finally, we report results of the experiments in which we tested a set of faulty quantum programs with QuSBT to assess its effectiveness. Repository (code and experimental results): https://github.com/Simula-COMPLEX/qusbt-tool Video: https://youtu.be/3apRCtluAn4

ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)May 1, 20227 citations

A Framework for the Efficient Evaluation of Runtime Assertions on Quantum Computers

Damian Rovara, Lukas Burgholzer, Robert Wille

The continuous growth of quantum computing and the increasingly complex quantum programs resulting from it lead to unprecedented obstacles in ensuring program correctness. Runtime assertions are, therefore, becoming a crucial tool in the development of quantum programs. They assist developers in the debugging process and help to test and verify the program. However, while assertions can be implemented in a straightforward manner on classical computers, physical limitations of quantum computers pose considerable challenges for the evaluation of quantum assertions. Access to the quantum state of a program is limited, execution time is expensive and noise can significantly distort measurement outcomes. To address these problems, this work proposes a framework that assists developers in the evaluation of runtime assertions on real quantum computers. It translates a variety of assertions into sets of measurements, reduces execution overhead where possible and evaluates the measurement results after the execution even in the presence of noise. This approach substantially aids developers in the debugging process, enabling efficient assertion-driven debugging even in large programs. The proposed framework is available as an open-source implementation at https://github.com/munich-quantum-toolkit/debugger

arXiv:2505.03885IEEE International Conference on Quantum Software (QSW)Jul 1, 20260 citations