ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)

Mutation testing of quantum programs written in QISKit

Daniel Fortunato, José Campos, Rui Abreu

Published May 21, 2022• 28 citations

Abstract

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum operations, we investigated a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further discuss QMutPy, an extension of the well-known and fully automated open-source mutation tool MutPy. To evaluate QMutPy's performance we conducted a case study on 11 real quantum programs written in the IBM's QISKit library. QMutPy has proven to be an effective quantum mutation tool, providing insight on the current state of quantum tests.

DOI: 10.1145/3510454.3528649

Related papers

Mutation Testing of Quantum Programs Written in QISKit

Daniel Fortunato, José Campos, Rui Abreu

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum operations, we investigated a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further discuss QMutPy, an extension of the well-known and fully automated open-source mutation tool MutPy. To evaluate QMutPy’s performance we conducted a case study on 11 real quantum programs written in the IBM’s QISKit library. QMutPy has proven to be an effective quantum mutation tool, providing insight on the current state of quantum tests.

ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)May 1, 20225 citations

Mutation Testing of Quantum Programs: A Case Study With Qiskit

Daniel Fortunato, José Campos, Rui Abreu

As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.

IEEE Transactions on Quantum EngineeringJan 1, 202262 citations

QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit

Daniel Fortunato, José Campos, Rui Abreu

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum gates, we describe our efforts in developing a tool, coined QMutPy, leveraging the well-known open-source mutation tool MutPy. We further discuss the design and implementation of QMutPy, and the usage of a novel set of mutation operators that generate mutants for qubit measurements and gates. To evaluate QMutPy’s performance, we conducted a preliminary study on 11 real quantum programs written in the IBM’s Qiskit library. QMutPy has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests. QMutPy is publicly available at https://github.com/danielfobooss/mutpy. Tool demo: https://youtu.be/fC4tOY5trqc.

ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)Jul 15, 202248 citations

Muskit: A Mutation Analysis Tool for Quantum Software Testing

Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini +1 more

Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917

IEEE/ACM International Conference on Automated Software Engineering (ASE)Nov 1, 202169 citations

Search-Based Generation of Undetected Quantum Circuit Mutants

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini +1 more

Quantum mutation analysis is emerging as an essential technique for evaluating test suites due to the limited availability of real faulty quantum programs. However, existing quantum mutation analysis tools use fixed gate-based mutations, resulting in easy-to-detect mutants, which reduces their effectiveness in assessing the quality of test suites. We propose QUMUG, a search-based approach for generating challenging mutants by utilising parameterisable quantum gates. QUMUG employs search algorithms to optimise mutation parameters and find non-equivalent mutants passing a given test suite. In our evaluation over 30 quantum programs, QUMUG produced mutants that are three times more challenging than the mutants generated by existing tools. Among the four evaluated search algorithms, the genetic algorithm was the most effective, generating an average of 494 undetected mutants per program with a 99.67% success rate and 94.3% non-equivalent ratio. The generated mutants demonstrated their effectiveness by requiring the addition of five times more test cases to the test suite than the mutants generated by existing tools. We also analysed the behaviour of higher order mutants in quantum circuits, and showed that while first order mutations are more effective for enhancing the test suite, higher order mutants highlight the need for new unique test cases.

arXiv:2608.09318Aug 10, 20260 citations

Development of a Tool for Finding Equivalent Mutants in Quantum Program: A Perspective to Measure the Quality of Quantum Software

Ajay Kumar, Prisha Kwatra, Sunita Garhwal

Abstract Software testing is an important activity to ensure the software's quality. Mutation testing is a fault-based technique for checking the effectiveness of test cases for the verification and validation of software. Testing quantum programs is challenging due to the involvement of quantum mechanics concepts (such as superposition and entanglement ) and the probabilistic nature of the outcome. In this paper, We will propose the modified mutation testing approach for quantum program. The proposed approach will help to reduce the cost, time, and effort of performing the mutation testing. Moreover, we also propose a method for determining the equivalent mutant and developing a tool to check whether the mutated and original quantum program behave semantically equivalent or differently.

Dec 9, 20222 citations