Errors in quantum programs are challenging to track down due to the uncertainty of quantum programs. Testing is, therefore, an indispensable method for assuring the quality of quantum software. Existing testing methods focus only on testing quantum programs with quantum circuits or single subroutines and, therefore, cannot effectively test quantum programs with multi-subroutines. In this paper, we first discuss several critical issues that must be considered when testing multi-subroutine quantum programs and point out the limitations and problems with existing testing methods. We then present a novel framework for testing multi-subroutine quantum programs that allow for both unit and integration testing. Our framework includes two novel test coverage criteria for the equivalent class partition of quantum variables to guide our testing tasks and techniques to test quantum programs with several common patterns. We also discuss how to generate test cases based on our framework. To evaluate the effectiveness of our testing framework, we implemented a tool called QSharpTester for testing Q\# programs with multiple subroutines. We used it to conduct experiments on hundreds of mutation programs deriving from seven original Q\# programs. The experimental results show that our testing methods can deal with broader types of quantum programs than existing ones and perform well on almost all faulty mutation programs.
Errors in quantum programs are challenging to track down due to the uncertainty of quantum programs. Testing is, therefore, an indispensable method for assuring the quality of quantum software. Existing testing methods focus only on testing quantum programs with quantum circuits or single subroutines and, therefore, cannot effectively test quantum programs with multi-subroutines. In this paper, we first discuss several critical issues that must be considered when testing multi-subroutine quantum programs and point out the limitations and problems with existing testing methods. We then present a novel framework for testing multi-subroutine quantum programs that allow for both unit and integration testing. Our framework includes two novel test coverage criteria for the equivalent class partition of quantum variables to guide our testing tasks and techniques to test quantum programs with several common patterns. We also discuss how to generate test cases based on our framework. To evaluate the effectiveness of our testing framework, we implemented a tool called QSharpTester for testing Q# programs with multiple subroutines. We used it to conduct experiments on hundreds of mutation programs deriving from seven original Q# programs. The experimental results show that our testing methods can deal with broader types of quantum programs than existing ones and perform well on almost all faulty mutation programs.
Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria.
Abstract Software testing is an important activity to ensure the software's quality. Mutation testing is a fault-based technique for checking the effectiveness of test cases for the verification and validation of software. Testing quantum programs is challenging due to the involvement of quantum mechanics concepts (such as superposition and entanglement ) and the probabilistic nature of the outcome. In this paper, We will propose the modified mutation testing approach for quantum program. The proposed approach will help to reduce the cost, time, and effort of performing the mutation testing. Moreover, we also propose a method for determining the equivalent mutant and developing a tool to check whether the mutated and original quantum program behave semantically equivalent or differently.
Abstract Software testing is a critical component of the classical software development lifecycle, and this principle is expected to hold true for quantum software as it evolves toward large-scale production and adherence to industry standards. Developing and testing quantum software presents unique challenges due to the non-deterministic nature of quantum information, the high dimensionality of the underlying Hilbert space, complex hardware noise, and the inherent non-local properties of quantum systems. In this work, we propose a unifying theoretical framework based on probabilistic assertions, which encompasses several testing approaches—such as quantum tomography and statistical tests—for developing practical unit tests for quantum subroutines. The framework is built upon the semantic equivalence between quantum subroutines and parameterized quantum channels, as established in this work. To address the computational complexity associated with unit testing in quantum systems, we propose incorporating context-awareness into the testing process. The trade-offs between accuracy, state space coverage, and efficiency associated with the proposed theoretical framework for quantum unit testing have been demonstrated through its application to a simple three-qubit quantum subroutine that prepares a Greenberger–Horne–Zeilinger state, as well as to subroutines within a program implementing Shor’s algorithm.
Abstract Software testing is an important activity to ensure the software's quality. Mutation testing is a fault-based technique for checking the effectiveness of test cases for the verification and validation of software. Testing quantum programs is challenging due to the involvement of quantum mechanics concepts (such as superposition and entanglement ) and the probabilistic nature of the outcome. In this paper, We will propose the modified mutation testing approach for quantum program. The proposed approach will help to reduce the cost, time, and effort of performing the mutation testing. Moreover, we also propose a method for determining the equivalent mutant and developing a tool to check whether the mutated and original quantum program behave semantically equivalent or differently.
As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.
Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917