Quantum computing is infused with an increased sense of anticipation and excitement. However, several obstacles need to be overcome before quantum computing can be widely adopted. One of these is determining whether or not a quantum program is behaving according to the expectation. There are several tried-and-true methods for identifying program flaws in the traditional world, but it is far from simple to adapt them to the quantum realm. In this paper, we explore Metamorphic Testing as a promising avenue. We explore this avenue on the well-known Shor's Algorithm, and show how it can be used to verify the validity of this testing approach on quantum algorithms.
Quantum computing is infused with an increased sense of anticipation and excitement. However, several obstacles need to be overcome before quantum computing can be widely adopted. One of these is determining whether or not a quantum program is behaving according to the expectation. There are several tried-and-true methods for identifying program flaws in the traditional world, but it is far from simple to adapt them to the quantum realm. In this paper, we explore Metamorphic Testing as a promising avenue. We explore this avenue on the well-known Shor's Algorithm, and show how it can be used to verify the validity of this testing approach on quantum algorithms.
Quantum Computing is regarded with a growing sense of excitement and expectation. While the preliminary evidences of its practical interest accumulate, there are numerous challenges that must be addressed before quantum computing can reach generalized adoption and realize its full potential. One such challenge concerns assessing whether a quantum program is correct, namely in the sense that it implements the desired functionality. In the classical realm, program testing is a well-established approach to find defects in programs. While this approach should inspire quantum software development, porting classical testing techniques to the quantum world is by no means trivial. We propose a novel approach to test quantum programs. Our approach is based on the use of metamorphic relations, which allows us to overcome the well-known quantum measurement problem. We describe a series of exploratory experiments whose results provide evidence of the usefulness of our approach.
Quantum computing has the potential to solve a wide variety of classically intractable problems. However, the inherent complexity, probabilistic behavior, and fragility of quantum systems pose significant challenges to ensuring the correctness and reliability of quantum programs. In this survey, we present a comprehensive overview of functional testing and validation techniques aimed at addressing these challenges for quantum circuits. We begin by outlining the unique characteristics of quantum systems that complicate traditional testing methodologies, including the absence of classical oracles, non-determinism due to quantum measurement, and the exponential growth of the state space. We then categorize and analyze a wide range of functional testing and validation techniques that have emerged to meet these challenges, including mutation testing, fuzz testing, and coverage testing. We further explore advanced semantic and white-box methods such as property-based testing, metamorphic testing, equivalence checking, fault testing, assertions, and concolic testing. Specifically, we discuss the applicability of these testing techniques to quantum circuits and examine their limitations in the current NISQ (Noisy Intermediate-Scale Quantum) era.
As quantum computing is becoming increasingly popular, the underlying quantum computing platforms are growing both in ability and complexity. Unfortunately, testing these platforms is challenging due to the relatively small number of existing quantum programs and because of the oracle problem, i.e., a lack of specifications of the expected behavior of programs. This paper presents MorphQ, the first metamorphic testing approach for quantum computing platforms. Our two key contributions are (i) a program generator that creates a large and diverse set of valid (i.e., non-crashing) quantum programs, and (ii) a set of program transformations that exploit quantum-specific metamorphic relationships to alleviate the oracle problem. Evaluating the approach by testing the popular Qiskit platform shows that the approach creates over 8k program pairs within two days, many of which expose crashes. Inspecting the crashes, we find 13 bugs, nine of which have already been confirmed. MorphQ widens the slim portfolio of testing techniques of quantum computing platforms, helping to create a reliable software stack for this increasingly important field.
Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation. With the increasing interest in the field, there are challenges and opportunities for academics and practitioners in terms of software engineering practices, particularly in testing quantum programs. This paper presents an empirical study of testing patterns in quantum algorithms. We analyzed all the tests handling quantum aspects of the implementations in the Qiskit Algorithms library and identified seven distinct patterns that make use of (1) fixed seeds for algorithms based on random elements; (2) deterministic oracles; (3) precise and approximate assertions; (4) Data-Driven Testing (DDT); (5) functional testing; (6) testing for intermediate parts of the algorithms being tested; and (7) equivalence checking for quantum circuits. Our results show a prevalence of classical testing techniques to test the quantum-related elements of the library, while recent advances from the research community have yet to achieve wide adoption among practitioners.
Quantum computing is a field that has drawn a lot of attention over the years as it has shown great potential to solve new problems and has seen rapid development. To meet the demands of this rapid development, software engineering researchers have been studying methods to address the unique challenges presented by the probabilistic nature of quantum computing in its software platforms. One important aspect of this is understanding how to test and validate quantum software platforms when the expected outputs when running a program are not always known. Recent research using metamorphic testing was able to detect 13 faults in the Qiskit quantum computing platform; however, this is a rapidly evolving platform. Since their work, Qiskit has already advanced ahead several versions, which brings up more questions regarding research for quantum software platforms: do these tools researchers create hold up across versions of these frequently-updated platforms, and how much work does it take to maintain and extend these tools? In this thesis, we study the ease of regression testing in the context of this quantum computing platform. We update to the latest version of Qiskit, extend two existing metamorphic transformations on a metamorphic testing platform for Qiskit, add six new metamorphic transformations, and perform a set of metamorphic testing runs to analyze how our changes perform. In doing so, we were able to discover five additional new faults on the Qiskit platform; however, we also uncovered several challenges, which we synthesize for the reader. We also offer discussion on where the future work of metamorphic testing on quantum software platforms may lead based on our challenges and successes during this process.
Quantum computing has been rapidly expanding, and many platforms for writing programs that can be compiled and run on quantum hardware (or simulated) are being developed. As with any compiler, transformation correctness is paramount as the machine readable code must maintain program semantics. Quantum programs are challenging to test given a lack of benchmark programs and the difficulty of defining an oracle. MorphQ solves both of these challenges by (1) generating syntactically correct quantum programs and (2) using metamorphic testing to avoid the oracle problem. However, Qiskit, the compiler it was built for, is rapidly evolving. This paper is a reproducibility study of the MorphQ platform, which we call MorphQ++. We have updated the core MorphQ engine to work on a newer version of Qiskit and added new metamorphic transformations. We find that our overall results are similar: we find a portion of the original faults (which were not yet fixed) and the distributions of types are not very different. Our new transformations lead to several new faults, suggesting there is room to expand the core framework. Additionally, we note the lack of power of the metamorphic relations in this context and suggest the need for more sophisticated relations and/or oracle evaluations.