Quantum Computing is regarded with a growing sense of excitement and expectation. While the preliminary evidences of its practical interest accumulate, there are numerous challenges that must be addressed before quantum computing can reach generalized adoption and realize its full potential. One such challenge concerns assessing whether a quantum program is correct, namely in the sense that it implements the desired functionality. In the classical realm, program testing is a well-established approach to find defects in programs. While this approach should inspire quantum software development, porting classical testing techniques to the quantum world is by no means trivial. We propose a novel approach to test quantum programs. Our approach is based on the use of metamorphic relations, which allows us to overcome the well-known quantum measurement problem. We describe a series of exploratory experiments whose results provide evidence of the usefulness of our approach.
Quantum Computing is regarded with a growing sense of excitement and expectation. While the preliminary evidences of its practical interest accumulate, there are numerous challenges that must be addressed before quantum computing can reach generalized adoption and realize its full potential. One such challenge concerns assessing whether a quantum program is correct, namely in the sense that it implements the desired functionality. In the classical realm, program testing is a well-established approach to find defects in programs. While this approach should inspire quantum software development, porting classical testing techniques to the quantum world is by no means trivial. We propose a novel approach to test quantum programs. Our approach is based on the use of metamorphic relations, which allows us to overcome the well-known quantum measurement problem. We describe a series of exploratory experiments whose results provide evidence of the usefulness of our approach.
Quantum computing is infused with an increased sense of anticipation and excitement. However, several obstacles need to be overcome before quantum computing can be widely adopted. One of these is determining whether or not a quantum program is behaving according to the expectation. There are several tried-and-true methods for identifying program flaws in the traditional world, but it is far from simple to adapt them to the quantum realm. In this paper, we explore Metamorphic Testing as a promising avenue. We explore this avenue on the well-known Shor's Algorithm, and show how it can be used to verify the validity of this testing approach on quantum algorithms.
As quantum computing is becoming increasingly popular, the underlying quantum computing platforms are growing both in ability and complexity. Unfortunately, testing these platforms is challenging due to the relatively small number of existing quantum programs and because of the oracle problem, i.e., a lack of specifications of the expected behavior of programs. This paper presents MorphQ, the first metamorphic testing approach for quantum computing platforms. Our two key contributions are (i) a program generator that creates a large and diverse set of valid (i.e., non-crashing) quantum programs, and (ii) a set of program transformations that exploit quantum-specific metamorphic relationships to alleviate the oracle problem. Evaluating the approach by testing the popular Qiskit platform shows that the approach creates over 8k program pairs within two days, many of which expose crashes. Inspecting the crashes, we find 13 bugs, nine of which have already been confirmed. MorphQ widens the slim portfolio of testing techniques of quantum computing platforms, helping to create a reliable software stack for this increasingly important field.
Quantum computing has been rapidly expanding, and many platforms for writing programs that can be compiled and run on quantum hardware (or simulated) are being developed. As with any compiler, transformation correctness is paramount as the machine readable code must maintain program semantics. Quantum programs are challenging to test given a lack of benchmark programs and the difficulty of defining an oracle. MorphQ solves both of these challenges by (1) generating syntactically correct quantum programs and (2) using metamorphic testing to avoid the oracle problem. However, Qiskit, the compiler it was built for, is rapidly evolving. This paper is a reproducibility study of the MorphQ platform, which we call MorphQ++. We have updated the core MorphQ engine to work on a newer version of Qiskit and added new metamorphic transformations. We find that our overall results are similar: we find a portion of the original faults (which were not yet fixed) and the distributions of types are not very different. Our new transformations lead to several new faults, suggesting there is room to expand the core framework. Additionally, we note the lack of power of the metamorphic relations in this context and suggest the need for more sophisticated relations and/or oracle evaluations.
Quantum computing has been rapidly expanding, and many platforms for writing programs that can be compiled and run on quantum hardware (or simulated) are being developed. As with any compiler, transformation correctness is paramount as the machine readable code must maintain program semantics. Quantum programs are challenging to test given a lack of benchmark programs and the difficulty of defining an oracle. MorphQ solves both of these challenges by (1) generating syntactically correct quantum programs and (2) using metamorphic testing to avoid the oracle problem. However, Qiskit, the compiler it was built for, is rapidly evolving. This paper is a reproducibility study of the MorphQ platform, which we call MorphQ++. We have updated the core MorphQ engine to work on a newer version of Qiskit and added new metamorphic transformations. We find that our overall results are similar: we find a portion of the original faults (which were not yet fixed) and the distributions of types are not very different. Our new transformations lead to several new faults, suggesting there is room to expand the core framework. Additionally, we note the lack of power of the metamorphic relations in this context and suggest the need for more sophisticated relations and/or oracle evaluations.
Quantum computing is a field that has drawn a lot of attention over the years as it has shown great potential to solve new problems and has seen rapid development. To meet the demands of this rapid development, software engineering researchers have been studying methods to address the unique challenges presented by the probabilistic nature of quantum computing in its software platforms. One important aspect of this is understanding how to test and validate quantum software platforms when the expected outputs when running a program are not always known. Recent research using metamorphic testing was able to detect 13 faults in the Qiskit quantum computing platform; however, this is a rapidly evolving platform. Since their work, Qiskit has already advanced ahead several versions, which brings up more questions regarding research for quantum software platforms: do these tools researchers create hold up across versions of these frequently-updated platforms, and how much work does it take to maintain and extend these tools? In this thesis, we study the ease of regression testing in the context of this quantum computing platform. We update to the latest version of Qiskit, extend two existing metamorphic transformations on a metamorphic testing platform for Qiskit, add six new metamorphic transformations, and perform a set of metamorphic testing runs to analyze how our changes perform. In doing so, we were able to discover five additional new faults on the Qiskit platform; however, we also uncovered several challenges, which we synthesize for the reader. We also offer discussion on where the future work of metamorphic testing on quantum software platforms may lead based on our challenges and successes during this process.
Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation. With the increasing interest in the field, there are challenges and opportunities for academics and practitioners in terms of software engineering practices, particularly in testing quantum programs. This paper presents an empirical study of testing patterns in quantum algorithms. We analyzed all the tests handling quantum aspects of the implementations in the Qiskit Algorithms library and identified seven distinct patterns that make use of (1) fixed seeds for algorithms based on random elements; (2) deterministic oracles; (3) precise and approximate assertions; (4) Data-Driven Testing (DDT); (5) functional testing; (6) testing for intermediate parts of the algorithms being tested; and (7) equivalence checking for quantum circuits. Our results show a prevalence of classical testing techniques to test the quantum-related elements of the library, while recent advances from the research community have yet to achieve wide adoption among practitioners.