Quantum computing has been attracting the attention of both applied research and companies. Continuous progress on fundamental hardware technology promises to bring us more reliable and large-scale quantum computers on which to run the next generation of quantum algorithms. These programs are compiled and executed on dedicated platforms, and similarly to classical programs, a large effort is required to test these platforms and create a robust software toolchain.Unlike previous studies which focused on cross-optimization and cross-backend testing, this dissertation aims to create the first approach for cross-platform testing which compares execution on diverse quantum computing platforms. To inform the design of the method, we will first perform an empirical study of bugs in quantum computing platforms and a review of the characteristics of realistic quantum programs.The final approach for cross-platform testing will include three components: a learning-based method to generate realistic quantum programs, an approach to map and run them on multiple platforms, and finally a quantum-specific statistical test to compare two multivariate binary distributions returned as the output of quantum programs.
Quantum computing has been attracting the attention of both applied research and companies. Continuous progress on fundamental hardware technology promises to bring us more reliable and large-scale quantum computers on which to run the next generation of quantum algorithms. These programs are compiled and executed on dedicated platforms, and similarly to classical programs, a large effort is required to test these platforms and create a robust software toolchain.Unlike previous studies which focused on cross-optimization and cross-backend testing, this dissertation aims to create the first approach for cross-platform testing which compares execution on diverse quantum computing platforms. To inform the design of the method, we will first perform an empirical study of bugs in quantum computing platforms and a review of the characteristics of realistic quantum programs.The final approach for cross-platform testing will include three components: a learning-based method to generate realistic quantum programs, an approach to map and run them on multiple platforms, and finally a quantum-specific statistical test to compare two multivariate binary distributions returned as the output of quantum programs.
The interest in quantum computing is growing, and with it, the importance of software platforms to develop quantum programs. Ensuring the correctness of such platforms is important, and it requires a thorough understanding of the bugs they typically suffer from. To address this need, this paper presents the first in-depth study of bugs in quantum computing platforms. We gather and inspect a set of 223 real-world bugs from 18 open-source quantum computing platforms. Our study shows that a significant fraction of these bugs (39.9%) are quantum-specific, calling for dedicated approaches to prevent and find them. The bugs are spread across various components, but quantum-specific bugs occur particularly often in components that represent, compile, and optimize quantum programming abstractions. Many quantum-specific bugs manifest through unexpected outputs, rather than more obvious signs of misbehavior, such as crashes. Finally, we present a hierarchy of recurrent bug patterns, including ten novel, quantum-specific patterns. Our findings not only show the importance and prevalence bugs in quantum computing platforms, but they help developers to avoid common mistakes and tool builders to tackle the challenge of preventing, finding, and fixing these bugs.
Quantum computing platforms are susceptible to quantum-specific bugs (e.g., incorrect ordering of qubits or incorrect implementation of quantum abstractions), which are difficult to detect and require specialized expertise. The field faces challenges due to a fragmented landscape of platforms and rapid development cycles that often prioritize features over the development of robust platform testing frameworks, severely hindering the reliability of quantum software. To address these challenges, we present QITE, the first cross-platform testing framework for quantum computing platforms, which leverages QASM, an assembly-level representation, to ensure consistency across different platforms. QITE introduces the novel ITE process to generate equivalent quantum programs by iteratively (I)mporting assembly into platform representations, (T)ransforming via platform optimization and gate conversion, and (E)xporting back to assembly. It uses a crash oracle to detect failures during cross-platform transformations and an equivalence oracle to validate the semantic consistency of the final sets of assembly programs, which are expected to be equivalent by construction. We evaluate QITE on four widely-used quantum computing platforms: Qiskit, PennyLane, Pytket, and BQSKit, revealing 17 bugs, 14 of which are already confirmed or even fixed. Our results demonstrate QITE's effectiveness, its complementarity to existing quantum fuzzers in terms of code coverage, and its ability to expose bugs that have been out of reach for existing testing techniques.
Quantum computing platforms are susceptible to quantum-specific bugs, such as incorrectly ordering qubits or incorrectly implementing quantum abstractions. These bugs are difficult to detect and require specialized expertise. The field faces challenges due to a fragmented landscape of platforms and rapid development cycles that often prioritize new features over thorough testing, severely hindering the reliability of quantum software. To address these challenges, we present IterTestQ, a novel cross-platform testing approach for quantum computing platforms. The key technical contribution is our novel ITE process, which generates equivalent quantum programs by iteratively (I)mporting them into platform-specific representations, (T)ransforming the program via optimizations and gate conversions, and (E)xporting the program again. To transfer programs across platforms and test cross-platform consistency, IterTestQ leverages QASM, an assembly-level representation supported by most platforms. The approach uses a crash oracle to detect failures during cross-platform transformations and an equivalence oracle to validate the semantic consistency of the generated assembly programs, which are expected to be equivalent by construction. We evaluate IterTestQ on widely-used quantum computing platforms, including Qiskit, PennyLane, Pytket, BQSKit, and Cirq, revealing 23 bugs, 17 of which are already confirmed or fixed. Our results also demonstrate that IterTestQ complements existing quantum fuzzers (covering tens of thousands of otherwise uncovered lines), is efficient (with 0.00089 seconds per generated program), and that the ITE process is crucial for its effectiveness.
Quantum computing is getting increasing interest from both academia and industry, and the quantum software landscape has been growing rapidly. The quantum software stack comprises quantum programs, implementing algorithms, and platforms like IBM Qiskit, Google Cirq, and Microsoft Q#, enabling their development. To ensure the reliability and performance of quantum software, various techniques for testing and analyzing it have been proposed, such as test generation, bug pattern detection, and circuit optimization. However, the large amount of work and the fact that work on quantum software is performed by several research communities, make it difficult to get a comprehensive overview of the existing techniques. In this work, we provide an extensive survey of the state of the art in testing and analysis of quantum software. We discuss literature from several research communities, including quantum computing, software engineering, programming languages, and formal methods. Our survey covers a wide range of topics, including expected and unexpected behavior of quantum programs, testing techniques, program analysis approaches, optimizations, and benchmarks for testing and analyzing quantum software. We create novel connections between the discussed topics and present them in an accessible way. Finally, we discuss key challenges and open problems to inspire future research.
Quantum computing is a field that has drawn a lot of attention over the years as it has shown great potential to solve new problems and has seen rapid development. To meet the demands of this rapid development, software engineering researchers have been studying methods to address the unique challenges presented by the probabilistic nature of quantum computing in its software platforms. One important aspect of this is understanding how to test and validate quantum software platforms when the expected outputs when running a program are not always known. Recent research using metamorphic testing was able to detect 13 faults in the Qiskit quantum computing platform; however, this is a rapidly evolving platform. Since their work, Qiskit has already advanced ahead several versions, which brings up more questions regarding research for quantum software platforms: do these tools researchers create hold up across versions of these frequently-updated platforms, and how much work does it take to maintain and extend these tools? In this thesis, we study the ease of regression testing in the context of this quantum computing platform. We update to the latest version of Qiskit, extend two existing metamorphic transformations on a metamorphic testing platform for Qiskit, add six new metamorphic transformations, and perform a set of metamorphic testing runs to analyze how our changes perform. In doing so, we were able to discover five additional new faults on the Qiskit platform; however, we also uncovered several challenges, which we synthesize for the reader. We also offer discussion on where the future work of metamorphic testing on quantum software platforms may lead based on our challenges and successes during this process.