Noisy Intermediate-Scale Quantum (NISQ) hardware has transformed quantum computing into a practical experimental platform while introducing fundamental challenges associated with hardware noise, decoherence, and probabilistic execution. Although quantum software testing (QST) has rapidly emerged as an active research area, it remains unclear to what extent existing testing methodologies explicitly engage with noisy and hardware-constrained execution environments. This paper presents a systematic empirical landscape analysis of noise-awareness in QST research through a deterministic, taxonomy-guided analytical framework applied to 61 papers published between 2018 and 2025 across eight testing paradigms, contrasting how contributions are framed with how they are evaluated. Among the analyzed papers, 47 (77.0%) are noise-oblivious and only 14 (23.0%) explicitly engage with noise-related execution concerns, while only 11 (18.0%) report evaluation on physical hardware or a noisy simulator. Test generation, the largest paradigm, exhibits a noise-awareness rate of only 14.3%, and debugging and repair 18.2%; coverage-based, metamorphic, and differential testing contain no noise-aware contributions. Noise-aware methodologies emerged late and remain outnumbered in all analyzed years. Framing and evaluation are strongly associated (odds ratio 40.5, p < 0.001) yet diverge for individual papers: 5 of the 14 noise-aware papers evaluate only on ideal simulators. Inferential statistical tests are reported in only 16.4% of papers. The classification framework was validated against a blinded, human-coded reference sample, achieving almost perfect inter-rater agreement on testing-technique classification and substantial agreement on noise-awareness. The framework provides a reusable empirical basis for longitudinal analysis of hardware-conscious testing practice.
Noisy Intermediate-Scale Quantum (NISQ) hardware has transformed quantum computing into a practical experimental platform while introducing fundamental challenges associated with hardware noise, decoherence, and probabilistic execution. Although quantum software testing (QST) has rapidly emerged as an active research area, it remains unclear to what extent existing testing methodologies explicitly engage with noisy and hardware-constrained execution environments. This paper presents a systematic empirical landscape analysis of noise-awareness in QST research through a deterministic, taxonomy-guided analytical framework applied to 61 papers published between 2018 and 2025 across eight testing paradigms, contrasting how contributions are framed with how they are evaluated. Among the analyzed papers, 47 (77.0%) are noise-oblivious and only 14 (23.0%) explicitly engage with noise-related execution concerns, while only 11 (18.0%) report evaluation on physical hardware or a noisy simulator. Test generation, the largest paradigm, exhibits a noise-awareness rate of only 14.3%, and debugging and repair 18.2%; coverage-based, metamorphic, and differential testing contain no noise-aware contributions. Noise-aware methodologies emerged late and remain outnumbered in all analyzed years. Framing and evaluation are strongly associated (odds ratio 40.5, p < 0.001) yet diverge for individual papers: 5 of the 14 noise-aware papers evaluate only on ideal simulators. Inferential statistical tests are reported in only 16.4% of papers. The classification framework was validated against a blinded, human-coded reference sample, achieving almost perfect inter-rater agreement on testing-technique classification and substantial agreement on noise-awareness. The framework provides a reusable empirical basis for longitudinal analysis of hardware-conscious testing practice.
Quantum computing in the Noisy IntermediateScale Quantum (NISQ) era presents significant challenges in differentiating quantum software bugs from hardware noise. Traditional debugging techniques from classical software engineering cannot directly resolve this issue due to the inherently stochastic nature of quantum computation mixed with noises from NISQ computers. To address this gap, we propose a statistical approach leveraging probabilistic metrics to differentiate between quantum software bugs and hardware noise. We evaluate our methodology empirically using well-known quantum algorithms, including Grover's algorithm, Deutsch-Jozsa algorithm, and Simon's algorithm. Experimental results demonstrate the efficacy and practical applicability of our approach, providing quantum software developers with a reliable analytical tool to identify and classify unexpected behavior in quantum programs.
In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity. Since QST evaluates quantum programs through execution under designed test inputs, empirical studies are widely used to assess the effectiveness of testing approaches. However, the design and reporting of empirical studies in QST remain highly diverse, and a shared methodological understanding has yet to emerge, making it difficult to interpret results and compare findings across studies. This paper presents a methodological analysis of empirical studies in QST through a systematic examination of 59 primary studies identified from a literature pool of size 384. We organize our analysis around ten research questions that cover key methodological dimensions of QST empirical studies, including objects under test, baseline comparison, testing setup, experimental configuration, and tool and artifact support. Through cross-study analysis along these dimensions, we characterize current empirical practices in QST, identify recurring limitations and inconsistencies, and highlight open methodological challenges. Based on our findings, we derive insights and recommendations to inform the design, execution, and reporting of future empirical studies in QST.
Quantum software testing (QST) is a central quality-assurance activity for checking whether quantum programs conform to their specifications. In the noisy intermediate-scale quantum (NISQ) era, gate, decoherence, and measurement errors distort observed outputs, making it difficult to distinguish program defects from noise. The existing noise-aware QST literature typically mitigates or models noise before applying a test oracle. However, these strategies can incur substantial cost and depend on calibration data, model training, or backend-specific information, limiting their applicability. We present Qomino, a dominance-guided QST approach for judging tests from noisy samples without a separate noise-estimation or mitigation phase. The approach combines decomposition-aware inverse construction, comparison of the expected outcome with the strongest observed competitor, and bounded retries for inconclusive decisions. We evaluate Qomino on 30 controlled buggy variants from six Qiskit programs and six simulated noisy backends, comparing it with six baselines spanning statistical, quantum-specific, and learning-based methods. Qomino achieves 85.65% accuracy over all six programs and 99.95% in the restricted two-program evaluation, significantly outperforming the corresponding baselines. Across the evaluated programs and backends under the stated settings, Qomino is faster than the compared quantum-specific and learning-based methods. Ablation and retry analyses support the contributions of its main components.
This paper presents preliminary findings from a multivocal literature review investigating how quantum software testing is characterized across academic and practitioner-oriented sources. Our study integrated peer-reviewed studies with gray literature, including blogs, tutorials, forums, technical reports, documentation pages, and company webpages. Our results indicate a rapidly evolving but fragmented ecosystem involving classical adapted testing approaches, quantum-specific techniques, statistical validation methods, simulators, debugging environments, and verification frameworks. The reviewed material also revealed recurring challenges related to scalability limitations, hardware noise, probabilistic execution, limited observability, and immature tooling ecosystems. These findings provide an initial characterization of how research and practice currently discuss quantum software testing challenges, techniques, and tooling.
Abstract Quantum optimization has become a leading application for near-term quantumcomputing, and yet many publications compare algorithms against idealizedassumptions and small toy benchmarks. This limits the interpretability, reproducibility,and practical relevance of reported performance gains, particularlyin the noisy intermediate-scale quantum (NISQ) era. In this work, we presenta systematic literature review that investigate quantum optimization beyondtoy benchmarks. Following established SLR protocols, we analyze the literaturealong multiple methodological dimensions, including algorithmic approach,benchmark realism, encoding strategies, hybrid quantum-classical workflows,hardware and noise modeling, evaluation metrics, and reporting practices. Weintroduce a unified taxonomy that captures the interaction between problemformulation, encoding overhead, noise-aware execution, and hybrid optimizationloops. In addition, we propose a reproducibility checklist and scoring rubric toassess reporting completeness and experimental rigor across studies. Instead ofdeveloping new quantum optimization algorithms or making theoretical quantumadvantage claims, the main contribution of this work is that it providesa methodological analysis on benchmarking realism and encoding or evaluationpractices as well as reproducibility rigor in NISQ-era quantum optimization studies.Our review uncovers a continued chasm between algorithmic innovation andevaluation maturity, with the number of publications and diversity of methodsgrowing without corresponding growth in standardized benchmarks, strongclassical baselines, noise-consistent evaluation, or reproducibility.
Mutation analysis has long been used in classical software testing and has recently been adopted for assessing the robustness of quantum software testing techniques. However, existing studies assume ideal, noiseless execution, overlooking the impact of quantum hardware noise. In this paper, we present an empirical study of noise-aware mutation analysis for quantum programs. We analyze how noise affects mutant detection using 41 quantum programs, executed on noiseless and noisy simulators emulating three IBM devices with different noise profiles. We compare several distance metrics and thresholding strategies to evaluate mutant detection under realistic noise. Our results show that noise significantly alters the behavioral distance between programs and mutants, making equivalent mutants harder to distinguish from real faults. Density-matrix metrics achieve the best discrimination, with misclassification rates up to 16.77%, but are not accessible on real hardware. Among practical alternatives, output-distribution metrics reach up to 73.03% accuracy and 74.89% F1-score. Noise-specific thresholds further improve detection compared to noiseless thresholds. We also find that noise effects correlate more with algorithm and circuit characteristics than with mutation types. Overall, our results highlight the need to adapt mutation analysis, and more generally quantum program comparison, to the noise profiles of target quantum devices.