Quantum compilers are judged correct by an output-equivalence oracle: the compiled circuit must compute the same unitary as the original, modulo global phase and a qubit-layout permutation. This oracle, by construction, checks only that semantic map, not the circuit's own layout, permutation, or phase records: a defect there, or in a fixed-seed run's determinism, can pass unseen though the record is public. We measure how often this happens in real merged compiler fixes: a systematically identified corpus of Qiskit transpiler bug-fixes, classified by an independently dual-coded, source-validated manifestation taxonomy. Nineteen of 68 fixes (28%, 95% Wilson CI 19-40%) repair faults that this equivalence screen does not catch, even one augmented with compilation-validity, circuit-quality, and performance checks, and an extended 104-fix corpus over a wider window holds at the same rate with a tighter interval (29/104, 27.9%, CI 20-37%). A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel (a corrupted layout or permutation record): 10 of 68 fixes (15%, CI 8-25%) beneath the 28% headline. The gap is not Qiskit-specific: it replicates in tket (7 of 21, 33%), with Cirq smaller but consistent. We detected no systematic differences on five inexpensive PR-level characteristics (19 vs 49, underpowered on its own; the same comparison on the extended 29-vs-75 corpus tightens every interval toward zero). This class dominates the invisible set, concentrating at representation-boundary crossings. We release the corpus, codebook, and coding artifacts. Here we only measure it.
Quantum compilers are judged correct by an output-equivalence oracle: the compiled circuit must compute the same unitary as the original, modulo global phase and a qubit-layout permutation. This oracle, by construction, checks only that semantic map, not the circuit's own layout, permutation, or phase records: a defect there, or in a fixed-seed run's determinism, can pass unseen though the record is public. We measure how often this happens in real merged compiler fixes: a systematically identified corpus of Qiskit transpiler bug-fixes, classified by an independently dual-coded, source-validated manifestation taxonomy. Nineteen of 68 fixes (28%, 95% Wilson CI 19-40%) repair faults that this equivalence screen does not catch, even one augmented with compilation-validity, circuit-quality, and performance checks, and an extended 104-fix corpus over a wider window holds at the same rate with a tighter interval (29/104, 27.9%, CI 20-37%). A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel (a corrupted layout or permutation record): 10 of 68 fixes (15%, CI 8-25%) beneath the 28% headline. The gap is not Qiskit-specific: it replicates in tket (7 of 21, 33%), with Cirq smaller but consistent. We detected no systematic differences on five inexpensive PR-level characteristics (19 vs 49, underpowered on its own; the same comparison on the extended 29-vs-75 corpus tightens every interval toward zero). This class dominates the invisible set, concentrating at representation-boundary crossings. We release the corpus, codebook, and coding artifacts. Here we only measure it.
Test oracles for quantum transpilers typically judge correctness by comparing compiled output against a reference: a statevector, a sampled distribution, or a unitary compared modulo global phase. A companion empirical study measures how often that choice fails. Roughly 28% of merged Qiskit transpiler bug-fixes (95% Wilson CI 19-40%) repair a fault that corrupts layout metadata, global phase, or run-to-run reproducibility while output stays correct: invisible to a black-box output-equivalence oracle by construction. This paper closes that gap with a fault-class-matched, layout-aware, width-tiered oracle family: a layout/permutation contract checker and a contract-level metamorphic relation (MR-1) for the metadata channel, a global-phase tracker for the phase channel, and a determinism runner for reproducibility. Verified from source on nine real, merged Qiskit transpiler regressions (three per channel), the output-equivalence oracle is blind throughout and the matched mechanism fires on every case. A 675-configuration sweep of the contract/metadata invariant finds no false positive. Synthetic mutant families confirm reliability at scale: 1.00 sensitivity and specificity across 36 mutants apiece for the contract/metadata and global-phase channels, and 1.00 sensitivity (95% CI 0.44-1.00) for reproducibility on the three circuits where the mutation is constructible. The contract checker costs two to six orders of magnitude less than a plain output check, the global-phase tracker is comparably cheap within its exact tier, and only the metamorphic relation carries a bounded cost. Ported natively to pytket/tket, the global-phase mechanism transfers cleanly, an identical 1.00/1.00 result with phases recovered to double-precision accuracy, evidence against a Qiskit-specific artifact.
Quantum software relies on compilers, intermediate representations, and execution frameworks that transform programs before execution. Faults in these components can change the behavior of otherwise correct quantum programs. Yet quantum testing techniques are often evaluated using mutation-generated faults and curated bug benchmarks. We investigate how well the faults represented by these artifacts reflect those encountered in modern quantum software. We audit four quantum mutation-testing tools and three curated bug benchmarks, and collect 410 verified bugs from ten quantum software repositories over two years. We classify faults from all three sources into a common taxonomy of 56 bug families and compare their representation across mutation operators, curated benchmarks, and recent real-world bugs. The mutation tools support 20 bug families, and curated benchmarks add 15 more. Yet only 56 of the 410 real-world bugs (13.7%) belong to families represented by either source; the remaining 354 (86.3%) belong to families absent from both. The largest unrepresented families concern compiler transformations (66 bugs), type handling (53), and input validation (47). Unlike local gate or statement changes, these faults often depend on intermediate representations, compiler pipelines, input types, or dependency versions. Existing mutation operators primarily modify circuits and program constructs, whereas many real-world faults arise in the compilers, representations, and framework interfaces that process them. Addressing this gap requires looking beyond circuit-level faults toward testing semantic preservation across compiler transformations, correctness at representation boundaries, and interactions between software components. Our study provides empirical evidence to guide fault models and benchmarks that capture these mechanisms and preserve the contexts needed to reproduce them.
A compiler pass over a quantum error correction circuit must preserve what the circuit computes. The artefacts a practitioner reaches for when reasoning about such a circuit, detector samples and the detector error model, are defined relative to the circuit's own noiseless execution. This self-reference makes them exactly blind, rather than merely insensitive, in the regime where a pass has to be checked: before noise is added, which is also where the checking must happen if noise channels are not to survive the gates they describe. Across 335 mutants that provably change what a repetition-code memory circuit computes, detector sampling and detector error model comparison detect none. On a rotated surface code memory experiment, deleting every single-qubit gate leaves the detector error model byte-identical, 29 lines and 24 detectors either way, while 21 of 33 measurement outcomes change by up to 0.50. Two negative controls, each equivalent on every one of 200 trials with no oracle raising a false positive, establish that the measurement is not an artefact of the mutation set. The paper gives the measurement-record oracle that closes the gap and is exact for noiseless stabiliser circuits, and reports two negative results: one compiler test suite whose oracle detected every fault that could be constructed against it, and one transformation library that came back clean. All code and data are public.
Quantum software has entered production use, but its reliability remains sensitive. Bugs in compilers, simulators, and algorithm-layer source codes do not lead to program crashes; rather, they emerge as faulty probability distributions, successfully compiled but incorrect circuits, and differences between platforms, which classical testing oracles are incapable of noticing. This paper surveys quantum fuzzing as a response to that gap. This paper is organize in the field along the classical grey-box, white-box, and black-box axis and trace, for each category, the sequence of tools that have advanced the state of the art QuanTest, QSolver, and NovaQ representing the current frontiers respectively. Characterized the defect landscape these tools have collectively exposed, showing that bugs concentrate disproportionately in compilers and transpilation passes, that simulator and algorithm-layer faults form a distinct oracle-evading class, and that roughly 40% of confirmed quantum bugs are quantum-specific in symptom. Across all categories mentioned the same that crash-based detection is structurally inadequate for quantum software, and that the most productive fuzzers replace it with statistical, metamorphic, entanglement-guided, or formal-invariant oracles. We close by identifying the unification of these oracle strategies within a single campaign as the principal open challenge for the field.
Mutation testing evaluates test suite quality by introducing artificial faults (mutants) and checking whether tests detect (kill) them. A central challenge is the equivalent mutant problem: some mutants are syntactically different from the original program but semantically identical to it and therefore cannot be killed by any test. If left unidentified, such mutants waste testing effort and distort mutation scores. In quantum software, mutation testing is increasingly used, but the equivalent mutant problem remains unsolved. A recent study generated more than 700,000 quantum circuit mutants and found that roughly half survived the available tests, making it unclear whether these survivors reflect weak tests or semantic equivalence. We propose Transpiler-Based Equivalence (TBE), a lightweight approach that identifies equivalent quantum mutants by transpiling original and mutated circuits under the same configuration and comparing their resulting OpenQASM code. We evaluate TBE on 348,299 surviving mutants, 92,011 of which are equivalent; TBE identifies 29,536 of them (32.1%) as equivalent while achieving 100% precision and 82% accuracy.
Quantum computing has been rapidly expanding, and many platforms for writing programs that can be compiled and run on quantum hardware (or simulated) are being developed. As with any compiler, transformation correctness is paramount as the machine readable code must maintain program semantics. Quantum programs are challenging to test given a lack of benchmark programs and the difficulty of defining an oracle. MorphQ solves both of these challenges by (1) generating syntactically correct quantum programs and (2) using metamorphic testing to avoid the oracle problem. However, Qiskit, the compiler it was built for, is rapidly evolving. This paper is a reproducibility study of the MorphQ platform, which we call MorphQ++. We have updated the core MorphQ engine to work on a newer version of Qiskit and added new metamorphic transformations. We find that our overall results are similar: we find a portion of the original faults (which were not yet fixed) and the distributions of types are not very different. Our new transformations lead to several new faults, suggesting there is room to expand the core framework. Additionally, we note the lack of power of the metamorphic relations in this context and suggest the need for more sophisticated relations and/or oracle evaluations.