Test oracles for quantum transpilers typically judge correctness by comparing compiled output against a reference: a statevector, a sampled distribution, or a unitary compared modulo global phase. A companion empirical study measures how often that choice fails. Roughly 28% of merged Qiskit transpiler bug-fixes (95% Wilson CI 19-40%) repair a fault that corrupts layout metadata, global phase, or run-to-run reproducibility while output stays correct: invisible to a black-box output-equivalence oracle by construction. This paper closes that gap with a fault-class-matched, layout-aware, width-tiered oracle family: a layout/permutation contract checker and a contract-level metamorphic relation (MR-1) for the metadata channel, a global-phase tracker for the phase channel, and a determinism runner for reproducibility. Verified from source on nine real, merged Qiskit transpiler regressions (three per channel), the output-equivalence oracle is blind throughout and the matched mechanism fires on every case. A 675-configuration sweep of the contract/metadata invariant finds no false positive. Synthetic mutant families confirm reliability at scale: 1.00 sensitivity and specificity across 36 mutants apiece for the contract/metadata and global-phase channels, and 1.00 sensitivity (95% CI 0.44-1.00) for reproducibility on the three circuits where the mutation is constructible. The contract checker costs two to six orders of magnitude less than a plain output check, the global-phase tracker is comparably cheap within its exact tier, and only the metamorphic relation carries a bounded cost. Ported natively to pytket/tket, the global-phase mechanism transfers cleanly, an identical 1.00/1.00 result with phases recovered to double-precision accuracy, evidence against a Qiskit-specific artifact.
Comments
20 pages, 2 figures. Companion paper to arXiv:2609.13839
Test oracles for quantum transpilers typically judge correctness by comparing compiled output against a reference: a statevector, a sampled distribution, or a unitary compared modulo global phase. A companion empirical study measures how often that choice fails. Roughly 28% of merged Qiskit transpiler bug-fixes (95% Wilson CI 19-40%) repair a fault that corrupts layout metadata, global phase, or run-to-run reproducibility while output stays correct: invisible to a black-box output-equivalence oracle by construction. This paper closes that gap with a fault-class-matched, layout-aware, width-tiered oracle family: a layout/permutation contract checker and a contract-level metamorphic relation (MR-1) for the metadata channel, a global-phase tracker for the phase channel, and a determinism runner for reproducibility. Verified from source on nine real, merged Qiskit transpiler regressions (three per channel), the output-equivalence oracle is blind throughout and the matched mechanism fires on every case. A 675-configuration sweep of the contract/metadata invariant finds no false positive. Synthetic mutant families confirm reliability at scale: 1.00 sensitivity and specificity across 36 mutants apiece for the contract/metadata and global-phase channels, and 1.00 sensitivity (95% CI 0.44-1.00) for reproducibility on the three circuits where the mutation is constructible. The contract checker costs two to six orders of magnitude less than a plain output check, the global-phase tracker is comparably cheap within its exact tier, and only the metamorphic relation carries a bounded cost. Ported natively to pytket/tket, the global-phase mechanism transfers cleanly, an identical 1.00/1.00 result with phases recovered to double-precision accuracy, evidence against a Qiskit-specific artifact.
Quantum compilers face the test oracle problem, judged by an output-equivalence oracle: the compiled circuit must compute the same unitary as the original, modulo global phase and a qubit-layout permutation. This oracle, by construction, checks only that semantic map, not the circuit's own layout, permutation, or phase records: a defect there, or in a fixed-seed run's determinism, can pass unseen though the record is public. This empirical software engineering study of quantum transpiler correctness uses repository mining to measure how often this happens in real merged compiler fixes: a systematically identified corpus of Qiskit transpiler bug-fixes, classified by an independently dual-coded, source-validated fault-manifestation taxonomy. Nineteen of 68 fixes (28%, 95% Wilson CI 19-40%) repair faults invisible to this equivalence screen, even one augmented with compilation-validity, circuit-quality, and performance checks, and an extended 104-fix corpus over a wider window holds at the same rate with a tighter interval (29/104, 27.9%, CI 20-37%). A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel (a corrupted layout or permutation record): 10 of 68 fixes (15%, CI 8-25%) beneath the 28% headline. The gap is not Qiskit-specific: it replicates in tket (7/21, 33%), with Cirq smaller but consistent. We detected no systematic differences on five inexpensive PR-level characteristics (19 vs 49, underpowered alone; the extended 29-vs-75 corpus tightens every interval toward zero). This class dominates the invisible set, concentrating at representation-boundary crossings. We release the corpus, codebook, and coding artifacts for quantum software testing research. Here we only measure it.
Quantum compiler assurance requires a semantic relation that matches the intended use and observations that can expose violations at the delivered interface. This critical integrative survey compares verified transformations, equivalence checking, differential and metamorphic testing, property-based testing, and evidence for assurance across revisions. A source catalogue and focused extraction matrix distinguish formal guarantees, reported empirical findings, analytical deductions, and proposed practice. A recorded coverage update and source-level selection decisions make the review auditable within its declared scope. Four documented failure cases connect conditional applicability, parameter association, termination, and phase conventions to method selection. A worked example involving layout metadata, parameter bindings, and ancillary qubits demonstrates why validating a circuit under a reconstructed interpretation can leave the delivered interface unchecked. An illustrative revision scenario then separates fault detection from change attribution. The resulting guidance specifies applicability, complementary observations, and treatment of inconclusive outcomes. Published bug counts, mutation results, and processing benchmarks support different claims and do not establish a common effectiveness ranking. The proposed regression architecture remains unevaluated as a complete system; its practical value requires controlled comparisons on independently adjudicated compiler-change events.
Quantum software relies on compilers, intermediate representations, and execution frameworks that transform programs before execution. Faults in these components can change the behavior of otherwise correct quantum programs. Yet quantum testing techniques are often evaluated using mutation-generated faults and curated bug benchmarks. We investigate how well the faults represented by these artifacts reflect those encountered in modern quantum software. We audit four quantum mutation-testing tools and three curated bug benchmarks, and collect 410 verified bugs from ten quantum software repositories over two years. We classify faults from all three sources into a common taxonomy of 56 bug families and compare their representation across mutation operators, curated benchmarks, and recent real-world bugs. The mutation tools support 20 bug families, and curated benchmarks add 15 more. Yet only 56 of the 410 real-world bugs (13.7%) belong to families represented by either source; the remaining 354 (86.3%) belong to families absent from both. The largest unrepresented families concern compiler transformations (66 bugs), type handling (53), and input validation (47). Unlike local gate or statement changes, these faults often depend on intermediate representations, compiler pipelines, input types, or dependency versions. Existing mutation operators primarily modify circuits and program constructs, whereas many real-world faults arise in the compilers, representations, and framework interfaces that process them. Addressing this gap requires looking beyond circuit-level faults toward testing semantic preservation across compiler transformations, correctness at representation boundaries, and interactions between software components. Our study provides empirical evidence to guide fault models and benchmarks that capture these mechanisms and preserve the contexts needed to reproduce them.
A compiler pass over a quantum error correction circuit must preserve what the circuit computes. The artefacts a practitioner reaches for when reasoning about such a circuit, detector samples and the detector error model, are defined relative to the circuit's own noiseless execution. This self-reference makes them exactly blind, rather than merely insensitive, in the regime where a pass has to be checked: before noise is added, which is also where the checking must happen if noise channels are not to survive the gates they describe. Across 335 mutants that provably change what a repetition-code memory circuit computes, detector sampling and detector error model comparison detect none. On a rotated surface code memory experiment, deleting every single-qubit gate leaves the detector error model byte-identical, 29 lines and 24 detectors either way, while 21 of 33 measurement outcomes change by up to 0.50. Two negative controls, each equivalent on every one of 200 trials with no oracle raising a false positive, establish that the measurement is not an artefact of the mutation set. The paper gives the measurement-record oracle that closes the gap and is exact for noiseless stabiliser circuits, and reports two negative results: one compiler test suite whose oracle detected every fault that could be constructed against it, and one transformation library that came back clean. All code and data are public.
A quantum error correction decoder is a library other people's results depend on, judged in one dominant way. Sample errors, decode, and count wrong logical observables. We ask what else can be checked there. Our conformance contract needs no oracle. One check asks that a returned correction explain the syndrome in the caller's index space. The other hands a decoder one instance under two presentations differing only in bookkeeping, where two feasible corrections of different weight prove the heavier is not minimum-weight. Verdicts are gated on what each library declares, so a firing contradicts a published guarantee. Nine configurations from five public libraries give three results. Documentation answers 4 of 54 capability questions. Bounded-distance correctness, the property callers most depend on, has a direct declaration yield of 0.0%, though its hypotheses hold in 62.1% of cases. Presentation sensitivity is real but shallow. One solver moved to a 26% heavier correction under a different numbering, which reaches the logical class at most once in twenty thousand shots. Established evaluation misses corruptions that preserve logical parity, while one summation over the caller's weights catches every one we injected. All 639 certificates ship as bundles a standalone verifier re-derives from first principles.
Quantum software has entered production use, but its reliability remains sensitive. Bugs in compilers, simulators, and algorithm-layer source codes do not lead to program crashes; rather, they emerge as faulty probability distributions, successfully compiled but incorrect circuits, and differences between platforms, which classical testing oracles are incapable of noticing. This paper surveys quantum fuzzing as a response to that gap. This paper is organize in the field along the classical grey-box, white-box, and black-box axis and trace, for each category, the sequence of tools that have advanced the state of the art QuanTest, QSolver, and NovaQ representing the current frontiers respectively. Characterized the defect landscape these tools have collectively exposed, showing that bugs concentrate disproportionately in compilers and transpilation passes, that simulator and algorithm-layer faults form a distinct oracle-evading class, and that roughly 40% of confirmed quantum bugs are quantum-specific in symptom. Across all categories mentioned the same that crash-based detection is structurally inadequate for quantum software, and that the most productive fuzzers replace it with statistical, metamorphic, entanglement-guided, or formal-invariant oracles. We close by identifying the unification of these oracle strategies within a single campaign as the principal open challenge for the field.