Draft preprint. Not submitted to a venue. Quantum compiler benchmarks are used to accept or reject changes to production transpilers, but the compilers they measure are stochastic. At the pinned revision of IBM's Benchpress benchmark suite, its Qiskit gym compiles without passing seed_transpiler, so every gate count it reports is one draw from an unmeasured distribution — while the BQSKit gym in the same repository does seed its compiler. We quantify the consequence as finite-sample decision risk: the probability that a regression verdict computed from k runs per version disagrees with the verdict implied by the long-run mean of the same measurement. On bv_n140 mapped to a heavy-hex lattice, measured over 400 seeds per version across 21 OS processes, the change from Qiskit 1.4.3 to 2.0.0 has a long-run mean of +5.37% (95% CI +4.27% to +6.50%). The suite's own protocol — three unseeded runs per version — reports it as a ≥+10% regression 24.4% of the time (95% CI 19.5–31.1%). A disjoint 200-seed sample across ten fresh processes reproduces this at 22.6%. Twenty runs per version, roughly 40 hours of compute, still leaves 3.7%. A pre-registered replication across 39 circuits is reported in full: the analysis code was committed before the data existed, and the commit ordering is included in this archive as COMMIT_TIMESTAMPS.txt. The pre-registered primary endpoint is 12 of 26 eligible circuits (46.2%, Wilson 95% CI 28.8–64.5%), reported alongside a cluster-robust interval of [17.4%, 81.0%] that reflects algorithm-family clustering. What this archive contains: the paper, the complete analysis and measurement code, and all 41,790 raw per-seed measurements. verify.py re-runs six checks — the toolchain pin, the test suite, the numeric inventory, the replication artifact, a proof of a withdrawn analytical claim, and a recomputation of every quantitative claim in the manuscript against the text — in about two minutes. Four claims from earlier phases of this work were withdrawn on the record after adversarial review, and the withdrawals remain in the repository with the evidence that defeated them. Passing seed_transpiler removes false positives on the circuits tested but was worse on three of four circuits exhibiting false negatives, and is not presented as a general remedy. Changes in v2 (2026-09-10). This version corrects two overstatements carried by the v1 manuscript and adds one new pre-registered result. No measurement changed, and no reported figure moved. Withdrawn. v1 stated that Benchpress sets no transpiler seed anywhere. That is narrowed to the claim the evidence supports: at the pinned revision, the Qiskit gym compiles without passing seed_transpiler — while the sibling BQSKit gym does seed its compiler, and six further gyms were not assessed. v1 also stated that the work identifies the mechanism that governs the effect. The paper now reports what predicts the risk and explicitly declines to call that association a compiler mechanism. Added: a pre-registered cross-machine determinism control. All 216 per-seed gate counts are identical across two physical machines with different CPU vendors (AMD64 AuthenticAMD, 16 cores, Python 3.10.10; Intel64 GenuineIntel, 8 cores, Python 3.10.21), over all three Qiskit versions including the 1.4.3 → 2.0.0 pair the primary study uses. The design was committed before the second machine existed and inherits the frozen six-circuit, twelve-seed selection, so it introduces no selection freedom. Stated scope limit: both machines ran rustworkx 0.18.1, so this cannot separate “the CPU does not matter” from “the graph library was identical, so the CPU never had the opportunity to matter”. The primary 200-seed study itself ran on one machine. Also in this archive. The settled-question ledger now holds 33 entries, each naming the attack, who made it, the command that settled it and the result; it includes three rounds of external adversarial audit of the cross-machine comparator itself, in which fourteen defects were found in the checking instrument and none in the measurements. The cross-machine evidence, its pre-registration, the comparator and its property-based fuzz tests are included. The numeric inventory now records a provenance tier per figure (29 re-derived from raw per-seed files, 12 read from a summary table, 5 not applicable). verify.py now runs six checks rather than five, the sixth recomputing every quantitative claim in the manuscript and comparing it to the text. Two reproducibility defects in the README were fixed: a shallow clone that could not reach the pinned Benchpress revision, and a missing environment without which the primary 1.4.3 → 2.0.0 study could not be reproduced at all.
Draft preprint. Not submitted to a venue. Quantum compiler benchmarks are used to accept or reject changes to production transpilers, but the compilers they measure are stochastic. At the pinned revision of IBM's Benchpress benchmark suite, its Qiskit gym compiles without passing seed_transpiler, so every gate count it reports is one draw from an unmeasured distribution — while the BQSKit gym in the same repository does seed its compiler. We quantify the consequence as finite-sample decision risk: the probability that a regression verdict computed from k runs per version disagrees with the verdict implied by the long-run mean of the same measurement. On bv_n140 mapped to a heavy-hex lattice, measured over 400 seeds per version across 21 OS processes, the change from Qiskit 1.4.3 to 2.0.0 has a long-run mean of +5.37% (95% CI +4.27% to +6.50%). The suite's own protocol — three unseeded runs per version — reports it as a ≥+10% regression 24.4% of the time (95% CI 19.5–31.1%). A disjoint 200-seed sample across ten fresh processes reproduces this at 22.6%. Twenty runs per version, roughly 40 hours of compute, still leaves 3.7%. A pre-registered replication across 39 circuits is reported in full: the analysis code was committed before the data existed, and the commit ordering is included in this archive as COMMIT_TIMESTAMPS.txt. The pre-registered primary endpoint is 12 of 26 eligible circuits (46.2%, Wilson 95% CI 28.8–64.5%), reported alongside a cluster-robust interval of [17.4%, 81.0%] that reflects algorithm-family clustering. What this archive contains: the paper, the complete analysis and measurement code, and all 41,790 raw per-seed measurements. verify.py re-runs six checks — the toolchain pin, the test suite, the numeric inventory, the replication artifact, a proof of a withdrawn analytical claim, and a recomputation of every quantitative claim in the manuscript against the text — in about two minutes. Four claims from earlier phases of this work were withdrawn on the record after adversarial review, and the withdrawals remain in the repository with the evidence that defeated them. Passing seed_transpiler removes false positives on the circuits tested but was worse on three of four circuits exhibiting false negatives, and is not presented as a general remedy. Changes in v2 (2026-09-10). This version corrects two overstatements carried by the v1 manuscript and adds one new pre-registered result. No measurement changed, and no reported figure moved. Withdrawn. v1 stated that Benchpress sets no transpiler seed anywhere. That is narrowed to the claim the evidence supports: at the pinned revision, the Qiskit gym compiles without passing seed_transpiler — while the sibling BQSKit gym does seed its compiler, and six further gyms were not assessed. v1 also stated that the work identifies the mechanism that governs the effect. The paper now reports what predicts the risk and explicitly declines to call that association a compiler mechanism. Added: a pre-registered cross-machine determinism control. All 216 per-seed gate counts are identical across two physical machines with different CPU vendors (AMD64 AuthenticAMD, 16 cores, Python 3.10.10; Intel64 GenuineIntel, 8 cores, Python 3.10.21), over all three Qiskit versions including the 1.4.3 → 2.0.0 pair the primary study uses. The design was committed before the second machine existed and inherits the frozen six-circuit, twelve-seed selection, so it introduces no selection freedom. Stated scope limit: both machines ran rustworkx 0.18.1, so this cannot separate “the CPU does not matter” from “the graph library was identical, so the CPU never had the opportunity to matter”. The primary 200-seed study itself ran on one machine. Also in this archive. The settled-question ledger now holds 33 entries, each naming the attack, who made it, the command that settled it and the result; it includes three rounds of external adversarial audit of the cross-machine comparator itself, in which fourteen defects were found in the checking instrument and none in the measurements. The cross-machine evidence, its pre-registration, the comparator and its property-based fuzz tests are included. The numeric inventory now records a provenance tier per figure (29 re-derived from raw per-seed files, 12 read from a summary table, 5 not applicable). verify.py now runs six checks rather than five, the sixth recomputing every quantitative claim in the manuscript and comparing it to the text. Two reproducibility defects in the README were fixed: a shallow clone that could not reach the pinned Benchpress revision, and a missing environment without which the primary 1.4.3 → 2.0.0 study could not be reproduced at all.
Quantum compilers face the test oracle problem, judged by an output-equivalence oracle: the compiled circuit must compute the same unitary as the original, modulo global phase and a qubit-layout permutation. This oracle, by construction, checks only that semantic map, not the circuit's own layout, permutation, or phase records: a defect there, or in a fixed-seed run's determinism, can pass unseen though the record is public. This empirical software engineering study of quantum transpiler correctness uses repository mining to measure how often this happens in real merged compiler fixes: a systematically identified corpus of Qiskit transpiler bug-fixes, classified by an independently dual-coded, source-validated fault-manifestation taxonomy. Nineteen of 68 fixes (28%, 95% Wilson CI 19-40%) repair faults invisible to this equivalence screen, even one augmented with compilation-validity, circuit-quality, and performance checks, and an extended 104-fix corpus over a wider window holds at the same rate with a tighter interval (29/104, 27.9%, CI 20-37%). A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel (a corrupted layout or permutation record): 10 of 68 fixes (15%, CI 8-25%) beneath the 28% headline. The gap is not Qiskit-specific: it replicates in tket (7/21, 33%), with Cirq smaller but consistent. We detected no systematic differences on five inexpensive PR-level characteristics (19 vs 49, underpowered alone; the extended 29-vs-75 corpus tightens every interval toward zero). This class dominates the invisible set, concentrating at representation-boundary crossings. We release the corpus, codebook, and coding artifacts for quantum software testing research. Here we only measure it.
H. Billah and M.M.H. Sarker contributed equally to this work. Interpreting a fixed quantum circuit's output as evidence of a hardware effect requires ruling out two things usually conflated with it: finite-sample fluctuation, and structure already present in the circuit's own noiseless output. Less commonly checked is a third: whether the reference circuit compared against is the one actually executed, once compiler-level qubit routing is accounted for. We execute a fixed 35-qubit, 243-gate diagnostic circuit on IBM Heron-family hardware (three runs, two backends) and show this third check is essential: the routed circuit submitted to hardware has different noiseless predictions than the logical circuit intended, for several qubits. Against the correct, routed-circuit reference, the aggregate deviation is significant in two of three runs, and four registers (0, 9, 14, 27) show a significant output-probability deviation in every run — though we distinguish a high-confidence subset, {0, 9, 27}, with tightly time-matched calibration, from qubit 14, included only on an exploratory basis given its calibration was retrieved roughly two weeks post-execution. Same-backend replicates correlate more strongly by the Pearson measure, though a rank-based (Spearman) test shows this gap does not survive at the level of register ordering, indicating substantial leverage from a small number of registers rather than a broadly robust pattern; these three executions also share a byte-identical payload and are not independent replications in the strictest sense. The mechanism is only partially resolved: independent single-qubit readout error and raw gate count are ruled out as complete explanations for most of these registers, but the simplest version of the leading hypothesis for the single most-affected register does not reproduce the observed cross-backend pattern even with live, backend-specific timing data. A second, structurally different circuit validated only in simulation calibrates the statistical procedure's behavior under one synthetic noise model; it does not itself establish generality to real hardware, and the specific register set found here should not be expected to generalize beyond this study. Note on scope: this is a methodology and case-study contribution centered on executed-circuit references for fixed-circuit diagnostics, not a characterization of IBM Heron hardware in general or a randomized-benchmarking experiment in the formal sense. Supporting data, the routed OpenQASM circuits, raw measurement counts, and analysis code for this study are available on request from the corresponding author. Affiliation: Sonicium Quantum Lab Ltd., Dhaka, Bangladesh.
Test oracles for quantum transpilers typically judge correctness by comparing compiled output against a reference: a statevector, a sampled distribution, or a unitary compared modulo global phase. A companion empirical study measures how often that choice fails. Roughly 28% of merged Qiskit transpiler bug-fixes (95% Wilson CI 19-40%) repair a fault that corrupts layout metadata, global phase, or run-to-run reproducibility while output stays correct: invisible to a black-box output-equivalence oracle by construction. This paper closes that gap with a fault-class-matched, layout-aware, width-tiered oracle family: a layout/permutation contract checker and a contract-level metamorphic relation (MR-1) for the metadata channel, a global-phase tracker for the phase channel, and a determinism runner for reproducibility. Verified from source on nine real, merged Qiskit transpiler regressions (three per channel), the output-equivalence oracle is blind throughout and the matched mechanism fires on every case. A 675-configuration sweep of the contract/metadata invariant finds no false positive. Synthetic mutant families confirm reliability at scale: 1.00 sensitivity and specificity across 36 mutants apiece for the contract/metadata and global-phase channels, and 1.00 sensitivity (95% CI 0.44-1.00) for reproducibility on the three circuits where the mutation is constructible. The contract checker costs two to six orders of magnitude less than a plain output check, the global-phase tracker is comparably cheap within its exact tier, and only the metamorphic relation carries a bounded cost. Ported natively to pytket/tket, the global-phase mechanism transfers cleanly, an identical 1.00/1.00 result with phases recovered to double-precision accuracy, evidence against a Qiskit-specific artifact.
Note: description expanded on 2026-10-14 for readability. It adds no new predictions or criteria. The frozen pre-registration is the attached PDF, unchanged since publication. Status: the experiment was run on 2026-10-14, after this record was published. Results, including every pre-registered test, will be reported in a separate record linked here. In plain language Theory says a quantum system never completely forgets where it started, even when its dynamics are chaotic. This leftover memory is called a quantum birthmark (Graf et al., Phys. Rev. X, 2026). Theory also says the birthmark should be larger when the system's rules run the same forwards and backwards in time (time-reversal symmetry) than when that symmetry is broken. This record pre-registers a test of that prediction on IBM's ibm_kingston quantum processor. We built two nearly identical sets of 4-qubit circuits. One set has time-reversal symmetry and the other does not. Both use exactly the same gates, so they experience the same hardware noise. We measure how often each circuit returns to its starting state, compared with the 1-in-16 chance level, and predict that the symmetric set shows about 1.67× more memory than the broken set. All predictions, pass/fail criteria, circuits and analysis code were deposited here before any quantum hardware was used. What is tested, and what is not Tested: whether the return probability stays above the ergodic baseline 1/16 at depths 8–23, and whether that excess is larger for the time-reversal-symmetric class by the ratio exact numerics predict for these fixed instances. Not tested: the withdrawn temporal-memory claims of 10.5281/zenodo.20470129. The birthmark is closed-system coherent memory, which measurement and reset erase, so no outcome here bears on those claims. Not tested: the universal random-matrix value 1.889. At this size (N = 16) the circuit family itself gives about 1.66, so a finite-size difference is expected. Not a quantum-advantage claim: a 16-dimensional system is exactly simulable. This is a hardware-validation experiment. Design Hardware: ibm_kingston, physical qubits 108–109–118–129 as an open chain, selected by a no-QPU calibration audit (included as v6_backend_audit.zip). Circuits: a kicked-Ising Floquet period Kx(a/2) → D(h1) → Kx(b) → D(h2) → Kx(a/2), with D(h) = exp(−iΣhiZi)·exp(−iπ/4 ΣZiZj), at exactly 6 CZ per period. COE (symmetric): h1 = h2. TRSB (broken): h1 = h + d, h2 = h − d. The classes differ only in virtual-Z angles. Instances: 16 seeded COE/TRSB pairs (seed 20261013), run at every depth from 8 to 23 (48–138 CZ). Observable: P(t) = |⟨0000|Ft|0000⟩|². The primary statistic is the readout-mitigated excess ratio R = ECOE / ETRSB, where E is the mean of P(t) − 1/16. A noise fidelity common to both classes cancels in R. Controls: 32 deep circuits (depths 60–63) as a decoherence-floor check. 8 state-preparation monitors. 16 readout-calibration circuits. Run: one SamplerV2 job of 568 circuits at 256 shots each, with dynamical decoupling and twirling off and the execution order shuffled. Predictions Quantity COE TRSB R Excess over 1/16, depths 8–23 (noiseless, primary) 0.0896 0.0537 1.667 Same instances, infinite-time limit 0.0952 0.0634 1.502 Circuit family, 10,000 random draws — — 1.661 Random-matrix reference, N = 16 — — 1.889 Under four simulated noise models (nominal, pessimistic, and each with coherent errors), the mitigated R stays between 1.64 and 1.77. The shot-noise standard deviation of R at 256 shots is 0.070–0.125. Pre-registered decision criteria Test Pass condition H1 (primary): symmetry dependence 99% CI lower bound of R > 1.0 H2 (primary): quantitative agreement 99% CI of R intersects 1.54–1.87 and is at most 1.0 wide S1: birthmark above decohered floor Window excess minus depth 60–63 excess, 99% CI lower bound > 0, for both classes S2: matched fidelity between classes 0.85 ≤ FCOE/FTRSB ≤ 1.15 S3: instance-level agreement Pearson r ≥ 0.80 against exact predictions S4 (flag only): state prep and drift Monitors ≥ 0.95, drift ≤ 0.045 Overall verdict: CONFIRMED if H1, H2 and S2 all pass; PARTIAL if H1 passes but H2 or S2 fails; NOT CONFIRMED if H1 fails. Every test is reported whatever the outcome. Uncertainty comes from a 10,000-sample parametric multinomial bootstrap over every circuit's counts, with instances held fixed. Amendment 1 (v1.1.0, made before any data) Amendment 1 changed two things relative to v1.0.0 (10.5281/zenodo.23355804): Shots: reduced from 1024 to 256 per circuit for QPU budget, cutting estimated execution from about 152 s to about 38 s. H2: changed from "point estimate in 1.54–1.87" to the CI rule above, because at 256 shots the point rule would falsely fail 20–25% of the time under the pessimistic models. A CI-width cap of 1.0 keeps H2 informative. Circuits, instances and predictions are unchanged. The trade-off is that H2 is now a weaker test. Provenance Every reported number is derived by script from raw counts: Source and transpiled circuits are saved as QPY. Before submission, every transpiled circuit is mapped back to 4 qubits and its unitary checked against the exact model to < 10−8. The analysis reads its decision bands only from predictions.json and its data only from the raw results file. Submission is gated on the DOI of this record. Files: The PDF is the authoritative pre-registration. predictions.json holds every predicted number and decision band. birthmark_prereg_v1.1.0.zip contains the circuit builder, verification, submission and frozen analysis scripts, plus simulated-data self-tests. v6_backend_audit.zip is the pre-submission calibration audit. Known limitations Small system size (N = 16), so the random-matrix value is not expected. The prediction is specific to depths 8–23 and to these 16 fixed instances. The noise models exclude leakage, TLS fluctuations and crosstalk from outside the chain. T1 relaxation biases P(0000) upward, which pulls R toward 1 (conservative). One backend, one path, one job: no replication. Data: CC BY 4.0. Code: MIT. Principal investigator: Amit Brahmbhatt, Quantum-Clarity LLC.
QECTOR Decoder Benchmark Datasets & Empirical Reports This repository hosts the evolving collection of reproducible benchmark datasets, JSON measurement artifacts, and empirical performance reports for the QECTOR Decoder v3 platform (Rust/Python quantum error-correction decoder). Each versioned deposit contains environment-stamped, content-hashed JSON artifacts together with the corresponding publication-grade report. All numerical results are fully regenerable from the published wheels and the scripts listed in the reproducibility sections. Scope across versions Code-capacity and circuit-level (Stim rotated_memory_z) evaluations Throughput (shots/s, measure-s/shot), logical error rates with 95 % Wilson intervals, syndrome faithfulness (GF(2) exact check), and CUDA-vs-CPU bit identity Decoder families: FastUnionFind, UnionFind, Blossom (exact MWPM), SparseBlossom, Batch/CPUBatch, BPOSD, HybridCascade, AmbiguityCluster, LookupTable, and CUDABatch (unweighted + DEM-weighted) Hardware provenance, licence-tier disclosure (GPU gated behind Enterprise), and SHA-256 inventories of every backing artifact Version history (selected) v0.6.6 (July 2026) – Workbench Benchmark Master Report: 1 858 decode-timing measurements from 105 runs, six code topologies (Heavy Hex, Repetition, Ring, Rotated Surface, Toric, Unrotated Surface), fixed p = 0.05, distances d = 3–19. Focus on raw latency, harmonic-mean throughput and peak memory. v0.6.8 - Empirical edge-hardware benchmarks. v0.7.0 (August 2026) – Full Empirical Decoder Benchmark Report on the published wheel: nine CPU families + CUDA batch path, circuit-level LER parity with PyMatching 2.4.0, exact syndrome faithfulness, DEM-weighted GPU scaling restoration, and complete artifact inventory (Appendix A). All figures and tables in the accompanying reports are regenerated from the JSON files listed with their SHA-256 hashes; no pre-release numbers are carried forward. Related resources PyPI: qector-decoder-v3 Repository: https://github.com/GuillaumeLessard/qector-decoder Website: https://www.qector.store
We present microbench.py, a compact (approx. 200 lines) Python script that automates the collection of key compiler metrics, i.e., gate depth, two-qubit-gate count, wall-clock compilation time, and memory footprint, across multiple open-source quantum circuit transpilers. The suite ships with six didactic circuits (3 to 8 qubits) implementing fundamental quantum algorithms and supports Qiskit, tket, Cirq, and the Qiskit-Braket provider; in this paper we showcase results for Qiskit 0.46 and Braket 1.16. The entire run completes in under three minutes on a laptop, emits a single CSV plus publisheable plot, and reproduces the figure here with one command. We release the code under the MIT licence to serve as a quick-start regression harness for NISQ compiler research.