HilbertBench Study G: A Blinded, Pre-Registered Comparison of Human and Mechanical Diagnosis at the Quantum-Machine-Learning Failure-Detection Boundary
Purpose and background. Variational quantum machine-learning runs fail silently: barren plateaus, shot starvation, and hardware noise all present as a loss that stops improving. The HilbertBench instrument records circuit executions non-intrusively into sealed traces and attributes such failures offline. Our first registered study (OSF 10.17605/OSF.IO/2PWHU) validated the human-plus-instrument workflow at ceiling (72/72 blinded diagnoses) on a corpus in which each run expressed one dominant failure mechanism by construction. A ceiling result on a corpus designed to be diagnosable, however, cannot locate the instrument's limits. This study measures those limits and asks a sharper question: as a planted failure mechanism is weakened from strongly expressed to physically absent, where does detection fail, and does a human diagnostician reading the full trace evidence outperform a frozen mechanical decision rule precisely in that transition region? Design. An independent diagnostician (Researcher B from the first study, who has never seen any run in this corpus, its composition, or its answer key) diagnoses 63 blinded, simulator-generated runs from their traces alone, assigning each a primary label, an optional secondary label, a confidence, and free-text notes. The runs are produced by the byte-identical generators of the first study and span graded mechanism strengths that bracket each detection boundary, plus two-mechanism mixtures and mechanism-free controls. The class balance, per-run composition, knob levels, seeds, and generation order are withheld from the diagnostician until unsealing; the full run-level assignment is committed by SHA-256 and revealed at unsealing. Because this registration itself specifies the composition and grid, it is embargoed until the diagnosis is complete and unsealed, so that its contents cannot reach the diagnostician before she finishes her single blinded pass; the registration timestamp and the SHA-256 commitments establish that the design was fixed in advance regardless of the embargo. The comparator is a frozen mechanical rule (the naive reading of the analyzer verdicts, with a fixed mechanism precedence) whose outputs on the blinded corpus are hash-committed before the diagnostician receives the corpus, so the comparator cannot react to her results. Design rationale (simulator-only). Locating a detection boundary requires planting exactly one mechanism at an exactly known strength per run. On real NISQ hardware, single mechanisms cannot be planted cleanly -- the circuit depth that makes a landscape barren also collapses fidelity, a co-occurrence we registered and confirmed in the first study -- so hardware would introduce an uncontrolled second mechanism into every run and invalidate the answer key. This follows the standard two-arm structure of diagnostic validation: real-device (clinical) validity was established in the blinded hardware stratum of the first study; the present study is the analytical arm (limit of detection), performed on controlled references by design. The noise arm still uses calibrated device noise models with ISA transpilation, so device physics remains represented while ground truth stays reproducible. Primary outcome. A paired, one-sided exact McNemar test comparing the human's and the rule's primary labels against the planted ground truth, over the discordant runs (H1: the human is correct on more discordant runs than the rule). The paired design is used because in the transition region the knob-defined label and the trace evidence genuinely diverge; both graders face the identical ambiguity on the identical runs, and the pairing cancels it. Secondary outcomes and predictions. (H2) The human recovers the specific rule-failure classes identified previously -- borderline-SNR runs the rule grades "marginal" rather than "dominated," and shallow converged trajectories whose low variance mimics a plateau -- at a higher rate than the rule. (H3) The human's reported confidence is lower inside the transition regions than on strongly expressed anchor runs. We also report human-versus-rule detection-rate curves by planted strength, two-mechanism flag-set recall, and false positives on mechanism-free runs. Analysis and integrity. Diagnosis is a single pass. The scoring program is committed in code before any run exists; no scoring choice is made after data are collected. The diagnostician's filled sheet is hash-committed before the answer key, the corpus composition, and the rule outputs are unsealed. All commitments are auditable: repository commit b35ee2b (protocol and scorer), assignment hash 540dfefccdf1d33312a7ab3711c2486040ace46c86c2ef312fb1b1e03f35b08a, answer-key hash e7dbf167da9a7dc33219e7721a697c28b2abefa6a76a85c5bf3725417baa0139, and rule-outputs hash 6da0c442e2ca143916921f162a6a3e01f90e0fca46964bbc22dd9e5d70a12c49. Status at registration. The corpus has been generated and blinded and the comparator outputs committed; the diagnostician has not yet received the corpus, and no outcome data exist.
Purpose and background. Variational quantum machine-learning runs fail silently: barren plateaus, shot starvation, and hardware noise all present as a loss that stops improving. The HilbertBench instrument records circuit executions non-intrusively into sealed traces and attributes such failures offline. Our first registered study (OSF 10.17605/OSF.IO/2PWHU) validated the human-plus-instrument workflow at ceiling (72/72 blinded diagnoses) on a corpus in which each run expressed one dominant failure mechanism by construction. A ceiling result on a corpus designed to be diagnosable, however, cannot locate the instrument's limits. This study measures those limits and asks a sharper question: as a planted failure mechanism is weakened from strongly expressed to physically absent, where does detection fail, and does a human diagnostician reading the full trace evidence outperform a frozen mechanical decision rule precisely in that transition region? Design. An independent diagnostician (Researcher B from the first study, who has never seen any run in this corpus, its composition, or its answer key) diagnoses 63 blinded, simulator-generated runs from their traces alone, assigning each a primary label, an optional secondary label, a confidence, and free-text notes. The runs are produced by the byte-identical generators of the first study and span graded mechanism strengths that bracket each detection boundary, plus two-mechanism mixtures and mechanism-free controls. The class balance, per-run composition, knob levels, seeds, and generation order are withheld from the diagnostician until unsealing; the full run-level assignment is committed by SHA-256 and revealed at unsealing. Because this registration itself specifies the composition and grid, it is embargoed until the diagnosis is complete and unsealed, so that its contents cannot reach the diagnostician before she finishes her single blinded pass; the registration timestamp and the SHA-256 commitments establish that the design was fixed in advance regardless of the embargo. The comparator is a frozen mechanical rule (the naive reading of the analyzer verdicts, with a fixed mechanism precedence) whose outputs on the blinded corpus are hash-committed before the diagnostician receives the corpus, so the comparator cannot react to her results. Design rationale (simulator-only). Locating a detection boundary requires planting exactly one mechanism at an exactly known strength per run. On real NISQ hardware, single mechanisms cannot be planted cleanly -- the circuit depth that makes a landscape barren also collapses fidelity, a co-occurrence we registered and confirmed in the first study -- so hardware would introduce an uncontrolled second mechanism into every run and invalidate the answer key. This follows the standard two-arm structure of diagnostic validation: real-device (clinical) validity was established in the blinded hardware stratum of the first study; the present study is the analytical arm (limit of detection), performed on controlled references by design. The noise arm still uses calibrated device noise models with ISA transpilation, so device physics remains represented while ground truth stays reproducible. Primary outcome. A paired, one-sided exact McNemar test comparing the human's and the rule's primary labels against the planted ground truth, over the discordant runs (H1: the human is correct on more discordant runs than the rule). The paired design is used because in the transition region the knob-defined label and the trace evidence genuinely diverge; both graders face the identical ambiguity on the identical runs, and the pairing cancels it. Secondary outcomes and predictions. (H2) The human recovers the specific rule-failure classes identified previously -- borderline-SNR runs the rule grades "marginal" rather than "dominated," and shallow converged trajectories whose low variance mimics a plateau -- at a higher rate than the rule. (H3) The human's reported confidence is lower inside the transition regions than on strongly expressed anchor runs. We also report human-versus-rule detection-rate curves by planted strength, two-mechanism flag-set recall, and false positives on mechanism-free runs. Analysis and integrity. Diagnosis is a single pass. The scoring program is committed in code before any run exists; no scoring choice is made after data are collected. The diagnostician's filled sheet is hash-committed before the answer key, the corpus composition, and the rule outputs are unsealed. All commitments are auditable: repository commit b35ee2b (protocol and scorer), assignment hash 540dfefccdf1d33312a7ab3711c2486040ace46c86c2ef312fb1b1e03f35b08a, answer-key hash e7dbf167da9a7dc33219e7721a697c28b2abefa6a76a85c5bf3725417baa0139, and rule-outputs hash 6da0c442e2ca143916921f162a6a3e01f90e0fca46964bbc22dd9e5d70a12c49. Status at registration. The corpus has been generated and blinded and the comparator outputs committed; the diagnostician has not yet received the corpus, and no outcome data exist.
This project pre-registers and reports a blinded, pre-registered validation of HilbertBench, a non-intrusive diagnostic framework for quantum machine learning (QML). HilbertBench records quantum-circuit executions into append-only, cryptographically sealed traces and computes mechanism-level diagnostics — barren plateaus, shot noise, expressibility, optimization convergence, circuit structure, and hardware noise — offline, without perturbing the experiment under test. The instrument is open-source and frozen at version 1.0.0 (Zenodo DOI: 10.5281/zenodo.21009245). Purpose. We test a single question: can an analyst with no knowledge of how a QML run was produced correctly identify why it failed — barren plateau, shot starvation, noise domination, or none — using only the recorded trace and the tool's built-in analyzers? To our knowledge this is the first blinded, pre-registered validation of a QML diagnostic, modeled on the methodology of clinical trials. Design. Researcher A generates a labeled corpus of runs with planted failure modes (each label known by construction), publishes a SHA-256 cryptographic commitment of the answer key, and — only after this pre-registration is timestamped — hands the blinded runs to an independent diagnostician, Researcher B, who has not seen the corpus design. The corpus has two strata: a simulated set with clean, isolated ground truth (the primary analysis) and a real-hardware subset on a single IBM device (real-device credibility), diagnosed with primary plus optional secondary labels. Analyzer thresholds are frozen at library defaults on a disjoint development corpus before any test data is generated. Expected outcomes. We report diagnosis accuracy against the 25% four-class chance baseline with 95% Wilson confidence intervals, per-class precision/recall, and a full confusion matrix — whatever the result. We additionally pre-register a hardware prediction: on real devices, barren plateaus and noise domination are physically entangled (the circuit depth that induces an intrinsic plateau also collapses fidelity), which the confusion matrix is expected to reveal. Supporting analyses recover the McClean (2018) barren-plateau scaling exponent and validate calibration-based fidelity prediction against real hardware. All data, code, seeds, and the answer key are archived for full reproducibility.
This preregistration describes a confirmatory study of an upstream conditioning architecture in a quantum error-correction workflow. The project’s purpose is to test whether an intervention applied prior to the primary decoder produces a real and reproducible decoder-relevant effect, remains compatible with more than one downstream decoder family, generates a measurable exposed-observable statistical footprint, and improves or preserves performance across a preregistered regime rather than only at a single operating point. The study is intentionally limited to architecture-level evaluation, with the public claim set confined to exposed observables, frozen transformations, preregistered statistical tests, explicit negative controls, and bounded interpretation rules. The confirmatory design is organized around five preregistered tests: category existence, decoder compatibility, structural footprint, observable statistical signature, and regime robustness. Across these tests, the central comparison is between a baseline control pipeline and a conditioned pipeline in which the upstream intervention is applied while all non-conditioning settings are held fixed. The study uses frozen observables and frozen execution rules, including matched seeds, shot counts, code distances, physical error rates, geometry, and baseline decoder settings, so that any supported effect can be interpreted as evidence about the architecture under test rather than as a byproduct of post hoc tuning. To preserve a clean-room boundary, the registration restricts primary analysis to exposed, decoder-facing quantities only. These include logical failure, an exposed defect-count summary, and an exposed decoder-difficulty proxy, along with a locked scalar error-load construction derived from control-only standardization. The study does not rely on decoder-internal state variables, hidden confidence traces, or implementation-specific internal diagnostics in its primary confirmatory analyses. This structure is intended to let readers evaluate whether the architecture leaves a reproducible footprint in decoder-facing behavior without disclosing reconstruction-level mechanism details. Expected outcomes are defined in confirmatory rather than speculative terms. If the preregistered hypotheses are supported, the study is expected to provide evidence that an upstream conditioning intervention exists as a real architectural category in QEC, acts before the primary decoder, is compatible with downstream decoding, changes decoder-relevant exposed structure, and produces a reproducible observable statistical footprint in logical-failure behavior. If support is absent, mixed, or null, those outcomes will still be recorded under the frozen rules without rescue through post hoc metric changes, reweighting, or reinterpretation. In either case, the expected result of the registration is a bounded, auditable public record of whether this architecture class shows reproducible category-level behavior under a preregistered confirmatory design. The preregistration is not intended to establish a unique mechanism identity, identify a privileged implementation, disclose implementation-specific reconstruction details, or support claims about exact targeting logic, suppression laws, subset-sizing logic, telemetry-governed control, or broader cross-domain theory narratives. Public-facing interpretation is therefore intentionally limited to category validation and exposed-observable behavior. This is meant to support rigorous public evaluation while preserving strict clean-room limits around protected implementation detail.
Empirical scaling laws for the Quantum Approximate Optimization Algorithm (QAOA) on near-term hardware are often motivated by noise-compounding arguments or circuit-structure heuristics. Such models are rarely evaluated under pre-specified failure criteria that prevent post-hoc rescue. This study reports an audit-disciplined attempt to construct and test a predictive scaling law for QAOA-based minimal cut set recovery in fault tree models. The workflow used hash-locked artifacts, pre-specified kill switches, one-time holdout evaluation, and explicit no-rerun and no-rescue rules at every phase boundary. The initial F²-grounded compounding specification failed its pre-specified identifiability gate (KS3). Under an anti-leakage filter applied to 108 candidate rescue terms, zero candidates recovered a passing α₁ slope while also adding explanatory power above the structural baseline. A pre-specified structure-dominated fallback model was then defined and fit. The fallback reproduced its locked training prediction pathway to machine precision, with maximum absolute prediction difference of 3.55 × 10⁻¹⁵. It then failed the locked 24-row transport holdout. Holdout RMSE was 5.2214 against a locked threshold of 0.3770, and all 24 residuals were positive, giving a two-sided sign-test p-value of 1.19 × 10⁻⁷. The mean log residual of 5.198 corresponds to approximately 180-fold underprediction on the linear scale. Post-failure diagnostics indicated lower-n domain shift with standardized mean difference of −2.27 and four holdout rows with G_2q below the training range. By design, these diagnostics were explanatory only and did not authorize rescue, refit, reweighting, row removal, or threshold revision. The results do not support a deployable scaling law on the present corpus. Instead, they demonstrate that pre-specified audit controls can convert an attempted positive scaling-law claim into a reproducible falsification result with clear claim boundaries. The audit framework, the locked failure chain, and the allowed-and-forbidden claim boundary together constitute the paper's contribution. The work is offered as a methodological reference for future QAOA-on-hardware scaling studies.
This record contains a manuscript and supporting reproducibility materials for “Ledger-Valid Recovery Diagnostics for Quantum Computing: A Prespecified Sequential Audit for Claim-Limited Recovery Validation.” The paper proposes a methodological validation framework for quantum-computing recovery claims. Its central claim is that endpoint recovery success is necessary but not sufficient for a strong recovery claim when calibration failure, false-carrier acceptance, postselection artifacts, leakage, sentinel blindness, or hidden discard can carry failure modes out of view. The manuscript introduces a prespecified S0–S4 ledger audit that converts familiar diagnostic ingredients—label calibration, candidate recovery, false-carrier challenge tests, postselection/leakage accounting, and sentinel observability—into an ordered, claim-limiting taxonomy. A supportive recovery claim is permitted only when all required gates pass; otherwise, the first failed stage determines a lower claim ceiling and an interpretable failure class. The evaluation is based on synthetic endpoint-level validation suites rather than circuit-level simulation or hardware data. These suites test the behavior of the decision architecture under clean support, degraded observability, false-carrier artifacts, postselection/leakage artifacts, sentinel-blind cases, correlated artifacts, and adversarial overlap. The results show a strong specificity/sensitivity tradeoff: the full ledger protocol sharply reduces synthetic non-support supportive-call rates relative to endpoint-only reporting, but with substantial loss of mechanism-positive support sensitivity. The paper explicitly treats this as a conservative claim-validity architecture, not as a new quantum error-correcting code, new physical observable, hardware speedup, or proof of improved device performance. The supporting materials document the final synthetic validation protocol, endpoint definitions, decision-bound implementation, direction-aware threshold sensitivity analysis, latent-threshold eligibility analysis, mechanism-negative/latent-eligible boundary cases, and figure/table traceability. The archive is intended to support review of the manuscript’s aggregate numerical claims and methodological provenance. Hardware implementation, independent simulator validation, and full raw-event-ledger reproduction are identified as future extensions. Suggested keywords: quantum computing; quantum error correction; validation methodology; recovery diagnostics; leakage; postselection; benchmarking; synthetic validation; decision architecture; reproducibility.
Quantum compilers face the test oracle problem, judged by an output-equivalence oracle: the compiled circuit must compute the same unitary as the original, modulo global phase and a qubit-layout permutation. This oracle, by construction, checks only that semantic map, not the circuit's own layout, permutation, or phase records: a defect there, or in a fixed-seed run's determinism, can pass unseen though the record is public. This empirical software engineering study of quantum transpiler correctness uses repository mining to measure how often this happens in real merged compiler fixes: a systematically identified corpus of Qiskit transpiler bug-fixes, classified by an independently dual-coded, source-validated fault-manifestation taxonomy. Nineteen of 68 fixes (28%, 95% Wilson CI 19-40%) repair faults invisible to this equivalence screen, even one augmented with compilation-validity, circuit-quality, and performance checks, and an extended 104-fix corpus over a wider window holds at the same rate with a tighter interval (29/104, 27.9%, CI 20-37%). A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel (a corrupted layout or permutation record): 10 of 68 fixes (15%, CI 8-25%) beneath the 28% headline. The gap is not Qiskit-specific: it replicates in tket (7/21, 33%), with Cirq smaller but consistent. We detected no systematic differences on five inexpensive PR-level characteristics (19 vs 49, underpowered alone; the extended 29-vs-75 corpus tightens every interval toward zero). This class dominates the invisible set, concentrating at representation-boundary crossings. We release the corpus, codebook, and coding artifacts for quantum software testing research. Here we only measure it.
This manuscript presents a methods-forward, operational protocol for evaluating whether any structured dependence remains in quantum-computing measurement records after explicit modeling of standard quantum and device-level mechanisms. Classical definiteness is defined at the record layer, and pre-record states are treated as predictive descriptions rather than as classical outcome facts. Within this operational stance, the protocol converts questions about outcome status and record formation into a falsifiable test program grounded in preregistered analysis rules and standard measurement telemetry. A locked null-model family is specified in advance, incorporating standard quantum dynamics together with documented device-level mechanisms including drift, correlated and non-Markovian noise, crosstalk proxies, and leakage indicators. The framework provides operational definitions, a conservative postulate set, preregistered endpoints, residual-adjudication criteria, negative controls, a fixed windowing discipline, and a bound-first reporting standard designed to yield informative constraints under null results. Any candidate contribution beyond the declared null family must appear as structured, reproducible residual dependence that survives the locked null family and the full residual-filter sequence. Version 4.4 refines the protocol specification, clarifies evidentiary boundaries, and strengthens the integration of the synthetic demonstration. The manuscript includes a fully specified classical simulation that executes the decision pipeline under controlled conditions. The synthetic study illustrates how the protocol classifies a bounded-null case, a measured-artifact case, a partial-artifact/unresolved-covariate case, an injected structured-residual case, and a failed-topology case. This demonstration establishes computational executability of the protocol and clarifies the behavior of the decision tree, but it does not constitute empirical evidence for any physical mechanism. Because the protocol relies only on standard measurement telemetry and preregistered analysis rules, it is directly implementable across contemporary quantum-computing platforms and does not depend on any single dataset, backend, or hardware realization. The manuscript is a specification paper: it defines a reproducible evidentiary framework for record-level claims, while large-scale empirical implementation, hardware benchmarking, and cross-platform replication remain subsequent studies.