This paper introduces HamilToniQ, an open-source benchmarking toolkit for Quantum Processing Units (QPUs). It addresses the complexities of quantum computations by providing a methodological framework to assess QPU types, topologies, and systems. HamilToniQ facilitates performance evaluations through steps like circuit compilation and quantum error mitigation, with strategies tailored for each stage. The toolkit's H-Score measures QPU fidelity and reliability, offering a comprehensive view of performance. Focused on the Quantum Approximate Optimization Algorithm (QAOA), HamilToniQ enables consistent QPU comparisons, enhancing benchmarking transparency. Validated on various IBM QPUs, the toolkit proves effective and robust, advancing quantum computing with precise benchmarking metrics.
This paper introduces HamilToniQ, an open-source benchmarking toolkit for Quantum Processing Units (QPUs). It addresses the complexities of quantum computations by providing a methodological framework to assess QPU types, topologies, and systems. HamilToniQ facilitates performance evaluations through steps like circuit compilation and quantum error mitigation, with strategies tailored for each stage. The toolkit's H-Score measures QPU fidelity and reliability, offering a comprehensive view of performance. Focused on the Quantum Approximate Optimization Algorithm (QAOA), HamilToniQ enables consistent QPU comparisons, enhancing benchmarking transparency. Validated on various IBM QPUs, the toolkit proves effective and robust, advancing quantum computing with precise benchmarking metrics.
Quantum computing holds the potential to provide speedups in solving complex problems that are currently difficult for classical computers. However, the realization of this potential is hindered by the issue of current hardware reliability, primarily due to noise and architectural imperfections. As quantum computing systems rapidly advance, there exists a need to create a generalizable benchmarking tool that can assess reliability across different hardware platforms. In this paper, we introduce GRADE (Grover-based Reliability Assessment for Device Evaluation), an open-source benchmarking toolkit to evaluate the reliability of quantum hardware using a generalized form of Grover's algorithm. GRADE operates by implementing Grover's algorithm to search through a variety of primitive collections that are customizable by the user, analyzing the probability distribution of the results to assess accuracy and stability. This approach aims to evaluate the hardware performance of Grover's algorithm, which is fundamental in unordered search problems, making it an ideal candidate for benchmarking purposes, as it is one of the few generalizable quantum algorithms that provide a direct speedup. Importantly, GRADE can adapt to a wide range of quantum computing platforms, ensuring it can be applied across different hardware architectures. Additionally, our work provides a scoring function for evaluating the hardware performance for multi-target Grover's implementation. Our work validates this approach across a multitude of simulators and hardware platforms from varying providers, demonstrating its adaptability to differing backend providers.
Quantum computers have the potential to provide an advantage over classical computers in a number of areas. Numerous metrics to benchmark the performance of quantum computers, ranging from their individual hardware components to entire applications, have been proposed over the years. Objective comparisons are further hampered in practice as different variations of the same metric are used, and the data disclosed together with a reported metric value is often not sufficient to reproduce the measurements. In this poster, we present our recent article, where we address these challenges by providing a review of metrics and benchmarks for quantum computers [1], including 1) a comprehensive collection of benchmarks allowing holistic comparisons of quantum computers, 2) a consistent format of the definitions across all metrics including a transparent description of the methodology and of the main assumptions and limitations, and 3) a reproducible approach by linking the metrics to open-source software used to evaluate them . We identify five areas where international standardization working groups could be established, namely: i) the identification and agreement on the categories of metrics that comprehensively benchmark device performance; ii) the identification and agreement on a set of well-established metrics that together comprehensively benchmark performance; iii) the identification of metrics specific to hardware platforms, including non-gate-based quantum computers; iv) inter-laboratory comparison studies to develop best practice guides for measurement methodology; and v) agreement on what data and software should be reported together with a metric value to ensure trust, transparency and reproducibility. We provide potential routes to advancing these areas. We expect that this work will accelerate the progress of quantum computing hardware towards quantum advantage. [1] D. Lall et al., A Review and Collection of Metrics and Benchmarks for Quantum Computers: Definitions, Methodologies and Software, arXiv:2502.06717.
Quantum computers have now surpassed classical simulation limits, yet noise continues to limit their practical utility. As the field shifts from proof-of-principle demonstrations to early deployments, there is no standard method for meaningfully and scalably comparing heterogeneous quantum hardware. Existing benchmarks typically focus on gate-level fidelity or constant-depth circuits, offering limited insight into algorithmic performance at depth. Here we introduce a benchmarking protocol based on the linear ramp quantum approximate optimization algorithm (LR-QAOA), a fixed-parameter, deterministic variant of QAOA. LR-QAOA quantifies a QPU's ability to preserve a coherent signal as circuit depth increases, identifying when performance becomes statistically indistinguishable from random sampling. We apply this protocol to 24 quantum processors from six vendors, testing problems with up to 156 qubits and 10,000 layers across 1D-chains, native layouts, and fully connected topologies. This constitutes the most extensive cross-platform quantum benchmarking effort to date, with circuits reaching a million two-qubit gates. LR-QAOA offers a scalable, unified benchmark across platforms and architectures, making it a tool for tracking performance in quantum computing.
In recent years, neutral atom-based quantum computation has been established as a competing alternative for the realization of fault-tolerant quantum computation. However, as with other quantum technologies, various sources of noise limit their performance. With processors continuing to scale up, new techniques are needed to characterize and compare them in order to track their progress. In this work, we present two systematic benchmarks that evaluate these quantum processors at scale. We use the quantum adiabatic algorithm (QAA) and the quantum approximate optimization algorithm (QAOA) to solve maximal independent set (MIS) instances of random unit-disk graphs. These benchmarks are scalable, relying not on prior knowledge of the system's evolution but on the quality of the MIS solutions obtained. Rather than isolating individual sources of noise, they provide an application-level figure of merit: improvements in a QPU and its implementation of the protocols should be reflected in higher-quality MIS solutions. We benchmark quera_aquila and pasqal_fresnel on problem sizes up to 102 and 85 qubits, respectively. Overall, quera_aquila performs better on QAOA and QAA instances. Finally, we generate MIS instances of up to 1000 qubits, providing scalable benchmarks for evaluating future, larger processors as they become available. The proposed protocols can therefore serve as a common reference for comparisons, allowing future work to assess whether advances in hardware translate into measurable improvements in MIS solution quality.
Existing protocols for benchmarking current quantum coprocessors fail to meet the usual standards for assessing the performance of high-performance-computing platforms. After a synthetic review of these protocols—whether at the gate, circuit, or application level—we introduce a new benchmark, dubbed Atos Q-score, which is application-centric, hardware-agnostic, and scalable to quantum advantage processor sizes and beyond. The Q-score measures the maximum number of qubits that can be usedeffectivelyto solve the MaxCut combinatorial optimization problem with the quantum approximate optimization algorithm. We give a robust definition of the notion of effective performance by introducing an improved approximation ratio based on the scaling of random and optimal algorithms. We illustrate the behavior of Q-score using perfect and noisy simulations of quantum processors. Finally, we provide an open-source implementation of Q-score that makes it easy to compute the Q-score of any quantum hardware.
We present Benchpress, a benchmarking suite for evaluating the performance and range of functionality of multiple quantum computing software development kits. This suite consists of a collection of over 1,000 tests measuring key performance metrics for a wide variety of operations on quantum circuits composed of up to 930 qubits and O(106) two-qubit gates, as well as an execution framework for running the tests over multiple quantum software packages in a unified manner. Here we give a detailed overview of the benchmark suite and its methodology and generate representative results over seven different quantum software packages. The flexibility of the Benchpress framework enables benchmarking that not only keeps pace with quantum hardware improvements but also can preemptively gauge the quantum circuit processing costs of future device architectures. In this Resource, the authors present an open-source extensible benchmark tool, Benchpress, for evaluating the performance of mainstream quantum computing software. Benchpress was demonstrated to perform over 1,000 tests with up to 930 qubits to compare the performance of quantum software, providing insight into how to best use current programming stacks.