Defining the right metrics to properly represent the performance of a quantum computer is critical to both users and developers of a computing system. In this white paper, we identify three key attributes for quantum computing performance: quality, speed, and scale. Quality and scale are measured by quantum volume and number of qubits, respectively. We propose a speed benchmark, using an update to the quantum volume experiments that allows the measurement of Circuit Layer Operations Per Second (CLOPS) and identify how both classical and quantum components play a role in improving performance. We prescribe a procedure for measuring CLOPS and use it to characterize the performance of some IBM Quantum systems.
Defining the right metrics to properly represent the performance of a quantum computer is critical to both users and developers of a computing system. In this white paper, we identify three key attributes for quantum computing performance: quality, speed, and scale. Quality and scale are measured by quantum volume and number of qubits, respectively. We propose a speed benchmark, using an update to the quantum volume experiments that allows the measurement of Circuit Layer Operations Per Second (CLOPS) and identify how both classical and quantum components play a role in improving performance. We prescribe a procedure for measuring CLOPS and use it to characterize the performance of some IBM Quantum systems.
As quantum processors scale to hundreds of qubits, execution speed is a critical performance dimension alongside scale and quality. While substantial progress has been made in benchmarking circuit fidelity, existing speed metrics often fail to reflect the sustained, end-to-end throughput experienced by users running utility-scale workloads. This shortfall is especially pronounced for layered, parameterized circuits executed repeatedly within classical-quantum workflows, such as variational algorithms and error-mitigated simulations.
In this work, we formalize CLOPS_h (Circuit Layer Operations Per Second) as a holistic speed benchmark defined over layered, hardware-aware circuits. CLOPS_h measures the sustained rate at which the system executes physical layers, parallel slices of qubit-disjoint two-qubit gates separated by synchronization barriers. Because each such layer is one time slice of an N-qubit circuit, this rate maps directly to the execution rate of layered N-qubit circuits, connecting CLOPS_h to published device capability claims, and to the device-level throughput ceiling we formalize as Max Circuits Per Second (MCPS). CLOPS_h is obtained under layer-fidelity operating conditions, binding the speed measurement to an independently verified quality envelope, and it shares its layer decomposition with scalable layer-fidelity (LF) quality benchmarks, enabling coherent interpretation of speed and quality without conflating the two.
As quantum processors continue to scale in size and complexity, the need for well-defined, reproducible, and technology-agnostic performance metrics becomes increasingly critical. Here we present a suite of scalable quantum computing benchmarks developed as key performance indicators (KPIs) within the EU Quantum Flagship. These proposed benchmarks are designed to assess holistic system performance rather than isolated components, and to remain applicable across both noisy intermediate-scale quantum (NISQ) devices and future fault-tolerant architectures. We introduce four core benchmarks addressing complementary aspects of quantum computing capability: large multi-qubit circuit execution via a Clifford Volume benchmark, scalable multipartite entanglement generation through GHZ-state preparation, a benchmark based on the application of Shor's period-finding subroutine to simple functions, and a protocol quantifying the benefit of quantum error correction using Bell states. Each benchmark is accompanied by clearly specified protocols, reporting standards, and scalable evaluation methods. Together, these KPIs provide a coherent framework for transparent and fair performance assessment across quantum hardware platforms and for tracking progress late-NISQ toward early fault-tolerant quantum computation.
As quantum processors grow, new performance benchmarks are required to capture the full quality of the devices at scale. While quantum volume is an excellent benchmark, it focuses on the highest quality subset of the device and so is unable to indicate the average performance over a large number of connected qubits. Furthermore, it is a discrete pass/fail and so is not reflective of continuous improvements in hardware nor does it provide quantitative direction to large-scale algorithms. For example, there may be value in error mitigated Hamiltonian simulation at scale with devices unable to pass strict quantum volume tests. Here we discuss a scalable benchmark which measures the fidelity of a connecting set of two-qubit gates over N qubits by measuring gate errors using simultaneous direct randomized benchmarking in disjoint layers. Our layer fidelity can be easily related to algorithmic run time, via γ defined in Ref.\cite{berg2022probabilistic} that can be used to estimate the number of circuits required for error mitigation. The protocol is efficient and obtains all the pair rates in the layered structure. Compared to regular (isolated) RB this approach is sensitive to crosstalk. As an example we measure a N=80(100) qubit layer fidelity on a 127 qubit fixed-coupling "Eagle" processor (ibm\_sherbrooke) of 0.26(0.19) and on the 133 qubit tunable-coupling "Heron" processor (ibm\_montecarlo) of 0.61(0.26). This can easily be expressed as a layer size independent quantity, error per layered gate (EPLG), which is here 1.7×10−2(1.7×10−2) for ibm\_sherbrooke and 6.2×10−3(1.2×10−2) for ibm\_montecarlo.
Quantum computing hardware is advancing rapidly toward utility-scale machines that will enable scientific breakthroughs. Many teams are pursuing distinct and difficult-to-compare routes to this goal, using different qubit technologies and logical architectures. Tracking progress toward quantum utility therefore requires rigorous benchmarks that measure computational capability relative to utility-scale challenge problems and enable fair comparison across disparate platforms. Here we demonstrate direct, cross-platform measurement of quantum computational capability using a new benchmark that quantifies the size of the largest computationally relevant quantum circuits that a machine can execute successfully and the speed at which it can execute them. We apply this quantum universal operation performance system (QUOPS) experimentally to leading processors from Quantinuum, Google, and IBM, computing directly on physical qubits. Translating state-of-the-art resource requirements for recognized challenge problems that represent useful quantum computation into effective QUOPS circuit sizes shows that computational capability must grow by 5 orders of magnitude, motivating fault-tolerant approaches. We use the same benchmark to assess the performance of a simple fault-tolerant logical-qubit processor implemented on up to eight [[7,1,3]]-encoded logical qubits using Quantinuum Helios-1, and project the growth of capability across successive generations of fault-tolerant quantum computers to show how QUOPS can track progress toward quantum scientific utility.
Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Quantum computing, taking advantage of its parallel computing, is expected to provide exponential acceleration on some difficult problems. It is one of the important directions for the leapfrog development of computing power in the future. However, due to the limitation of the number of quantum bits, coherence time, fidelity and other factors, the computing power of quantum computers has not been fully utilized at the current stage. In recent years, researchers have proposed a variety of quantum computing performance benchmarks to evaluate and research the performance of quantum computers from the bit, circuit, system, application and other levels. However, the existing performance benchmarks cannot directly evaluate the ability of quantum computers to solve specific problems. Considering the above problems, this paper proposes a VQA-classification-algorithm-oriented performance benchmarks for quantum computing, which takes the classification effect under different data scales as one of the evaluation indicators. This benchmark can comprehensively evaluate the ability of quantum computers to solve data classification problems from the three dimensions of scale, speed, and quality. The benchmark was used to evaluate the superconducting quantum computer and simulator of the IBM and Huawei quantum platform, and its effect was verified experimentally.