As quantum computers grow in size and scope, a question of great importance is how best to benchmark performance. Here we define a set of characteristics that any benchmark should follow -- randomized, well-defined, holistic, device independent -- and make a distinction between benchmarks and diagnostics. We use Quantum Volume (QV) [1] as an example case for clear rules in benchmarking, illustrating the implications for using different success statistics, as in Ref. [2]. We discuss the issue of benchmark optimizations, detail when those optimizations are appropriate, and how they should be reported. Reporting the use of quantum error mitigation techniques is especially critical for interpreting benchmarking results, as their ability to yield highly accurate observables comes with exponential overhead, which is often omitted in performance evaluations. Finally, we use application-oriented and mirror benchmarking techniques to demonstrate some of the highlighted optimization principles, and introduce a scalable mirror quantum volume benchmark. We elucidate the importance of simple optimizations for improving benchmarking results, and note that such omissions can make a critical difference in comparisons. For example, when running mirror randomized benchmarking, we observe a reduction in error per qubit from 2% to 1% on a 26-qubit circuit with the inclusion of dynamic decoupling.
As quantum computers grow in size and scope, a question of great importance is how best to benchmark performance. Here we define a set of characteristics that any benchmark should follow -- randomized, well-defined, holistic, device independent -- and make a distinction between benchmarks and diagnostics. We use Quantum Volume (QV) [1] as an example case for clear rules in benchmarking, illustrating the implications for using different success statistics, as in Ref. [2]. We discuss the issue of benchmark optimizations, detail when those optimizations are appropriate, and how they should be reported. Reporting the use of quantum error mitigation techniques is especially critical for interpreting benchmarking results, as their ability to yield highly accurate observables comes with exponential overhead, which is often omitted in performance evaluations. Finally, we use application-oriented and mirror benchmarking techniques to demonstrate some of the highlighted optimization principles, and introduce a scalable mirror quantum volume benchmark. We elucidate the importance of simple optimizations for improving benchmarking results, and note that such omissions can make a critical difference in comparisons. For example, when running mirror randomized benchmarking, we observe a reduction in error per qubit from 2% to 1% on a 26-qubit circuit with the inclusion of dynamic decoupling.
As quantum computers grow in size and scope, a question of great importance is how best to benchmark performance. Here we define a set of characteristics that any bench-mark should follow-randomized, well-defined, holistic, device independent-and make a distinction between benchmarks and diagnostics. We use Quantum Volume (QV) [1] as an example case for clear rules in benchmarking, illustrating the implications for using different success statistics, as in Ref. [2]. We discuss the is-sue of benchmark optimizations, detail when those optimizations are appropriate, and how they should be reported. Reporting the use of quantum error mitigation techniques is especially critical for interpreting benchmarking results, as their ability to yield highly accurate observables comes with exponential overhead, which is often omitted in performance evaluations. Finally, we use application-oriented and mirror benchmarking techniques to demonstrate some of the highlighted optimization principles, and introduce a scalable mirror quantum volume benchmark. We elucidate the importance of simple optimizations for improving benchmarking results, and note that such omissions can make a critical difference in comparisons. For example, when running mirror randomized benchmarking, we observe a reduction in error per qubit from 2 % to 1 % on a 26-qubit circuit with the inclusion of dynamical decoupling.
As quantum processors continue to scale in size and complexity, the need for well-defined, reproducible, and technology-agnostic performance metrics becomes increasingly critical. Here we present a suite of scalable quantum computing benchmarks developed as key performance indicators (KPIs) within the EU Quantum Flagship. These proposed benchmarks are designed to assess holistic system performance rather than isolated components, and to remain applicable across both noisy intermediate-scale quantum (NISQ) devices and future fault-tolerant architectures. We introduce four core benchmarks addressing complementary aspects of quantum computing capability: large multi-qubit circuit execution via a Clifford Volume benchmark, scalable multipartite entanglement generation through GHZ-state preparation, a benchmark based on the application of Shor's period-finding subroutine to simple functions, and a protocol quantifying the benefit of quantum error correction using Bell states. Each benchmark is accompanied by clearly specified protocols, reporting standards, and scalable evaluation methods. Together, these KPIs provide a coherent framework for transparent and fair performance assessment across quantum hardware platforms and for tracking progress late-NISQ toward early fault-tolerant quantum computation.
As quantum processors grow, new performance benchmarks are required to capture the full quality of the devices at scale. While quantum volume is an excellent benchmark, it focuses on the highest quality subset of the device and so is unable to indicate the average performance over a large number of connected qubits. Furthermore, it is a discrete pass/fail and so is not reflective of continuous improvements in hardware nor does it provide quantitative direction to large-scale algorithms. For example, there may be value in error mitigated Hamiltonian simulation at scale with devices unable to pass strict quantum volume tests. Here we discuss a scalable benchmark which measures the fidelity of a connecting set of two-qubit gates over N qubits by measuring gate errors using simultaneous direct randomized benchmarking in disjoint layers. Our layer fidelity can be easily related to algorithmic run time, via γ defined in Ref.\cite{berg2022probabilistic} that can be used to estimate the number of circuits required for error mitigation. The protocol is efficient and obtains all the pair rates in the layered structure. Compared to regular (isolated) RB this approach is sensitive to crosstalk. As an example we measure a N=80(100) qubit layer fidelity on a 127 qubit fixed-coupling "Eagle" processor (ibm\_sherbrooke) of 0.26(0.19) and on the 133 qubit tunable-coupling "Heron" processor (ibm\_montecarlo) of 0.61(0.26). This can easily be expressed as a layer size independent quantity, error per layered gate (EPLG), which is here 1.7×10−2(1.7×10−2) for ibm\_sherbrooke and 6.2×10−3(1.2×10−2) for ibm\_montecarlo.
Quantum computing technologies are undergoing rapid development. The different qubit modalities being considered for quantum computing each have their strengths and weaknesses, making it challenging to compare their performance relative to each other and the state-of-the-art in classical high-performance computing. To better understand the utility of a given quantum processor and to assess when and how it will be able to advance the frontiers of computational science, researchers need a robust approach to quantum benchmarking. A variety of approaches have been proposed, many of which characterize the presence of noise in current quantum devices. These efforts include component-level performance metrics, such as randomized benchmarking and gate set tomography; high-level application-dependent metrics; and devicelevel metrics, such as the Quantum Volume. However, it remains unclear how low-level metrics, such as fidelities and decoherence times, and global device metrics, such as Quantum Volume, relate to the computational utility and practical limitations of quantum processors to solve useful problems. In this paper, we describe our Hamiltonian-oriented approach to quantum benchmarking called HamPerf. Where previous application-dependent approaches specify a suite of benchmarking circuits inspired by applications, we place the problem Hamiltonian at the center. Our strategy allows us to probe the computational performance of a quantum processor on standardized and relevant problem sets, agnostic of the algorithms and hardware used to solve them; it also provides fundamental insights into how device characteristics correlate with computational utility.
Quantum computers have the potential to bring about a new age of technology, but in order for them to become a reality, the elephant in the room must be addressed, quantum errors. The complexity of quantum errors grows exponentially as the system size is increased; scalable characterization protocols which work under physically realistic noise are therefore needed to assess and optimize control. \n \nRandomized Benchmarking (RB) is a family of characterization protocols with the requisite qualities. RB protocols are independent of state preparation and measurement (SPAM) errors, and because they extract partial information about the error profiles in the form of figures of merit, they are scalable in the number of qubits. In this thesis, we study these figures of merit under various error models, gate-sets, and sample distributions, to further our understanding of their physical meaning, and to fine-tune the best practice for diagnosing errors in basic gate-sets. \n \nThe first study presented in this thesis looks at the physical meaning of the unitarity under physically realistic gate-dependent noise. Numerically, we demonstrate that under these realistic noise models, the unitarity obtained via Extended RB (XRB) overlaps strongly with the average of the unitarity of the errors over the set of individual operations. \n \nThe second study numerically and analytically proves that the Standard RB (SRB) decay parameter p coincides with the decay parameter of the novel figure of merit gate-set circuit fidelity. We also prove that, in the limit of high-fidelity single-qubit experiments, the disconnect between the average gate fidelity and SRB experimental results is caused by a basis mismatch between the gates and the state-preparation and measurement procedures. \n \nIn the third part of this thesis, we compare the results of two variations of the SRB protocol, one that samples uniformly (SRB) and one that samples non-uniformly (NIST RB) from the Clifford group. We prove that NIST RB produces an exponential fidelity curve that gives different infidelity results than SRB. \n \nThe fourth and final study gives a standard for assessing basic gate-sets. Experimental results are given from the assessment run on super-conducting and ion-trap devices. We also experimentally show how the systematic error bounds can be tightened by accounting for the error's degree of coherence, as well as using weaker randomizing sets.
Rapid development of quantum computing technology has led to a wide variety of sophisticated quantum devices. Benchmarking these systems becomes crucial for understanding their capabilities and paving the way for future advancements. The Quantum Volume (QV) test is one of the most widely used benchmarks for evaluating quantum computer performance due to its architecture independence. However, as the number of qubits in a quantum device grows, the test faces a significant limitation. It requires determining the subspace of the most probable outcomes, a task that is typically performed via classical simulation of the quantum circuit and therefore incurs an exponential computational cost. In this work, we propose modifications to the QV test, by adopting a carefully restricted circuit ensemble generated from a gate set that remains universal for quantum computation, that allows for the direct determination of the heavy-output subspace. Crucially, the modified circuits remain capable of general quantum computation. This approach overcomes the scalability barrier of the Quantum Volume test beyond classical computational limits, while still probing the key features of universal quantum computing.