Benchmarking quantum computers helps to quantify them and bringing the technology to the market. Various application-level metrics exist to benchmark a quantum device at an application level. This paper presents a revised holistic scoring method called the Quantum Application Score (QuAS) incorporating strong points of previous metrics, such as QPack and the Q-score. We discuss how to integrate both and thereby obtain an application-level metric that better quantifies the practical utility of quantum computers. We evaluate the new metric on different hardware platforms such as D- Wave and IBM as well as quantum simulators of Quantum Inspire and Rigetti.
Benchmarking quantum computers helps to quantify them and bringing the technology to the market. Various application-level metrics exist to benchmark a quantum device at an application level. This paper presents a revised holistic scoring method called the Quantum Application Score (QuAS) incorporating strong points of previous metrics, such as QPack and the Q-score. We discuss how to integrate both and thereby obtain an application-level metric that better quantifies the practical utility of quantum computers. We evaluate the new metric on different hardware platforms such as D- Wave and IBM as well as quantum simulators of Quantum Inspire and Rigetti.
This paper presents the benchmark score definitions of QPack, an application-oriented cross-platform benchmarking suite for quantum computers and simulators, which makes use of scalable Quantum Approximate Optimization Algorithm and Variational Quantum Eigensolver applications. Using a varied set of benchmark applications, an insight of how well a quantum computer or its simulator performs on a general NISQ-era application can be quantitatively made. This paper presents what quantum execution data can be collected and transformed into benchmark scores for application-oriented quantum benchmarking. Definitions are given for an overall benchmark score, as well as sub-scores based on runtime, accuracy, scalability and capacity performance. Using these scores, a comparison is made between various quantum computer simulators, running both locally and on vendors' remote cloud services. We also use the QPack benchmark to collect a small set of quantum execution data of the IBMQ Nairobi quantum processor. The goal of the QPack benchmark scores is to give a holistic insight into quantum performance and the ability to make easy and quick comparisons between different quantum computers
Evaluating the performance of quantum devices is an important step towards scaling quantum devices and eventually using them in practice. The great number of available quantum metrics and the different hardware technologies used to develop quantum computers complicate this evaluation. In addition, different computational paradigms implement quantum operations in different ways. A prominent quantum metric is given by the Q-score metric of Atos. This metric was originally introduced as a standalone way to benchmark devices using the Max-Cut problem. In this work, we show that the Q-score defines a framework of quantum metrics, which allows benchmarking using different problems, user settings and solvers. To showcase the applicability of the framework, we showcase a second Q-score in this framework, called the Q-score Max-Clique. This yields, to our knowledge, the first application-level metric capable of natively comparing three different paradigms of quantum computing. This metric is evaluated on these computational quantum paradigms – quantum annealing, gate-based quantum computing, and photonic quantum computing – and the results are compared to those obtained by classical solvers.
With the rise of quantum computing, many quantum devices have been developed and many more devices are being developed as we speak. This begs the question of which device excels at which tasks and how to compare these different quantum devices with one another. The answer is given by quantum metrics, of which many exist today already. Different metrics focus on different aspects of (quantum) devices and choosing the right metric to benchmark one device against another is a difficult choice. In this paper, we aim to give an overview of this zoo of metrics by grouping established metrics in three levels: component level, system level and application level. With this characterization, we also mention what the merits and uses are for each of the different levels. In addition, we evaluate these metrics on the Starmon-5 device of Quantum Inspire through the cloud access, giving the most complete benchmark of a quantum device from an user experience to date.
Quantum computers have the potential to provide an advantage over classical computers in a number of areas. Numerous metrics to benchmark the performance of quantum computers, ranging from their individual hardware components to entire applications, have been proposed over the years. Objective comparisons are further hampered in practice as different variations of the same metric are used, and the data disclosed together with a reported metric value is often not sufficient to reproduce the measurements. In this poster, we present our recent article, where we address these challenges by providing a review of metrics and benchmarks for quantum computers [1], including 1) a comprehensive collection of benchmarks allowing holistic comparisons of quantum computers, 2) a consistent format of the definitions across all metrics including a transparent description of the methodology and of the main assumptions and limitations, and 3) a reproducible approach by linking the metrics to open-source software used to evaluate them . We identify five areas where international standardization working groups could be established, namely: i) the identification and agreement on the categories of metrics that comprehensively benchmark device performance; ii) the identification and agreement on a set of well-established metrics that together comprehensively benchmark performance; iii) the identification of metrics specific to hardware platforms, including non-gate-based quantum computers; iv) inter-laboratory comparison studies to develop best practice guides for measurement methodology; and v) agreement on what data and software should be reported together with a metric value to ensure trust, transparency and reproducibility. We provide potential routes to advancing these areas. We expect that this work will accelerate the progress of quantum computing hardware towards quantum advantage. [1] D. Lall et al., A Review and Collection of Metrics and Benchmarks for Quantum Computers: Definitions, Methodologies and Software, arXiv:2502.06717.
Several benchmarks have been proposed to holistically measure quantum computing performance. While some have focused on the end user’s perspective (e.g., in application-oriented benchmarks), the real industrial value taking into account the physical footprint of the quantum processor are not discussed. Different use-cases come with different requirements for size, weight, power consumption, or data privacy while demanding to surpass certain thresholds of fidelity, speed, problem size, or precision. This paper aims to incorporate these characteristics into a concept coined quantum utility, which demonstrates the effectiveness and practicality of quantum computers for various applications where quantum advantage – defined as either being faster, more accurate, or demanding less energy – is achieved over a classical machine of similar size, weight, and cost. To successively pursue quantum utility, a level-based classification scheme – constituted as application readiness levels (ARLs) – as well as extended classification labels are introduced. These are demonstratively applied to different quantum applications from the fields of quantum chemistry, quantum simulation, quantum machine learning, and data analysis followed by a brief discussion.
Quantum computing technologies are undergoing rapid development. The different qubit modalities being considered for quantum computing each have their strengths and weaknesses, making it challenging to compare their performance relative to each other and the state-of-the-art in classical high-performance computing. To better understand the utility of a given quantum processor and to assess when and how it will be able to advance the frontiers of computational science, researchers need a robust approach to quantum benchmarking. A variety of approaches have been proposed, many of which characterize the presence of noise in current quantum devices. These efforts include component-level performance metrics, such as randomized benchmarking and gate set tomography; high-level application-dependent metrics; and devicelevel metrics, such as the Quantum Volume. However, it remains unclear how low-level metrics, such as fidelities and decoherence times, and global device metrics, such as Quantum Volume, relate to the computational utility and practical limitations of quantum processors to solve useful problems. In this paper, we describe our Hamiltonian-oriented approach to quantum benchmarking called HamPerf. Where previous application-dependent approaches specify a suite of benchmarking circuits inspired by applications, we place the problem Hamiltonian at the center. Our strategy allows us to probe the computational performance of a quantum processor on standardized and relevant problem sets, agnostic of the algorithms and hardware used to solve them; it also provides fundamental insights into how device characteristics correlate with computational utility.