arXiv:2302.00639 · IEEE International Conference on Quantum Computing and Engineering (QCE)

Extending the Q-Score to an Application-Level Quantum Metric Framework

Ward van der Schoot, Robert S. Wezeman, Niels M. P. Neumann, Frank Phillipson, Rob Kooij

Published Sep 15, 2024• 5 citations

Abstract

Evaluating the performance of quantum devices is an important step towards scaling quantum devices and eventually using them in practice. The great number of available quantum metrics and the different hardware technologies used to develop quantum computers complicate this evaluation. In addition, different computational paradigms implement quantum operations in different ways. A prominent quantum metric is given by the Q-score metric of Atos. This metric was originally introduced as a standalone way to benchmark devices using the Max-Cut problem. In this work, we show that the Q-score defines a framework of quantum metrics, which allows benchmarking using different problems, user settings and solvers. To showcase the applicability of the framework, we showcase a second Q-score in this framework, called the Q-score Max-Clique. This yields, to our knowledge, the first application-level metric capable of natively comparing three different paradigms of quantum computing. This metric is evaluated on these computational quantum paradigms – quantum annealing, gate-based quantum computing, and photonic quantum computing – and the results are compared to those obtained by classical solvers.

DOI: 10.1109/qce60285.2024.00113

Related papers

QuAS: Quantum Application Score for Benchmarking the Utility of Quantum Computers

Koen J. Mesman, Ward van der Schoot, Matthias Möller +1 more

Benchmarking quantum computers helps to quantify them and bringing the technology to the market. Various application-level metrics exist to benchmark a quantum device at an application level. This paper presents a revised holistic scoring method called the Quantum Application Score (QuAS) incorporating strong points of previous metrics, such as QPack and the Q-score. We discuss how to integrate both and thereby obtain an application-level metric that better quantifies the practical utility of quantum computers. We evaluate the new metric on different hardware platforms such as D- Wave and IBM as well as quantum simulators of Quantum Inspire and Rigetti.

arXiv:2406.03905IEEE International Conference on Quantum Computing and Engineering (QCE)Sep 15, 20245 citations

Benchmarking Quantum Coprocessors in an Application-Centric, Hardware-Agnostic, and Scalable Way

Simon Martiel, Thomas Ayral, Cyril Allouche

Existing protocols for benchmarking current quantum coprocessors fail to meet the usual standards for assessing the performance of high-performance-computing platforms. After a synthetic review of these protocols—whether at the gate, circuit, or application level—we introduce a new benchmark, dubbed Atos Q-score, which is application-centric, hardware-agnostic, and scalable to quantum advantage processor sizes and beyond. The Q-score measures the maximum number of qubits that can be usedeffectivelyto solve the MaxCut combinatorial optimization problem with the quantum approximate optimization algorithm. We give a robust definition of the notion of effective performance by introducing an improved approximation ratio based on the scaling of random and optimal algorithms. We illustrate the behavior of Q-score using perfect and noisy simulations of quantum processors. Finally, we provide an open-source implementation of Q-score that makes it easy to compute the Q-score of any quantum hardware.

arXiv:2102.12973IEEE Transactions on Quantum EngineeringJan 1, 202160 citations

Optimization Applications as Quantum Performance Benchmarks

Thomas Lubinski, Carleton Coffrin, Catherine C. McGeoch +4 more

Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. The Quantum Approximate Optimization Algorithm and Quantum Annealing can potentially demonstrate significant run-time performance benefits over current state-of-the-art solutions. Inspired by existing methods to characterize classical optimization algorithms, we analyze the solution quality obtained by solving Max-cut problems using gate-model quantum devices and a quantum annealing device. This is used to guide the development of an advanced benchmarking framework for quantum computers designed to evaluate the trade-off between run-time execution performance and the solution quality for iterative hybrid quantum-classical applications. The framework generates performance profiles through compelling visualizations that show performance progression as a function of time for various problem sizes and illustrates algorithm limitations uncovered by the benchmarking approach. As an illustration, we explore the factors that influence quantum computing system throughput, using results obtained through execution on various quantum simulators and quantum hardware systems.

arXiv:2302.02278ACM Transactions on Quantum ComputingJul 17, 202462 citations

Evaluating three levels of quantum metrics on quantum-inspire hardware

W.E. van der Schoot, Robert S. Wezeman, Pieter T. Eendebak +2 more

With the rise of quantum computing, many quantum devices have been developed and many more devices are being developed as we speak. This begs the question of which device excels at which tasks and how to compare these different quantum devices with one another. The answer is given by quantum metrics, of which many exist today already. Different metrics focus on different aspects of (quantum) devices and choosing the right metric to benchmark one device against another is a difficult choice. In this paper, we aim to give an overview of this zoo of metrics by grouping established metrics in three levels: component level, system level and application level. With this characterization, we also mention what the merits and uses are for each of the different levels. In addition, we evaluate these metrics on the Starmon-5 device of Quantum Inspire through the cloud access, giving the most complete benchmark of a quantum device from an user experience to date.

Jan 1, 20230 citations

Benchmarking quantum computers: towards a standard performance evaluation approach

Arturo Acuaviva, David Aguirre, Rubén Peña +1 more

Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).

arXiv:2407.10941Quantum Science and TechnologyFeb 5, 20269 citations

A Use-Case-Oriented Benchmark Framework for Classical, Quantum, Hybrid, and Quantum-Inspired Optimization

Dominik Leib, Tobias Seidel, Sebastian Velten +1 more

Abstract As interest in quantum and related optimization paradigms continues to grow, fairly and systematically comparing their performance across practical problem settings remains challenging. This challenge arises because these approaches often target different problem classes, use different mathematical representations, and are affected differently by modeling choices. As a result, observed performance depends not only on the solver itself, but also on how a problem is formulated, reformulated, and translated into a solver-specific input language. This work presents a publicly available and extensible benchmarking framework that unifies advances in quantum, quantum-hybrid, quantum-inspired, and classical optimization by enabling consistent comparison across use cases, solvers, and models. We address the diversity of solver “languages”—from mixed-integer programming (MIP) formulations for branch-and-bound methods over quadratic unconstrained binary optimization (QUBO) models for quantum annealers to constrained quadratic models (CQM) for hybrid approaches — as well as the influence of modeling choices, such as binary encodings and penalty formulations, on both performance and solution fidelity. By supporting systematic variation across the three dimensions of use case, solver, and model, the framework reveals trade-offs that single-dimension benchmarks overlook, clarifies when native formulations are preferable, and enables rigorous evaluation of practical combinatorial optimization problems. The two case studies, vehicle routing and maximum cut, illustrate this principle: their contrasting outcomes show that the effectiveness of a solver depends strongly on how well the chosen model representation matches the solver backend and the use case.

Jun 15, 20260 citations