Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Quantum computers have the potential to provide an advantage over classical computers in a number of areas. Numerous metrics to benchmark the performance of quantum computers, ranging from their individual hardware components to entire applications, have been proposed over the years. Objective comparisons are further hampered in practice as different variations of the same metric are used, and the data disclosed together with a reported metric value is often not sufficient to reproduce the measurements. In this poster, we present our recent article, where we address these challenges by providing a review of metrics and benchmarks for quantum computers [1], including 1) a comprehensive collection of benchmarks allowing holistic comparisons of quantum computers, 2) a consistent format of the definitions across all metrics including a transparent description of the methodology and of the main assumptions and limitations, and 3) a reproducible approach by linking the metrics to open-source software used to evaluate them . We identify five areas where international standardization working groups could be established, namely: i) the identification and agreement on the categories of metrics that comprehensively benchmark device performance; ii) the identification and agreement on a set of well-established metrics that together comprehensively benchmark performance; iii) the identification of metrics specific to hardware platforms, including non-gate-based quantum computers; iv) inter-laboratory comparison studies to develop best practice guides for measurement methodology; and v) agreement on what data and software should be reported together with a metric value to ensure trust, transparency and reproducibility. We provide potential routes to advancing these areas. We expect that this work will accelerate the progress of quantum computing hardware towards quantum advantage. [1] D. Lall et al., A Review and Collection of Metrics and Benchmarks for Quantum Computers: Definitions, Methodologies and Software, arXiv:2502.06717.
Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler's efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities.
Quantum AI is a new way to think about the future of Artificial Intelligence through the merging of two technologies: quantum computing and advanced machine learning. Quantum AI has the potential to provide solutions to computationally difficult problems that traditional systems cannot solve. Quantum AI's continued success will rely on advancements in hybrid quantum-classical solutions, as well as the expansion of quantum hardware; hence, research remains necessary to conduct systematic benchmarks as well as performance analysis in order to determine both how well quantum AI algorithms perform practically, and where their limitations lie. In this chapter, detailed examinations of benchmarks that have been developed specifically for quantum AI are presented alongside evaluative metrics across various key areas of performance including algorithmic efficiency, scalability, noise immunity, and resource use for various types of quantum architectures. The chapter integrates dialogue between theory and practice to demonstrate ways in which performance metrics (i.e., quantum circuit depth, fidelity, convergence behaviour and computational complexity) can effectively compare Quantum AI algorithms to classical algorithms. Particular attention is given to hybrid variational algorithms, quantum-enhanced optimization algorithms, and quantum machine learning. Using benchmark datasets, simulation frameworks and new evaluation methods, there are several challenges in terms of reproducibility, hardware variability and algorithm/hardware co-design that emerge through analysis of the above metrics as well. Aside from focusing on a more broad view of how Performance Evaluation relates to developing more effective Quantum AI Solutions. In addition to viewing Benchmarking as a technical exercise solely, this chapter provides a more holistic view in relation to a human perspective of the situation. This paper outlines how performance evaluation practices can be used as a considerable and well-designed means of bridging the gap between the promise of Quantum Technology at an abstract level, and the ability of Quantum Technology to produce tangible outcomes in the real world by adopting a more comprehensive Governance-oriented approach to Benchmarking Quantum AI Systems. The purpose of this chapter is to assist the growing community of researchers, practitioners, and Strategic Technology Professionals by providing them with a clear and structured framework for Evaluation of Quantum AI Algorithms. There is a significant focus of the chapter on establishing important reference points that organizations can use to determine when an intelligent system has successfully achieved a true and validated quantum advantage. The ultimate goal is to assist in guiding the responsible development and implementation of Quantum AI to enable the Potential Benefits of Quantum AI to be realized, in ways that are meaningful and impactful.
The quantum computer has been claimed to show more quantum advantage than the classical computer in solving some specific problems. Many companies and research institutes try to develop quantum computers with different physical implementations. Currently, most people only focus on the number of qubits in a quantum computer and consider it as a standard to evaluate the performance of the quantum computer intuitively. However, it is quite misleading in most times, especially for investors or governments. This is because the quantum computer works in a quite different way than classical computers. Thus, quantum benchmarking is of great importance. Currently, many quantum benchmarks are proposed from different aspects. In this paper, we review the existing performance benchmarking protocols, models, and metrics. We classify the benchmarking techniques into three categories: physical benchmarking, aggregative benchmarking, and application-level benchmarking. We also discuss the future trend for quantum computer's benchmarking and propose setting up the QTOP100.
As quantum computing systems continue to mature, there is an increasing need for benchmarking methodologies that capture performance in terms of meaningful, application-level metrics. In this work, we present a scalable framework for application-level quantum benchmarking that is designed to support internal system evaluation and cross-platform comparison across technology providers. Our framework is guided by a set of core principles, including measurability, simplicity, scalability, and extensibility. We present 13 benchmark families that reflect realistic workloads across multiple domains. This enables the systematic evaluation of the quality of solutions, the total execution time, total used energy, as well as Time-to-Solution. The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities. The framework aims to bridge the gap between low-level performance metrics and real-world value, providing a unified approach to assessing quantum systems. The resulting benchmarks support development and validation and contribute to the foundation of industry-wide benchmarking standards.
This paper presents a comprehensive review of quantum computing and the technological foundations of quantum processors. Quantum computing represents a transformative computational paradigm that leverages the fundamental principles of quantum mechanics—such as superposition, entanglement, and quantum interference—to solve complex problems that are intractable for classical computers. At the core of this paradigm lies the quantum processor, a specialized hardware system designed to initialize, control, and measure quantum bits (qubits) through precise gate operations and coherent interactions. The survey examines the essential theoretical concepts underlying quantum computation and provides a structured comparison of leading quantum processor platforms. Prominent architectures, including superconducting circuits, trapped-ion systems, and photonic implementations, are analyzed in terms of their physical mechanisms, scalability, coherence properties, and engineering challenges. Each platform offers distinct advantages and faces specific technical limitations related to noise, error correction, and fabrication complexity. In addition to reviewing hardware developments, this study highlights key obstacles in advancing quantum processors toward large-scale, fault-tolerant systems. Issues such as qubit stability, error mitigation, thermal management, and system integration are discussed to present a realistic perspective on current progress. Performance indicators including gate fidelity, coherence time, connectivity, and scalability are used to compare architectural approaches. The objective of this paper is to provide a clear and organized synthesis of recent advancements in quantum computing hardware, thereby offering researchers and practitioners a foundational reference to support continued innovation and development in this rapidly evolving field.