Benchmarking is how the performance of a computing system is determined. Surprisingly, even for classical computers this is not a straightforward process. One must choose the appropriate benchmark and metrics to extract meaningful results. Different benchmarks test the system in different ways, and each individual metric may or may not be of interest. Choosing the appropriate approach is tricky. The situation is even more open ended for quantum computers, where there is a wider range of hardware, fewer established guidelines, and additional complicating factors. Notably, quantum noise significantly impacts performance and is difficult to model accurately. Here, we discuss benchmarking of quantum computers from a computer architecture perspective and provide numerical simulations highlighting challenges that suggest caution.
Benchmarking is how the performance of a computing system is determined. Surprisingly, even for classical computers this is not a straightforward process. One must choose the appropriate benchmark and metrics to extract meaningful results. Different benchmarks test the system in different ways, and each individual metric may or may not be of interest. Choosing the appropriate approach is tricky. The situation is even more open ended for quantum computers, where there is a wider range of hardware, fewer established guidelines, and additional complicating factors. Notably, quantum noise significantly impacts performance and is difficult to model accurately. Here, we discuss benchmarking of quantum computers from a computer architecture perspective and provide numerical simulations highlighting challenges that suggest caution.
We present a framework that utilizes quantum algorithms, an architecture aware quantum noise model and an ideal simulator to benchmark quantum computers. The benchmark metrics highlight the difference between the quantum computer evolution and the simulated noisy and ideal quantum evolutions. We utilize our framework for benchmarking three IBMQ systems. The use of multiple algorithms, including continuous-time ones, as benchmarks stresses the computers in different ways highlighting their behaviour for a diverse set of circuits. The complexity of each quantum circuit affects the efficiency of each quantum computer, with increasing circuit size resulting in more noisy behaviour. Furthermore, the use of both a continuous-time quantum algorithm and the decomposition of its Hamiltonian also allows extracting valuable comparisons regarding the efficiency of the two methods on quantum systems. The results show that our benchmarks provide sufficient and well-rounded information regarding the performance of each quantum computer.
Context: The main obstacle of quantum computing on its way to scalability is the noisy nature of current hardware. Various types of errors lead to incorrect computational results,and measures such as quantum error mitigation are needed to counteract these errors. Many error mitigation techniques require a thorough understanding of the noise and its impact on computations. Therefore, accurate noise models are indispensable to facilitate meaningful quantum computing in the near future. They also enable realistic simulations of quantum devices. Since these devices are currently limited and quantum computing is expensive, researchers typically rely on simulations for testing their algorithms.Objective: In this thesis, we aim to construct a realistic noise model for quantum computing and optimise its model parameters. The model construction should build on the underlying physical processes, and the number of parameters should scale well with the system size. We also address the evaluation and comparison of different noise models by benchmarks using quantum circuits specific to real-world applications. Finally, we aim to develop useful graphical representations of noise models based on the existing quantum circuit model.Method: We present a benchmarking framework for quantum computing noise models and evaluate it based on benchmarking quality attributes from the literature. Moreover, we construct a quantum computing noise model and develop a training procedure for its parameters. We analyse the parameters of the trained noise model and the impact ofdifferent types of errors on computations. We benchmark the noise model with the above approach and compare it to a noise model provided by IBM’s software development kit Qiskit. Finally, we develop an extension of the quantum circuit model to represent noise channels.Result: The benchmarks performed in this thesis show that our noise model predicts noisy hardware behaviour of IBM’s ibmq_manila quantum device equally well as the Qiskit model, if not better, based on quantum circuits commonly used for variational quantum algorithms. Within the trained noise model, readout error has the most detrimental impact on computations. Our benchmarking approach satisfies relevant quality criteria by choosing suitable quantum circuits and objective functions to compare model predictionsto hardware data. Moreover, the quantum circuit model can be extended meaningfully by noise channels.Conclusion: Our benchmarking approach is suitable for evaluating and comparing quantum computing noise models. In future work, volumetric benchmarks should be performed for various noise models and application contexts. The parameter optimisation based on learning with training circuits yields accurate noise models and shows the importance of mitigating measurement error. Different optimisation algorithms and quantum circuits should be explored, and more types of noise should be incorporated into the noise model.
Quantum computers have the potential to provide an advantage over classical computers in a number of areas. Numerous metrics to benchmark the performance of quantum computers, ranging from their individual hardware components to entire applications, have been proposed over the years. Objective comparisons are further hampered in practice as different variations of the same metric are used, and the data disclosed together with a reported metric value is often not sufficient to reproduce the measurements. In this poster, we present our recent article, where we address these challenges by providing a review of metrics and benchmarks for quantum computers [1], including 1) a comprehensive collection of benchmarks allowing holistic comparisons of quantum computers, 2) a consistent format of the definitions across all metrics including a transparent description of the methodology and of the main assumptions and limitations, and 3) a reproducible approach by linking the metrics to open-source software used to evaluate them . We identify five areas where international standardization working groups could be established, namely: i) the identification and agreement on the categories of metrics that comprehensively benchmark device performance; ii) the identification and agreement on a set of well-established metrics that together comprehensively benchmark performance; iii) the identification of metrics specific to hardware platforms, including non-gate-based quantum computers; iv) inter-laboratory comparison studies to develop best practice guides for measurement methodology; and v) agreement on what data and software should be reported together with a metric value to ensure trust, transparency and reproducibility. We provide potential routes to advancing these areas. We expect that this work will accelerate the progress of quantum computing hardware towards quantum advantage. [1] D. Lall et al., A Review and Collection of Metrics and Benchmarks for Quantum Computers: Definitions, Methodologies and Software, arXiv:2502.06717.
Quantum AI is a new way to think about the future of Artificial Intelligence through the merging of two technologies: quantum computing and advanced machine learning. Quantum AI has the potential to provide solutions to computationally difficult problems that traditional systems cannot solve. Quantum AI's continued success will rely on advancements in hybrid quantum-classical solutions, as well as the expansion of quantum hardware; hence, research remains necessary to conduct systematic benchmarks as well as performance analysis in order to determine both how well quantum AI algorithms perform practically, and where their limitations lie. In this chapter, detailed examinations of benchmarks that have been developed specifically for quantum AI are presented alongside evaluative metrics across various key areas of performance including algorithmic efficiency, scalability, noise immunity, and resource use for various types of quantum architectures. The chapter integrates dialogue between theory and practice to demonstrate ways in which performance metrics (i.e., quantum circuit depth, fidelity, convergence behaviour and computational complexity) can effectively compare Quantum AI algorithms to classical algorithms. Particular attention is given to hybrid variational algorithms, quantum-enhanced optimization algorithms, and quantum machine learning. Using benchmark datasets, simulation frameworks and new evaluation methods, there are several challenges in terms of reproducibility, hardware variability and algorithm/hardware co-design that emerge through analysis of the above metrics as well. Aside from focusing on a more broad view of how Performance Evaluation relates to developing more effective Quantum AI Solutions. In addition to viewing Benchmarking as a technical exercise solely, this chapter provides a more holistic view in relation to a human perspective of the situation. This paper outlines how performance evaluation practices can be used as a considerable and well-designed means of bridging the gap between the promise of Quantum Technology at an abstract level, and the ability of Quantum Technology to produce tangible outcomes in the real world by adopting a more comprehensive Governance-oriented approach to Benchmarking Quantum AI Systems. The purpose of this chapter is to assist the growing community of researchers, practitioners, and Strategic Technology Professionals by providing them with a clear and structured framework for Evaluation of Quantum AI Algorithms. There is a significant focus of the chapter on establishing important reference points that organizations can use to determine when an intelligent system has successfully achieved a true and validated quantum advantage. The ultimate goal is to assist in guiding the responsible development and implementation of Quantum AI to enable the Potential Benefits of Quantum AI to be realized, in ways that are meaningful and impactful.
Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Different kinds of quantum computing prototypes and simulators have been developed rapidly, and entry-level cloud-based quantum computing services begun to emerge. Various noise in quantum computer had significant impact on its execution results, which were also focus of benchmark and performance evaluation. Based on IBM Quantum Lab platform, several application algorithms have been executed in both quantum backend and simulator to verify the impact of circuit depth, different quantum noise model and parameters on the fidelity of quantum computing results. The prospect of quantum computing benchmark evaluation was also discussed.