Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2202.03028 · IEEE International Conference on Quantum Computing and Engineering (QCE)

QUARK: A Framework for Quantum Computing Application Benchmarking

Jernej Rudi Finžgar, Philipp Ross, Leonhard Hölscher, Johannes Klepsch, André Luckow

Published Sep 1, 2022• 51 citations
arXivHTMLPDF

Abstract

Quantum computing (QC) is anticipated to provide a speedup over classical HPC approaches for specific problems in optimization, simulation, and machine learning. With the advances in quantum computing toward practical applications, the need to analyze and compare different quantum solutions increases. While different low-level benchmarks for QC exist, these benchmarks do not provide sufficient insights into real-world application-level performance. We propose an application-centric benchmark method and the QUantum computing Application benchmaRK (QUARK) framework to foster the investigation and creation of application benchmarks for QC. This paper establishes three significant contributions: (1) it makes a case for application-level benchmarks and provides an in-depth "pen and paper" benchmark formulation of two reference problems: robot path and vehicle option optimization from the industrial domain; (2) it proposes the open-source QUARK framework for designing, implementing, executing, and analyzing benchmarks; (3) it provides multiple reference implementations for these two reference problems based on different known, and where needed, extended, classical and quantum algorithmic approaches and analyzes their performance on different types of infrastructures.

DOI
10.1109/qce53715.2022.00042

Related papers

Measuring what matters: A scalable framework for application-level quantum benchmarking

Willie Aboumrad, Claudio Girotto, Joshua Goings +16 more

As quantum computing systems continue to mature, there is an increasing need for benchmarking methodologies that capture performance in terms of meaningful, application-level metrics. In this work, we present a scalable framework for application-level quantum benchmarking that is designed to support internal system evaluation and cross-platform comparison across technology providers. Our framework is guided by a set of core principles, including measurability, simplicity, scalability, and extensibility. We present 13 benchmark families that reflect realistic workloads across multiple domains. This enables the systematic evaluation of the quality of solutions, the total execution time, total used energy, as well as Time-to-Solution. The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities. The framework aims to bridge the gap between low-level performance metrics and real-world value, providing a unified approach to assessing quantum systems. The resulting benchmarks support development and validation and contribute to the foundation of industry-wide benchmarking standards.

quant-pharXiv:2604.11781•Apr 14, 2026•0 citations

Application-Oriented Performance Benchmarks for Quantum Computing

Thomas Lubinski, Sonika Johri, Paul Varosy +6 more

In this work we introduce an open source suite of quantum application-oriented performance benchmarks that is designed to measure the effectiveness of quantum computing hardware at executing quantum applications. These benchmarks probe a quantum computer's performance on various algorithms and small applications as the problem size is varied, by mapping out the fidelity of the results as a function of circuit width and depth using the framework of volumetric benchmarking. In addition to estimating the fidelity of results generated by quantum execution, the suite is designed to benchmark certain aspects of the execution pipeline in order to provide end-users with a practical measure of both the quality of and the time to solution. Our methodology is constructed to anticipate advances in quantum computing hardware that are likely to emerge in the next five years. This benchmarking suite is designed to be readily accessible to a broad audience of users and provides benchmarks that correspond to many well-known quantum computing algorithms.

arXiv:2110.03137•IEEE Transactions on Quantum Engineering•Jan 1, 2023•158 citations

Optimization Applications as Quantum Performance Benchmarks

Thomas Lubinski, Carleton Coffrin, Catherine C. McGeoch +4 more

Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. The Quantum Approximate Optimization Algorithm and Quantum Annealing can potentially demonstrate significant run-time performance benefits over current state-of-the-art solutions. Inspired by existing methods to characterize classical optimization algorithms, we analyze the solution quality obtained by solving Max-cut problems using gate-model quantum devices and a quantum annealing device. This is used to guide the development of an advanced benchmarking framework for quantum computers designed to evaluate the trade-off between run-time execution performance and the solution quality for iterative hybrid quantum-classical applications. The framework generates performance profiles through compelling visualizations that show performance progression as a function of time for various problem sizes and illustrates algorithm limitations uncovered by the benchmarking approach. As an illustration, we explore the factors that influence quantum computing system throughput, using results obtained through execution on various quantum simulators and quantum hardware systems.

arXiv:2302.02278•ACM Transactions on Quantum Computing•Jul 17, 2024•62 citations

Quantum computing: From the basic concepts to the embedding in an HPC environment for application purposes

Kristel F. L. Michielsen

Quantum computing promises unprecedented possibilities for important computing tasks such as quantum simulations in chemistry and materials science or optimization and machine learning. With this potential, quantum computing is increasingly attracting interest from industry and scientific communities that use high performance computing (HPC) for their applications.Practical application requires the integration of quantum computers into existing HPC infrastructures in the form of quantum-classical hybrid computing models.The Jülich UNified Infrastructure for Quantum computing (JUNIQ), a manufacturer-independent quantum computing user facility established at the Jülich Supercomputing Centre (JSC) aims to address these needs.As an example, we present benchmarking results for the quantum approximate optimization algorithm (QAOA) emulated on a supercomputer and for the D-Wave quantum annealers for the tail assignment problem, a planning problem from aircraft industry.

•Jan 1, 2025•0 citations

Benchmarking and Performance Evaluation of Quantum AI Algorithms

Talabattula Divya Sai Sri Akanksha, Chodapuneedi Sowjanya, Manjula Devarakonda Venkata

Quantum AI is a new way to think about the future of Artificial Intelligence through the merging of two technologies: quantum computing and advanced machine learning. Quantum AI has the potential to provide solutions to computationally difficult problems that traditional systems cannot solve. Quantum AI's continued success will rely on advancements in hybrid quantum-classical solutions, as well as the expansion of quantum hardware; hence, research remains necessary to conduct systematic benchmarks as well as performance analysis in order to determine both how well quantum AI algorithms perform practically, and where their limitations lie. In this chapter, detailed examinations of benchmarks that have been developed specifically for quantum AI are presented alongside evaluative metrics across various key areas of performance including algorithmic efficiency, scalability, noise immunity, and resource use for various types of quantum architectures. The chapter integrates dialogue between theory and practice to demonstrate ways in which performance metrics (i.e., quantum circuit depth, fidelity, convergence behaviour and computational complexity) can effectively compare Quantum AI algorithms to classical algorithms. Particular attention is given to hybrid variational algorithms, quantum-enhanced optimization algorithms, and quantum machine learning. Using benchmark datasets, simulation frameworks and new evaluation methods, there are several challenges in terms of reproducibility, hardware variability and algorithm/hardware co-design that emerge through analysis of the above metrics as well. Aside from focusing on a more broad view of how Performance Evaluation relates to developing more effective Quantum AI Solutions. In addition to viewing Benchmarking as a technical exercise solely, this chapter provides a more holistic view in relation to a human perspective of the situation. This paper outlines how performance evaluation practices can be used as a considerable and well-designed means of bridging the gap between the promise of Quantum Technology at an abstract level, and the ability of Quantum Technology to produce tangible outcomes in the real world by adopting a more comprehensive Governance-oriented approach to Benchmarking Quantum AI Systems. The purpose of this chapter is to assist the growing community of researchers, practitioners, and Strategic Technology Professionals by providing them with a clear and structured framework for Evaluation of Quantum AI Algorithms. There is a significant focus of the chapter on establishing important reference points that organizations can use to determine when an intelligent system has successfully achieved a true and validated quantum advantage. The ultimate goal is to assist in guiding the responsible development and implementation of Quantum AI to enable the Potential Benefits of Quantum AI to be realized, in ways that are meaningful and impactful.

Indian Journal of Computer Science•Oct 1, 2026•0 citations

Application Performance Benchmarks for Quantum Computers

Krzysztof Kurowski, Piotr Rydlichowski, Wojciechowski, Konrad +2 more

Current technological advancements of quantum computers highlight the need for application-driven, practical and well-defined methods of benchmarking their performance. As the existing NISQ device's quality of two-qubit gate errors rate is even around one percent and the number of qubits is still limited to a few or several dozen, naturally, we need to propose rather small algorithms instances taken from key promising application areas, such as quantum chemistry, combinatorial optimisation or machine learning. While many techniques for assessing the performance of logical components, such as gate fidelity and qubit coherence exist, it is still challenging to extrapolate those values onto the performance of different quantum algorithms and subroutines. This work aims to introduce a series of initial quantum application benchmarks together with a methodology of execution for measuring the performance and fidelity of the results. The proposed suite refers to several variational algorithms widely used on available NISQ devices but also includes examples of quantum circuits designed for a fault-tolerant quantum computer.

arXiv:2310.13637•Oct 20, 2023•1 citation
Quantum computing (QC) is anticipated to provide a speedup over classical HPC approaches for specific problems in optimization, simulation, and machine learning. With the advances in quantum computing toward practical applications, the need to analyze and compare different quantum solutions increases. While different low-level benchmarks for QC exist, these benchmarks do not provide sufficient insights into real-world application-level performance. We propose an application-centric benchmark method and the QUantum computing Application benchmaRK (QUARK) framework to foster the investigation and creation of application benchmarks for QC. This paper establishes three significant contributions: (1) it makes a case for application-level benchmarks and provides an in-depth "pen and paper" benchmark formulation of two reference problems: robot path and vehicle option optimization from the industrial domain; (2) it proposes the open-source QUARK framework for designing, implementing, executing, and analyzing benchmarks; (3) it provides multiple reference implementations for these two reference problems based on different known, and where needed, extended, classical and quantum algorithmic approaches and analyzes their performance on different types of infrastructures.
As quantum computing systems continue to mature, there is an increasing need for benchmarking methodologies that capture performance in terms of meaningful, application-level metrics. In this work, we present a scalable framework for application-level quantum benchmarking that is designed to support internal system evaluation and cross-platform comparison across technology providers. Our framework is guided by a set of core principles, including measurability, simplicity, scalability, and extensibility. We present 13 benchmark families that reflect realistic workloads across multiple domains. This enables the systematic evaluation of the quality of solutions, the total execution time, total used energy, as well as Time-to-Solution. The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities. The framework aims to bridge the gap between low-level performance metrics and real-world value, providing a unified approach to assessing quantum systems. The resulting benchmarks support development and validation and contribute to the foundation of industry-wide benchmarking standards.
In this work we introduce an open source suite of quantum application-oriented performance benchmarks that is designed to measure the effectiveness of quantum computing hardware at executing quantum applications. These benchmarks probe a quantum computer's performance on various algorithms and small applications as the problem size is varied, by mapping out the fidelity of the results as a function of circuit width and depth using the framework of volumetric benchmarking. In addition to estimating the fidelity of results generated by quantum execution, the suite is designed to benchmark certain aspects of the execution pipeline in order to provide end-users with a practical measure of both the quality of and the time to solution. Our methodology is constructed to anticipate advances in quantum computing hardware that are likely to emerge in the next five years. This benchmarking suite is designed to be readily accessible to a broad audience of users and provides benchmarks that correspond to many well-known quantum computing algorithms.
Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. The Quantum Approximate Optimization Algorithm and Quantum Annealing can potentially demonstrate significant run-time performance benefits over current state-of-the-art solutions. Inspired by existing methods to characterize classical optimization algorithms, we analyze the solution quality obtained by solving Max-cut problems using gate-model quantum devices and a quantum annealing device. This is used to guide the development of an advanced benchmarking framework for quantum computers designed to evaluate the trade-off between run-time execution performance and the solution quality for iterative hybrid quantum-classical applications. The framework generates performance profiles through compelling visualizations that show performance progression as a function of time for various problem sizes and illustrates algorithm limitations uncovered by the benchmarking approach. As an illustration, we explore the factors that influence quantum computing system throughput, using results obtained through execution on various quantum simulators and quantum hardware systems.
Quantum computing promises unprecedented possibilities for important computing tasks such as quantum simulations in chemistry and materials science or optimization and machine learning. With this potential, quantum computing is increasingly attracting interest from industry and scientific communities that use high performance computing (HPC) for their applications.Practical application requires the integration of quantum computers into existing HPC infrastructures in the form of quantum-classical hybrid computing models.The Jülich UNified Infrastructure for Quantum computing (JUNIQ), a manufacturer-independent quantum computing user facility established at the Jülich Supercomputing Centre (JSC) aims to address these needs.As an example, we present benchmarking results for the quantum approximate optimization algorithm (QAOA) emulated on a supercomputer and for the D-Wave quantum annealers for the tail assignment problem, a planning problem from aircraft industry.
Quantum AI is a new way to think about the future of Artificial Intelligence through the merging of two technologies: quantum computing and advanced machine learning. Quantum AI has the potential to provide solutions to computationally difficult problems that traditional systems cannot solve. Quantum AI's continued success will rely on advancements in hybrid quantum-classical solutions, as well as the expansion of quantum hardware; hence, research remains necessary to conduct systematic benchmarks as well as performance analysis in order to determine both how well quantum AI algorithms perform practically, and where their limitations lie. In this chapter, detailed examinations of benchmarks that have been developed specifically for quantum AI are presented alongside evaluative metrics across various key areas of performance including algorithmic efficiency, scalability, noise immunity, and resource use for various types of quantum architectures. The chapter integrates dialogue between theory and practice to demonstrate ways in which performance metrics (i.e., quantum circuit depth, fidelity, convergence behaviour and computational complexity) can effectively compare Quantum AI algorithms to classical algorithms. Particular attention is given to hybrid variational algorithms, quantum-enhanced optimization algorithms, and quantum machine learning. Using benchmark datasets, simulation frameworks and new evaluation methods, there are several challenges in terms of reproducibility, hardware variability and algorithm/hardware co-design that emerge through analysis of the above metrics as well. Aside from focusing on a more broad view of how Performance Evaluation relates to developing more effective Quantum AI Solutions. In addition to viewing Benchmarking as a technical exercise solely, this chapter provides a more holistic view in relation to a human perspective of the situation. This paper outlines how performance evaluation practices can be used as a considerable and well-designed means of bridging the gap between the promise of Quantum Technology at an abstract level, and the ability of Quantum Technology to produce tangible outcomes in the real world by adopting a more comprehensive Governance-oriented approach to Benchmarking Quantum AI Systems. The purpose of this chapter is to assist the growing community of researchers, practitioners, and Strategic Technology Professionals by providing them with a clear and structured framework for Evaluation of Quantum AI Algorithms. There is a significant focus of the chapter on establishing important reference points that organizations can use to determine when an intelligent system has successfully achieved a true and validated quantum advantage. The ultimate goal is to assist in guiding the responsible development and implementation of Quantum AI to enable the Potential Benefits of Quantum AI to be realized, in ways that are meaningful and impactful.
Current technological advancements of quantum computers highlight the need for application-driven, practical and well-defined methods of benchmarking their performance. As the existing NISQ device's quality of two-qubit gate errors rate is even around one percent and the number of qubits is still limited to a few or several dozen, naturally, we need to propose rather small algorithms instances taken from key promising application areas, such as quantum chemistry, combinatorial optimisation or machine learning. While many techniques for assessing the performance of logical components, such as gate fidelity and qubit coherence exist, it is still challenging to extrapolate those values onto the performance of different quantum algorithms and subroutines. This work aims to introduce a series of initial quantum application benchmarks together with a methodology of execution for measuring the performance and fidelity of the results. The proposed suite refers to several variational algorithms widely used on available NISQ devices but also includes examples of quantum circuits designed for a fault-tolerant quantum computer.