As quantum hardware continues to advance across a growing landscape of vendors, it is increasingly important to evaluate device capabilities for specific computational tasks. However, cross-platform benchmarking remains challenging due to diverse hardware architectures, proprietary access interfaces, and the lack of standardized benchmarking frameworks. We introduce metriq-gym, an open-source Python framework designed to facilitate both system-level and application-level benchmarking of quantum devices across multiple providers. The metriq-gym framework offers a unified workflow for executing a suite of community-driven benchmarks, enabling consistent comparisons across heterogeneous quantum platforms. In this poster, we provide an overview of the metriq-gym architecture and demonstrate its capabilities through a curated set of benchmark results obtained from several leading quantum hardware systems.
As quantum hardware continues to advance across a growing landscape of vendors, it is increasingly important to evaluate device capabilities for specific computational tasks. However, cross-platform benchmarking remains challenging due to diverse hardware architectures, proprietary access interfaces, and the lack of standardized benchmarking frameworks. We introduce metriq-gym, an open-source Python framework designed to facilitate both system-level and application-level benchmarking of quantum devices across multiple providers. The metriq-gym framework offers a unified workflow for executing a suite of community-driven benchmarks, enabling consistent comparisons across heterogeneous quantum platforms. In this poster, we provide an overview of the metriq-gym architecture and demonstrate its capabilities through a curated set of benchmark results obtained from several leading quantum hardware systems.
The fragmented landscape of quantum computer benchmarks, characterized by system-specific tools and inconsistent evaluation methodologies, hinders reliable cross-platform performance assessment. We introduce Metriq, an open-source collaborative platform for reproducible cross-platform quantum benchmarking that integrates benchmark definition and execution, data collection, and public presentation into a unified workflow. The Metriq benchmark suite spans both system-level metrics that characterize fundamental device properties such as entanglement quality, gate performance, and circuit speed, as well as application-inspired protocols that assess performance on quantum machine learning, optimization, and quantum simulation tasks. Benchmarks are chosen to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation. Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors, enabling systematic cross-platform comparison. The resulting curated dataset also reveals the practical strengths and limitations of individual benchmarks, creating a feedback loop that informs the ongoing refinement of the suite. To summarize performance across the benchmark suite, we introduce the Metriq Score, a composite index aggregating benchmark outcomes. We further present cross-benchmark analyses enabled by the shared dataset and their correlations with hardware calibration metrics. Through open development and data sharing, Metriq provides a practical foundation for reproducible benchmarking of quantum computers as hardware and benchmarking methods continue to evolve.
As quantum computing systems continue to mature, there is an increasing need for benchmarking methodologies that capture performance in terms of meaningful, application-level metrics. In this work, we present a scalable framework for application-level quantum benchmarking that is designed to support internal system evaluation and cross-platform comparison across technology providers. Our framework is guided by a set of core principles, including measurability, simplicity, scalability, and extensibility. We present 13 benchmark families that reflect realistic workloads across multiple domains. This enables the systematic evaluation of the quality of solutions, the total execution time, total used energy, as well as Time-to-Solution. The benchmarks are designed to be reproducible, interpretable across stakeholder groups, and adaptable to evolving system capabilities. The framework aims to bridge the gap between low-level performance metrics and real-world value, providing a unified approach to assessing quantum systems. The resulting benchmarks support development and validation and contribute to the foundation of industry-wide benchmarking standards.
As quantum technologies continue to advance, the proliferation of hardware architectures with diverse capabilities and limitations has underscored the importance of benchmarking as a tool to compare performance across platforms. Achieving fair, scalable and consistent evaluations is a key open problem in quantum computing, particularly in the pre-fault-tolerant era. To address this challenge, we introduce QuSquare, a quality-oriented benchmark suite designed to provide a scalable, fair, reproducible, and well-defined framework for assessing the performance of quantum devices across hardware architectures. QuSquare consists of four benchmark tests that evaluate quantum hardware performance at both the system and application levels: Partial Clifford Randomized, Multipartite Entanglement, Transverse Field Ising Model (TFIM) Hamiltonian Simulation, and Data Re-Uploading Quantum Neural Network (QNN). Together, these benchmarks offer an integral, hardware-agnostic, and impartial methodology to quantify the quality and capabilities of current quantum computers, supporting fair cross-platform comparisons and fostering the development of future performance standards.
Quantum software tools for a wide variety of design tasks on and across different levels of abstraction are crucial in order to eventually realize useful quantum applications. This requires practical and relevant benchmarks for new software tools to be empirically evaluated and compared to the current state of the art. Although benchmarks for specific design tasks are commonly available, the demand for an overarching cross-level benchmark suite has not yet been fully met and there is no mutual consolidation in how quantum software tools are evaluated thus far. In this work, we propose the MQT Bench benchmark suite (as part of the Munich Quantum Toolkit , MQT) based on four core traits: (1) cross-level support for different abstraction levels, (2) accessibility via an easy-to-use web interface (https://www.cda.cit.tum.de/mqtbench/) and a Python package, (3) provision of a broad selection of benchmarks to facilitate generalizability, as well as (4) extendability to future algorithms, gate-sets, and hardware architectures. By comprising more than 70,000 benchmark circuits ranging from 2 to 130 qubits on four abstraction levels, MQT Bench presents a first step towards benchmarking different abstraction levels with a single benchmark suite to increase comparability, reproducibility, and transparency.
We present an updated version of PyQBench, an open-source Python library designed for benchmarking gate-based quantum computers, with a focus on certifying qubit von Neumann measurements. This version extends PyQBench's capabilities by incorporating a certification scheme of quantum measurements that evaluates the accuracy on Noisy Intermediate-Scale Quantum devices, alongside its original functionality of von Neumann measurements' discrimination. PyQBench offers a user-friendly command-line interface and Python library integration, allowing users to implement custom measurement schemes and error models for more advanced benchmarking tasks. The new version is specifically designed to support IBM Q devices through integration with the Qiskit library, enabling seamless benchmarking on real quantum hardware. By providing flexible benchmarking tools, PyQBench addresses the critical need for reliable performance metrics in the NISQ era, contributing to the development of error mitigation techniques and the verification of quantum measurement fidelity. The source code is available on GitHub under an open-source license, encouraging community collaboration and further advancements in quantum hardware benchmarking.
We present a platform-agnostic modular architecture that addresses the increasingly fragmented landscape of quantum computing benchmarking by decoupling problem generation, circuit execution, and results analysis into independent, interoperable components. Supporting over 20 benchmark variants ranging from simple algorithmic tests like Bernstein-Vazirani to complex Hamiltonian simulation with observable calculations, the system integrates with multiple circuit generation APIs (Qiskit, CUDA-Q, Cirq) and enables diverse workflows. We validate the architecture through successful integration with Sandia's pyGSTi for advanced circuit analysis and CUDA-Q for multi-GPU HPC simulations. Extensibility of the system is demonstrated by implementing dynamic circuit variants of existing benchmarks and a new quantum reinforcement learning benchmark, which become readily available across multiple execution and analysis modes. Our primary contribution is identifying and formalizing modular interfaces that enable interoperability between incompatible benchmarking frameworks, demonstrating that standardized interfaces reduce ecosystem fragmentation while preserving optimization flexibility. This architecture has been developed as a key enhancement to the continually evolving QED-C Application-Oriented Performance Benchmarks for Quantum Computing suite.