We present an updated version of PyQBench, an open-source Python library designed for benchmarking gate-based quantum computers, with a focus on certifying qubit von Neumann measurements. This version extends PyQBench's capabilities by incorporating a certification scheme of quantum measurements that evaluates the accuracy on Noisy Intermediate-Scale Quantum devices, alongside its original functionality of von Neumann measurements' discrimination. PyQBench offers a user-friendly command-line interface and Python library integration, allowing users to implement custom measurement schemes and error models for more advanced benchmarking tasks. The new version is specifically designed to support IBM Q devices through integration with the Qiskit library, enabling seamless benchmarking on real quantum hardware. By providing flexible benchmarking tools, PyQBench addresses the critical need for reliable performance metrics in the NISQ era, contributing to the development of error mitigation techniques and the verification of quantum measurement fidelity. The source code is available on GitHub under an open-source license, encouraging community collaboration and further advancements in quantum hardware benchmarking.
We present an updated version of PyQBench, an open-source Python library designed for benchmarking gate-based quantum computers, with a focus on certifying qubit von Neumann measurements. This version extends PyQBench's capabilities by incorporating a certification scheme of quantum measurements that evaluates the accuracy on Noisy Intermediate-Scale Quantum devices, alongside its original functionality of von Neumann measurements' discrimination. PyQBench offers a user-friendly command-line interface and Python library integration, allowing users to implement custom measurement schemes and error models for more advanced benchmarking tasks. The new version is specifically designed to support IBM Q devices through integration with the Qiskit library, enabling seamless benchmarking on real quantum hardware. By providing flexible benchmarking tools, PyQBench addresses the critical need for reliable performance metrics in the NISQ era, contributing to the development of error mitigation techniques and the verification of quantum measurement fidelity. The source code is available on GitHub under an open-source license, encouraging community collaboration and further advancements in quantum hardware benchmarking.
Abstract In this report, we explore benchmarking gate-based quantum devices based on the certification of quantum measurements, which can be viewed as an extension of quantum hypothesis testing. Here, we are interested in a binary certification scheme in which the null and alternative hypotheses are qubit von Neumann measurements. The certification procedure can be used to evaluate the accuracy of Noisy Intermediate-Scale Quantum devices. To support this methodology, we contribute benchmarks to PyQBench, an open-source Python library for benchmarking gate-based quantum computers. We provide a user-friendly command-line interface for certifying measurements from the Fourier family. For more advanced scenarios, PyQBench can also be used as a Python library, allowing users to define custom measurement schemes and error models. The latest version of PyQBench is designed to support IBM Q devices through integration with the Qiskit library, enabling seamless benchmarking on real quantum hardware. By providing flexible benchmarking tools, PyQBench addresses the critical need for reliable performance metrics in the NISQ era, contributing to the development of error mitigation techniques and the verification of quantum measurement fidelity.
In the near-term "NISQ"-era of noisy, intermediate-scale, quantum hardware and beyond, reliably determining the quality of quantum devices becomes increasingly important: users need to be able to compare them with one another, and make an estimate whether they are capable of performing a given task ahead of time. In this work, we develop and release an advanced quantum benchmarking framework in order to help assess the state of the art of current quantum devices. Our testing framework measures the performance of universal quantum devices in a hardware-agnostic way, with metrics that are aimed to facilitate an intuitive understanding of which device is likely to outperform others on a given task. This is achieved through six structured tests that allow for an immediate, visual assessment of how devices compare. Each test is designed with scalability in mind, making this framework not only suitable for testing the performance of present-day quantum devices, but also of those released in the foreseeable future. The series of tests are motivated by real-life scenarios, and therefore emphasise the interplay between various relevant characteristics of quantum devices, such as qubit count, connectivity, and gate and measurement fidelity. We present the benchmark results of twenty-one different quantum devices from IBM, Rigetti and IonQ.
Reproducibility and comparability remain fundamental challenges in the empirical evaluation of Quantum Software Engineering (QSE) tools. These challenges arise from the intrinsic probabilistic behavior of quantum programs and the variability introduced by quantum hardware and simulator backends. While existing quantum benchmarks typically offer only static circuit definitions, they often leave researchers responsible for data generation and execution, hindering the creation of standardized and reproducible experiments. In this paper, we introduce QSimBench, a novel, open-source benchmark suite and accompanying Python library designed to address these limitations. QSimBench provides over 20 million precomputed shot-level execution traces:20,000measurements each, across14quantum algorithms, multiple circuit sizes (4−15qubits), and six Qiskit-based backends including both ideal and noise-injected simulations. Each configuration includes not only detailed measurement histories but also the original circuit in OpenQASM, the backend's noise model, and configuration metadata, ensuring full transparency and enabling in-depth analysis. QSimBench empowers researchers to conduct scalable, reproducible, and statistically rigorous evaluations of QSE tools, such as those for monitoring, orchestration, and statistical post-processing, without the need for expensive or difficult-to-reproduce quantum executions. The benchmark is distributed with a lightweight Python API that facilitates fast data retrieval, flexible sampling strategies, and seamless integration into experimental pipelines. QSimBench is publicly available, aiming to foster collaborative and reproducible research within the quantum software community.
The fragmented landscape of quantum computer benchmarks, characterized by system-specific tools and inconsistent evaluation methodologies, hinders reliable cross-platform performance assessment. We introduce Metriq, an open-source collaborative platform for reproducible cross-platform quantum benchmarking that integrates benchmark definition and execution, data collection, and public presentation into a unified workflow. The Metriq benchmark suite spans both system-level metrics that characterize fundamental device properties such as entanglement quality, gate performance, and circuit speed, as well as application-inspired protocols that assess performance on quantum machine learning, optimization, and quantum simulation tasks. Benchmarks are chosen to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation. Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors, enabling systematic cross-platform comparison. The resulting curated dataset also reveals the practical strengths and limitations of individual benchmarks, creating a feedback loop that informs the ongoing refinement of the suite. To summarize performance across the benchmark suite, we introduce the Metriq Score, a composite index aggregating benchmark outcomes. We further present cross-benchmark analyses enabled by the shared dataset and their correlations with hardware calibration metrics. Through open development and data sharing, Metriq provides a practical foundation for reproducible benchmarking of quantum computers as hardware and benchmarking methods continue to evolve.
We present microbench.py, a compact (approx. 200 lines) Python script that automates the collection of key compiler metrics, i.e., gate depth, two-qubit-gate count, wall-clock compilation time, and memory footprint, across multiple open-source quantum circuit transpilers. The suite ships with six didactic circuits (3 to 8 qubits) implementing fundamental quantum algorithms and supports Qiskit, tket, Cirq, and the Qiskit-Braket provider; in this paper we showcase results for Qiskit 0.46 and Braket 1.16. The entire run completes in under three minutes on a laptop, emits a single CSV plus publisheable plot, and reproduces the figure here with one command. We release the code under the MIT licence to serve as a quick-start regression harness for NISQ compiler research.
As quantum hardware continues to advance across a growing landscape of vendors, it is increasingly important to evaluate device capabilities for specific computational tasks. However, cross-platform benchmarking remains challenging due to diverse hardware architectures, proprietary access interfaces, and the lack of standardized benchmarking frameworks. We introduce metriq-gym, an open-source Python framework designed to facilitate both system-level and application-level benchmarking of quantum devices across multiple providers. The metriq-gym framework offers a unified workflow for executing a suite of community-driven benchmarks, enabling consistent comparisons across heterogeneous quantum platforms. In this poster, we provide an overview of the metriq-gym architecture and demonstrate its capabilities through a curated set of benchmark results obtained from several leading quantum hardware systems.