We present microbench.py, a compact (approx. 200 lines) Python script that automates the collection of key compiler metrics, i.e., gate depth, two-qubit-gate count, wall-clock compilation time, and memory footprint, across multiple open-source quantum circuit transpilers. The suite ships with six didactic circuits (3 to 8 qubits) implementing fundamental quantum algorithms and supports Qiskit, tket, Cirq, and the Qiskit-Braket provider; in this paper we showcase results for Qiskit 0.46 and Braket 1.16. The entire run completes in under three minutes on a laptop, emits a single CSV plus publisheable plot, and reproduces the figure here with one command. We release the code under the MIT licence to serve as a quick-start regression harness for NISQ compiler research.
We present microbench.py, a compact (approx. 200 lines) Python script that automates the collection of key compiler metrics, i.e., gate depth, two-qubit-gate count, wall-clock compilation time, and memory footprint, across multiple open-source quantum circuit transpilers. The suite ships with six didactic circuits (3 to 8 qubits) implementing fundamental quantum algorithms and supports Qiskit, tket, Cirq, and the Qiskit-Braket provider; in this paper we showcase results for Qiskit 0.46 and Braket 1.16. The entire run completes in under three minutes on a laptop, emits a single CSV plus publisheable plot, and reproduces the figure here with one command. We release the code under the MIT licence to serve as a quick-start regression harness for NISQ compiler research.
Efficient compilation of quantum algorithms is vital in the era of Noisy Intermediate-Scale Quantum (NISQ) devices. While multiple open-source quantum compilation and circuit optimization frameworks are available, e.g. IBM Qiskit, CQC Tket, Google Cirq, Rigetti Quilc, PyZX, their relative performance is not always clear to a quantum programmer. The growth of complexity and diversity of quantum circuit compilation algorithms creates a demand for a dedicated tool for cross-benchmarking and profiling of inner workflow of the quantum compilation stack. We present an open-source software package, Arline Benchmarks, that is designed to perform automated benchmarking of quantum compilers with the focus on NISQ applications. The name "Arline" was given in honour of Arline Greenbaum Feynman, the first wife of Richard Feynman, the pioneer of quantum computing. We compared several quantum compilation frameworks based on a set of important metrics such as post-optimization gate counts, circuit depth, hardware-dependent circuit cost function, compiler run time etc. with a detailed analysis of metrics for each compilation stage. We performed a variety of compiler tests for random circuits and structured quantum algorithms (VQE, Trotter decomposition, Grover search, Option Pricing via Amplitude Estimation) for several popular quantum hardware architectures. Leveraging cross-platform functionality of Arline, we propose a concept of composite compilation pipeline that combines compiler-specific circuit optimization subroutines in a single compilation stack and finds an optimized sequence of compilation passes. By providing detailed insights into the compilation flow of quantum compilers, Arline Benchmarks offers a valuable toolkit for quantum computing researchers and software developers to gain additional insights into compilers' characteristics.
We present an updated version of PyQBench, an open-source Python library designed for benchmarking gate-based quantum computers, with a focus on certifying qubit von Neumann measurements. This version extends PyQBench's capabilities by incorporating a certification scheme of quantum measurements that evaluates the accuracy on Noisy Intermediate-Scale Quantum devices, alongside its original functionality of von Neumann measurements' discrimination. PyQBench offers a user-friendly command-line interface and Python library integration, allowing users to implement custom measurement schemes and error models for more advanced benchmarking tasks. The new version is specifically designed to support IBM Q devices through integration with the Qiskit library, enabling seamless benchmarking on real quantum hardware. By providing flexible benchmarking tools, PyQBench addresses the critical need for reliable performance metrics in the NISQ era, contributing to the development of error mitigation techniques and the verification of quantum measurement fidelity. The source code is available on GitHub under an open-source license, encouraging community collaboration and further advancements in quantum hardware benchmarking.
Quantum programming languages and their compilers form the critical software layer between abstract quantum algorithm specifications and physical hardware execution -- yet compiler quality varies enormously across the quantum software ecosystem, with suboptimal compilation directly increasing circuit depth, gate count, and consequently error rates on NISQ hardware. This paper proposes the Quantum Compiler Design and Evaluation (QCDE) framework, a systematic methodology for quantum compiler optimisation across four compilation stages: gate synthesis (decomposing abstract gates to hardware-native gate sets), routing (mapping virtual qubits to physical qubits under connectivity constraints), scheduling (ordering gates to minimise decoherence), and cross-layer optimisation (combining all stages with error mitigation awareness). QCDE evaluates six quantum compilers (Qiskit transpiler L1-L3, TKET, BQSKit, Cirq, Quilc, and a novel QCDE-Opt compiler) on 48 benchmark circuits from the QOAS, HQCAD, and QNNDE algorithm ecosystems. QCDE-Opt achieves 18.4% lower CX gate count vs. Qiskit L3 (best existing compiler) through a novel error-aware peephole optimisation pass. BQSKit provides the best synthesis quality at 28.4% CX reduction but 12.4x compilation overhead. TKET achieves the best balance of quality and speed (8.4% CX reduction, 1.8x overhead). QCDE provides a compiler selection guide and the QCDE-Opt compiler as an open-source contribution.
In this study, we present a comprehensive evaluation of popular SDK-to-SDK quantum transpilers (that is transpilers that takes a quantum circuit from an initial SDK and output a quantum circuit in another SDK), focusing on critical metrics such as correctness, failure rate, and transpilation time. To ensure unbiased evaluation and accommodate diverse quantum computing scenarios, we developed two dedicated tools: RandomQC, for generating random quantum circuits across various types (pure random, VQE-like, and SDK-specific circuits), and Benchmarq, to streamline the benchmarking process. Using these tools, we benchmarked prominent quantum transpilers as of February 2024. Our results highlight the superior performance of the qiskit-braket-provider, a specialized transpiler from Qiskit to Braket, achieving a remarkably low failure rate of 0. 2 %. The qBraid-SDK, offering generalized transpilation across multiple SDKs, demonstrated robust but slower performance. The pytket extensions, while fast, faced limitations with complex circuits due to their one-to-one transpilation approach. In particular, the exceptional performance of the qiskit-bracket-provider stems not only from its specialization but also from its architecture, which combines one-to-one transpilation with gate decomposition for unsupported gates, enhancing both speed and capability. This study aims to provide practical guidelines to users of SDK-to-SDK quantum transpilers and guidance to developers for improving the design and development of future tools.
Realizing large-scale quantum advantage is expected to require quantum error correction (QEC), making the compilation and optimization of logical operations a critical area of research. Logical computation imposes distinct constraints and operational paradigms that differ from those of the Noisy Intermediate-Scale Quantum (NISQ) regime, motivating the continued evolution of compilation tools. Given the complexity of this emerging stack, where factors such as gate decomposition precision and computational models must be co-designed, standardized benchmarks and toolkits are valuable for evaluating progress. To support this need, we introduce FTCircuitBench, which serves as: (1) a benchmark suite of impactful quantum algorithms, featuring pre-compiled instances in both Clifford+T and Pauli Based Computation models; (2) a modular end-to-end pipeline allowing users to compile and decompose algorithms for various fault-tolerant architectures, supporting both prebuilt and custom optimization passes; and (3) a toolkit for evaluating the impact of algorithms and optimization across the full compilation stack, providing detailed numerical analysis at each stage. FTCircuitBench is fully open-sourced and maintained on Github.
Reproducibility and comparability remain fundamental challenges in the empirical evaluation of Quantum Software Engineering (QSE) tools. These challenges arise from the intrinsic probabilistic behavior of quantum programs and the variability introduced by quantum hardware and simulator backends. While existing quantum benchmarks typically offer only static circuit definitions, they often leave researchers responsible for data generation and execution, hindering the creation of standardized and reproducible experiments. In this paper, we introduce QSimBench, a novel, open-source benchmark suite and accompanying Python library designed to address these limitations. QSimBench provides over 20 million precomputed shot-level execution traces:20,000measurements each, across14quantum algorithms, multiple circuit sizes (4−15qubits), and six Qiskit-based backends including both ideal and noise-injected simulations. Each configuration includes not only detailed measurement histories but also the original circuit in OpenQASM, the backend's noise model, and configuration metadata, ensuring full transparency and enabling in-depth analysis. QSimBench empowers researchers to conduct scalable, reproducible, and statistically rigorous evaluations of QSE tools, such as those for monitoring, orchestration, and statistical post-processing, without the need for expensive or difficult-to-reproduce quantum executions. The benchmark is distributed with a lightweight Python API that facilitates fast data retrieval, flexible sampling strategies, and seamless integration into experimental pipelines. QSimBench is publicly available, aiming to foster collaborative and reproducible research within the quantum software community.