Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.
AI Summary
Can LLMs translate executable quantum programs between high-level SDKs while preserving required behavior? This exploratory study evaluates Qiskit-to-PennyLane and Qiskit-to-Q# translation using four LLM configurations on 43 tasks adapted from QuanBench. The workflow supplies source code, task descriptions, assertions, SDK-version constraints, runtime guidance, and quantum-semantic instructions, including qubit ordering and control–target conventions. Generated programs are executed in versioned native environments and checked against benchmark assertions, with ten translation attempts per model–task–target combination.
Zero-shot task-averaged Pass@1 ranges from 16.28% to 72.33%. Flash performs best on PennyLane at 72.33%, while GPT performs best on Q# at 71.63%; their respective Pass@3 scores are 86.53% and 82.11%. One verified source–target example improves both metrics for Pro and GLM on both targets. Across 3,440 zero-shot attempts, 1,787 fail: missing generated source artifacts account for 39.2% of failures, compilation errors for 34.9%, and assertion violations for 9.0%. Failure categories are assigned automatically from generation status, execution status, and tracebacks.
The stated scope is preliminary: four configurations, two translation directions, and a small benchmark originally designed for code generation rather than translation. One original task is excluded because its reference implementation could not reliably pass its assertions. Passing the available tests does not establish general semantic or structural preservation; dedicated checks for relative phase, bit ordering, and probabilistic outputs, broader benchmarks, and execution-guided repair are left for future work.
Little sign of AI-generated textAI use disclosedIncrementalMissing comparisonUnclear novelty
LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness.
Error found: The supplied paper omits the numerical one-shot results needed to substantiate its claim that both tested models improve on both target SDKs, although its failure counts establish an aggregate improvement.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.
Can LLMs translate executable quantum programs between high-level SDKs while preserving required behavior? This exploratory study evaluates Qiskit-to-PennyLane and Qiskit-to-Q# translation using four LLM configurations on 43 tasks adapted from QuanBench. The workflow supplies source code, task descriptions, assertions, SDK-version constraints, runtime guidance, and quantum-semantic instructions, including qubit ordering and control–target conventions. Generated programs are executed in versioned native environments and checked against benchmark assertions, with ten translation attempts per model–task–target combination.
Zero-shot task-averaged Pass@1 ranges from 16.28% to 72.33%. Flash performs best on PennyLane at 72.33%, while GPT performs best on Q# at 71.63%; their respective Pass@3 scores are 86.53% and 82.11%. One verified source–target example improves both metrics for Pro and GLM on both targets. Across 3,440 zero-shot attempts, 1,787 fail: missing generated source artifacts account for 39.2% of failures, compilation errors for 34.9%, and assertion violations for 9.0%. Failure categories are assigned automatically from generation status, execution status, and tracebacks.
The stated scope is preliminary: four configurations, two translation directions, and a small benchmark originally designed for code generation rather than translation. One original task is excluded because its reference implementation could not reliably pass its assertions. Passing the available tests does not establish general semantic or structural preservation; dedicated checks for relative phase, bit ordering, and probabilistic outputs, broader benchmarks, and execution-guided repair are left for future work.
There exist various Software Development Kits (SDKs) tailored to different quantum computing platforms. These are known as Quantum SDKs (QSDKs). Examples include but are not limited to Qiskit, Cirq, and PennyLane. However, this diversity presents significant challenges for interoperability and cross-platform development of hybrid quantum-classical software systems. Traditional rule-based transpilers for translating code between QSDKs are time-consuming to design and maintain, requiring deep expertise and rigid mappings in the source and destination code. In this study, we explore the use of Large Language Models (LLMs) as a flexible and automated solution. Leveraging their pretrained knowledge and contextual reasoning capabilities, we position LLMs as programming language-agnostic transpilers capable of converting quantum programs from one QSDK to another while preserving functional equivalence. Our approach eliminates the need for manually defined transformation rules and offers a scalable solution to quantum software portability. This work represents a step toward enabling intelligent, generalpurpose transpilation in the quantum computing ecosystem.
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift and introduce quantum-api-drift, a benchmark for measuring version fidelity, defined here as execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair in LLM-generated quantum SDK code. We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0. We evaluate 17 models on 50 tasks with 3 samples per prompt, yielding 450 generated samples and 1,350 executions per model. Sixteen models are tested in a matched REST API setting with a 1024-token output cap, while GPT-5.4 (Codex CLI) is reported separately as a reference configuration. Across the 16 matched REST models, diagonal Pass@1 ranges from 0.02 to 0.85. Claude Opus 4.7 is strongest on v0.43 and v2.0, while Grok 4.20 is strongest on v1.3 at 0.513. Error profiles differ systematically by model strength: weaker models fail mainly with broken imports, while stronger models more often reach deprecation-level failures. Documentation-guided repair succeeds for 0.19 to 0.59 of repair attempts overall and is consistently much more effective for migration to v2.0 than to v1.3. The benchmark artifacts are publicly available at https://github.com/arasyi/quantum-api-drift. These results show that version alignment is a distinct evaluation axis for quantum code generation and that API drift remains only partly recoverable even with migration guidance.
Large Language Models (LLMs) are increasingly deployed as quantum coding copilots, assisting researchers by generating complex programs for frameworks such as Qiskit, PennyLane, and OpenQASM. However, accurately evaluating the practical utility of these models remains a significant challenge. Existing benchmarks are highly fragmented, ranging from syntax-centric unit tests to simulator- and hardware-aware metrics, making results difficult to compare across different languages and hardware targets. To address this fragmentation, this paper systematically reviews recent evaluation suites and post-training methodologies for quantum code generation. We distill current evaluation criteria into four practical dimensions: syntax and API validity, semantic and physical correctness, resource and hardware feasibility, and execution-based feedback. Building upon these dimensions, we present a unified taxonomy that maps current benchmarks to emerging model adaptation techniques, specifically retrieval-augmented generation (RAG), fine-tuning, multi-agent workflows, and reinforcement learning (RL). Our analysis highlights a clear shift: moving away from prompt-only baselines and toward agentic workflows driven by quantum-verifiable rewards. This shift directly targets the semantic gap between superficially compilable code and physically executable quantum programs. In addition, we identify key research opportunities-such as standardized semantic metrics and integration with quantum error correction-aimed at fostering a more reproducible, verifiable, and hardware-grounded ecosystem for automated quantum software development.
Abstract Quantum computing is steadily transitioning from theoretical exploration to experimental deployment, bringing with it distinct engineering challenges. This shift has led to the emergence of Quantum Software Engineering (QSE), a research area that adapts and extends software engineering principles to the quantum domain. In parallel, Large Language Models (LLMs) have become widely adopted in classical software development workflows. Although their potential to assist QSE tasks is increasingly discussed, existing evidence remains fragmented across research venues and practitioner sources. This study presents a Multivocal Literature Review that systematically synthesizes peer-reviewed and grey literature to characterize how LLMs are used in QSE and assess their reported limitations. We identified and analyzed 24 primary sources, mapped their contributions onto the quantum software development lifecycle, examined the distribution of adopted LLM families, and synthesized recurring limitations. The results show that LLM adoption in QSE is predominantly code-centric, with most studies focusing on program synthesis, repair, and refactoring, while lifecycle phases such as requirements engineering, architectural reasoning, and comprehensive validation receive limited attention. Reported challenges include semantic inaccuracies, execution failures on real quantum backends, sensitivity to prompt formulation, and limited domain coverage. Moreover, the empirical landscape is largely shaped by proprietary models, raising concerns regarding reproducibility and longitudinal comparability. Overall, current evidence suggests that LLMs function primarily as assistive tools for bounded tasks rather than as robust, lifecycle-spanning engineering agents in quantum software development.
Recent advances in large language models (LLMs) have enabled the automation of an increasing number of programming tasks, including code generation for scientific and engineering domains. In rapidly evolving software ecosystems such as quantum software development, where frameworks expose complex abstractions, a central question is how best to incorporate domain knowledge into LLM-based assistants while preserving maintainability as libraries evolve.
In this work, we study specialization strategies for Qiskit code generation using the Qiskit-HumanEval benchmark. We compare a parameter-specialized fine-tuned baseline introduced in prior work against a range of recent general-purpose LLMs enhanced with retrieval-augmented generation (RAG) and agent-based inference with execution feedback.
Our results show that modern general-purpose LLMs consistently outperform the parameter-specialized baseline. While the fine-tuned model achieves approximately 47% pass@1 on Qiskit-HumanEval, recent general-purpose models reach 60-65% under zero-shot and retrieval-augmented settings, and up to 85% for the strongest evaluated model when combined with iterative execution-feedback agents -representing an improvement of more than 20% over zero-shot general-purpose performance and more than 35% over the parameter-specialized baseline.
Agentic execution feedback yields the most consistent improvements, albeit at increased runtime cost, while RAG provides modest and model-dependent gains. These findings indicate that performance gains can be achieved without domain-specific fine-tuning, instead relying on inference-time augmentation, thereby enabling a more flexible and maintainable approach to LLM-assisted quantum software development.
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.