As large language models (LLMs) become embedded in quantum simulation workflows (IDE copilots, notebook assistants, agentic pipelines), evaluation must move beyond functional correctness to anticipate and catch structured failures before they propagate through expensive pipelines. We present a layered evaluation framework for materials-informed Variational Quantum Eigensolver (VQE) circuit generation: (i) a gatekeeper screening rubric across seven physical and framework criteria; (ii) a circuit fidelity analysis comparing model outputs against analytical and reference-implementation values for H2/STO-3G/Jordan-Wigner/UCCSD, with ansatz classification and gate-composition breakdown; and (iii) design entropy, a run-to-run behavioral consistency metric. We surface a taxonomy of five distinct LLM failure modes (geometry hallucination, nonexistent API usage, runtime integration failures, constraint violations, and plausible-but-unverifiable output), each with distinct detectability profiles and structural to the task rather than to any one model. A forensic audit of the evaluation platform's own source code further establishes that two apparent model failures originated in the harness through silent fallback-template substitution, demonstrating that evaluation infrastructure belongs inside the same trust boundary as the models it tests. Applied across multiple foundation models on a Materials Project integrated pipeline, the framework shows that gatekeeper-style validation is necessary, not optional, for reliable deployment.
As large language models (LLMs) become embedded in quantum simulation workflows (IDE copilots, notebook assistants, agentic pipelines), evaluation must move beyond functional correctness to anticipate and catch structured failures before they propagate through expensive pipelines. We present a layered evaluation framework for materials-informed Variational Quantum Eigensolver (VQE) circuit generation: (i) a gatekeeper screening rubric across seven physical and framework criteria; (ii) a circuit fidelity analysis comparing model outputs against analytical and reference-implementation values for H2/STO-3G/Jordan-Wigner/UCCSD, with ansatz classification and gate-composition breakdown; and (iii) design entropy, a run-to-run behavioral consistency metric. We surface a taxonomy of five distinct LLM failure modes (geometry hallucination, nonexistent API usage, runtime integration failures, constraint violations, and plausible-but-unverifiable output), each with distinct detectability profiles and structural to the task rather than to any one model. A forensic audit of the evaluation platform's own source code further establishes that two apparent model failures originated in the harness through silent fallback-template substitution, demonstrating that evaluation infrastructure belongs inside the same trust boundary as the models it tests. Applied across multiple foundation models on a Materials Project integrated pipeline, the framework shows that gatekeeper-style validation is necessary, not optional, for reliable deployment.
This study evaluates a protocol-guided large language model (LLM) agent workflow for quantum program synthesis and execution. A versioned YAML protocol specifies interface and quantum-semantic requirements while leaving circuit design to the model. The workflow combines Qiskit circuit generation, evaluator-guided repair, and quantum processing unit (QPU) deployment. A matched ablation evaluates four LLM endpoints using the variational quantum eigensolver (VQE) for H2, the quantum approximate optimization algorithm (QAOA) for MaxCut, and Fashion-MNIST quantum-kernel classification. Across 240 trials, protocol guidance increased evaluator completion from 80.0\% to 99.2\%, reduced mean repairs from 2.56 to 0.75, and reduced mean agent time from 55.70 to 30.58~s. Provider-reported token use decreased for three of four endpoints. Completed VQE trials meeting chemical accuracy increased from 16/40 to 29/40, while valid QAOA approximation ratios remained comparable. Semantic review identified rank-one fidelity kernels, reinforcing the distinction between evaluator completion and semantic validity. A 210-run IBM QPU study evaluated the combined generation-and-selection pipelines and confirmed deployment feasibility. The results support protocol guidance as a software-reliability layer that must be complemented by quantum-semantic validation.
Large language models (LLMs) can generate quantum programs, but reliable evaluation remains difficult because benchmarks differ in task composition, scoring, and failure handling. We evaluate 15 hosted frontier, open-weight, quantum-specialized, and retrieval-augmented models on five quantum code-generation suites (530 tasks), plus 10 models on our new QuantumBenchEval (QBE) benchmark (96 tasks across six topics). We report bootstrap confidence intervals, multiplicity-adjusted paired comparisons, and audits of output validity and evaluator behavior. QiskitHumanEval-Standard approaches saturation (98.7% pass@1), while its Hard variant retains headroom (63.8%). None of the five top-two comparisons across primary suites remains significant after multiple-comparison correction. Evaluator design substantially changes scores: removing checks for unspecified quantities raises GPT-6 Astra's QBE pass@1 from 0% to 94.1% (T1) and from 1.2% to 92.9% (T6), without changing model outputs. We identify 17 QuanBench-117 tasks with shared failures linked to benchmark-owned imports. Controlled probes also reveal unexecuted unit tests, leakage of canonical functions into candidate execution, and reference imports that block otherwise valid candidates. These defects can produce both false failures and false passes, so historical QuanBench results require revalidation under an isolated evaluator. Overall, leaderboard scores alone do not adequately characterize progress in quantum code generation. We release our harness, per-sample results, and audit artifacts and recommend validating evaluators, isolating candidate code, and reporting uncertainty and output validity.
The design of high performing quantum circuits remains largely dependent on human expertise. We introduce an autonomous agentic framework that employs large language models (LLMs) to conduct iterative quantum circuit designs under explicit design constraints. Our system integrates seven components: Exploration, Generation, Discussion, Validation, Storage, Evaluation, and Review. These components form a closed-loop workflow that combines web-based knowledge acquisition, literature-grounded critique, executable code generation, and experimental feedback. We evaluate the framework on two tasks: quantum feature map construction for quantum machine learning and ansatz generation for variational quantum eigensolver applications in quantum chemistry. In image classification benchmarks, the best generated feature map outperforms representative quantum feature maps and, when scaled to larger qubit counts, surpasses the classical radial basis function kernel. In molecular ground state estimation across seven molecules, the generated ansatz attains competitive accuracy with widely used chemically inspired and hardware-efficient constructions while satisfying the imposed scaling constraints. These results establish LLM driven agentic system as a viable paradigm for automated quantum circuit design and illustrate how AI systems can participate in iterative scientific optimization workflows across scientific domains.
Large Language Models (LLMs) are increasingly deployed as quantum coding copilots, assisting researchers by generating complex programs for frameworks such as Qiskit, PennyLane, and OpenQASM. However, accurately evaluating the practical utility of these models remains a significant challenge. Existing benchmarks are highly fragmented, ranging from syntax-centric unit tests to simulator- and hardware-aware metrics, making results difficult to compare across different languages and hardware targets. To address this fragmentation, this paper systematically reviews recent evaluation suites and post-training methodologies for quantum code generation. We distill current evaluation criteria into four practical dimensions: syntax and API validity, semantic and physical correctness, resource and hardware feasibility, and execution-based feedback. Building upon these dimensions, we present a unified taxonomy that maps current benchmarks to emerging model adaptation techniques, specifically retrieval-augmented generation (RAG), fine-tuning, multi-agent workflows, and reinforcement learning (RL). Our analysis highlights a clear shift: moving away from prompt-only baselines and toward agentic workflows driven by quantum-verifiable rewards. This shift directly targets the semantic gap between superficially compilable code and physically executable quantum programs. In addition, we identify key research opportunities-such as standardized semantic metrics and integration with quantum error correction-aimed at fostering a more reproducible, verifiable, and hardware-grounded ecosystem for automated quantum software development.
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
Although most software engineering work today targets classical systems, quantum computing has become an area of growing interest for developers who want to understand alternative computational models. This thesis addresses the difficulty of learning to construct and reason about quantum circuits by focusing on small, illustrative circuit-based algorithms (Bell states, GHZ states, and a reduced Grover’s search) as structured examples around which an interactive learning platform is built. The aim is to support step-by-step understanding of these circuits while investigating how large language models (LLMs) can assist with circuit generation and explanation. The work presents a web-based platform that allows users to incrementally build circuits, visualize their layer-by-layer evolution, and inspect key properties such as gate composition and measurement probabilities. On top of this platform, two LLM-based assistance strategies are implemented: a stateless baseline that operates solely on naturallanguage prompts, and a context-aware, agentic mode that receives structured information about the current circuit and simulation results. Their behavior is compared in an offline evaluation on a fixed set of exemplar tasks, using a rubric that scores correctness of the resulting circuits, relevance of explanations to the actual circuit, and pedagogical clarity. The results indicate that the agentic, state-aware integration produces more accurate, better-grounded, and more instructionally useful outputs than the stateless baseline on the examined tasks. The thesis contributes an example-driven approach to quantum computing education for software engineers, an inspection-oriented platform design, and empirical evidence that platform-state-aware LLM agents can provide higher-quality assistance than purely prompt-based interaction in this domain.