Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2512.00766 · ACM Transactions on Software Engineering and Methodology

Code Comments for Quantum Software Development Kits: An Empirical Study on Qiskit

Zenghui Zhou, Yuechen Li, Yi Cai, Jinlong Wen, Xiaohan Yu, Zheng Zheng, Beibei Yin

Published Sep 15, 2026• 0 citations
arXivHTMLPDF

Abstract

Quantum computing is gaining attention from academia and industry. With the quantum Software Development Kits (SDKs), programmers can develop quantum software to explore the power of quantum computing. However, programmers may face challenges in understanding quantum software due to the non-intuitive quantum mechanics. To facilitate software development and maintenance, code comments offered in quantum SDKs serve as a natural language explanation of program functionalities and logical flows. Despite their importance, scarce research systematically reports their value and provides constructive guidelines for programmers. To address this gap, our paper focuses on Qiskit, one of the most popular quantum SDKs, and presents CC4Q, the first dataset of code comments for quantum computing. CC4Q incorporates 12,812 code comment pairs and 28,076 sentence-level code comment units, the latter of which involve heavy human annotation. Regarding the annotation, we validate the applicability of the developer-intent taxonomy used in classical programs, and also propose a new taxonomy considering quantum-specific knowledge. We conduct an empirical study comprehensively interpreting code comments from three perspectives: comment structure and coverage, developers’ intentions, and associated quantum topics. Our findings uncover key differences in code comments between classical and quantum software, and also outline quantum-specific knowledge relevant to quantum software development.

DOI
10.1145/3847113

Related papers

CC4Q Workbench: From Quantum Repositories to Analyzable Code Comments

Zenghui Zhou, Yuechen Li, Zheng Zheng

Quantum software development kits (SDKs) expose high-level APIs whose use requires an understanding of underlying mathematical principles, circuit semantics, and hardware properties. Code comments can provide natural-language or structural explanations that are not explicit in API calls. However, studying comments in a new quantum SDK still requires researchers to coordinate separate steps for comment extraction, quantum-specific classification, and manual verification, resulting in repeated integration effort. To address this problem, we present CC4Q Workbench, an open-source workbench that integrates these steps into a reusable workflow. Given a Python repository at a specified revision, CC4Q Workbench automatically extracts code-comment pairs, segments comments into sentence-level units, and applies two trained classifiers to identify quantum-specific content and assign quantum-knowledge categories. Its web interface supports exploration of bundled datasets, inspection of code snippets, corpus construction from new repositories, manual annotation with model predictions hidden during binary labeling, and classifier fine-tuning using exported annotations. The release provides aggregate statistics for six SDK datasets comprising 39,593 sentence-level code comment units (SCCUs), of which 17,517 are browsable in the web interface. Applied to three additional application-layer Qiskit repositories outside the original CC4Q dataset, the pipeline ran without code changes and produced 11,517 SCCUs. Manual verification of 379 SCCUs showed 85.0% agreement between human annotations and the binary classifier's predictions.

arXiv:2610.32651•Oct 23, 2026•0 citations

Understanding Quantum Software Engineering Challenges An Empirical Study on Stack Exchange Forums and GitHub Issues

Mohamed Raed El aoun, Heng Li, Foutse Khomh +1 more

With the advance of quantum computing, quantum software becomes critical for exploring the full potential of quantum computing systems. Recently, quantum software engineering (QSE) becomes an emerging area attracting more and more attention. However, it is not clear what are the challenges and opportunities of quantum computing facing the software engineering community. This work aims to understand the QSE-related challenges perceived by developers. We perform an empirical study on Stack Exchange forums where developers post-QSE-related questions & answers and Github issue reports where developers raise QSE-related issues in practical quantum computing projects. Based on an existing taxonomy of question types on Stack Overflow, we first perform a qualitative analysis of the types of QSE-related questions asked on Stack Exchange forums. We then use automated topic modeling to uncover the topics in QSE-related Stack Exchange posts and GitHub issue reports. Our study highlights some particularly challenging areas of QSE that are different from that of traditional software engineering, such as explaining the theory behind quantum computing code, interpreting quantum program outputs, and bridging the knowledge gap between quantum computing and classical computing, as well as their associated opportunities.

arXiv:2205.03181•IEEE International Conference on Software Maintenance (ICSM)•Sep 1, 2021•37 citations

Qiskit HumanEval: An Evaluation Benchmark for Quantum Code Generative Models

Sanjay Vishwakarma, Francis Harkins, Siddharth Golecha +7 more

Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit – a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.

arXiv:2406.14712•IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 15, 2024•19 citations

Large Language Models for Quantum Code Translation: An Exploratory Study

Xiyuan Li, Yuechen Li, Jianjun Zhao

Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.

cs.SECross-listedarXiv:2610.34841•Oct 27, 2026•0 citations

Software Engineering for Quantum Programming: How Far Are We?

Manuel De Stefano, Fabiano Pecorelli, Dario Di Nucci +2 more

Quantum computing is no longer only a scientific interest but is rapidly becoming an industrially available technology that can potentially overcome the limits of classical computation. Over the last years, all major companies have provided frameworks and programming languages that allow developers to create their quantum applications. This shift has led to the definition of a new discipline called quantum software engineering, which is demanded to define novel methods for engineering large-scale quantum applications. While the research community is successfully embracing this call, we notice a lack of systematic investigations into the state of the practice of quantum programming. Understanding the challenges that quantum developers face is vital to precisely define the aims of quantum software engineering. Hence, in this paper, we first mine all the GitHub repositories that make use of the most used quantum programming frameworks currently on the market and then conduct coding analysis sessions to produce a taxonomy of the purposes for which quantum technologies are used. In the second place, we conduct a survey study that involves the contributors of the considered repositories, which aims to elicit the developers' opinions on the current adoption and challenges of quantum programming. On the one hand, the results highlight that the current adoption of quantum programming is still limited. On the other hand, there are many challenges that the software engineering community should carefully consider: these do not strictly pertain to technical concerns but also socio-technical matters.

arXiv:2203.16969•Mar 31, 2022•4 citations

Quantum software development: a survey

Luis Jiménez-Navajas, Fabian Bühler, Frank Leymann +3 more

Over the last few years, quantum computing has been growing at an exponential pace. Every day, new techniques, frameworks, modeling, and programming languages are emerging that aim to facilitate the development of quantum software, which is key to achieving the promising applications of quantum computing. However, which of these are actively used and the degree of satisfaction of researchers and developers regarding these quantum software frameworks and languages is not known. To address this, we conducted a survey to characterize which modeling tools and which quantum programming languages are used during the quantum software lifecycle. Researchers in academia and industry developers were surveyed, and a total of 57 responses were collected. The results indicate that during quantum software development, some models and diagrams are used to guide development. In addition, the survey results show what quantum programming languages are the most used alongside the classical programming languages employed to build hybrid programs, among other important insights. The implications of this survey are: (i) to find out what the current trends are within quantum software development and (ii) to find out what the needs are of quantum software developers with respect to current modeling and programming languages and tools.

Quantum Information and Computation•Jun 1, 2024•5 citations
Quantum computing is gaining attention from academia and industry. With the quantum Software Development Kits (SDKs), programmers can develop quantum software to explore the power of quantum computing. However, programmers may face challenges in understanding quantum software due to the non-intuitive quantum mechanics. To facilitate software development and maintenance, code comments offered in quantum SDKs serve as a natural language explanation of program functionalities and logical flows. Despite their importance, scarce research systematically reports their value and provides constructive guidelines for programmers. To address this gap, our paper focuses on Qiskit, one of the most popular quantum SDKs, and presents CC4Q, the first dataset of code comments for quantum computing. CC4Q incorporates 12,812 code comment pairs and 28,076 sentence-level code comment units, the latter of which involve heavy human annotation. Regarding the annotation, we validate the applicability of the developer-intent taxonomy used in classical programs, and also propose a new taxonomy considering quantum-specific knowledge. We conduct an empirical study comprehensively interpreting code comments from three perspectives: comment structure and coverage, developers’ intentions, and associated quantum topics. Our findings uncover key differences in code comments between classical and quantum software, and also outline quantum-specific knowledge relevant to quantum software development.
Quantum software development kits (SDKs) expose high-level APIs whose use requires an understanding of underlying mathematical principles, circuit semantics, and hardware properties. Code comments can provide natural-language or structural explanations that are not explicit in API calls. However, studying comments in a new quantum SDK still requires researchers to coordinate separate steps for comment extraction, quantum-specific classification, and manual verification, resulting in repeated integration effort. To address this problem, we present CC4Q Workbench, an open-source workbench that integrates these steps into a reusable workflow. Given a Python repository at a specified revision, CC4Q Workbench automatically extracts code-comment pairs, segments comments into sentence-level units, and applies two trained classifiers to identify quantum-specific content and assign quantum-knowledge categories. Its web interface supports exploration of bundled datasets, inspection of code snippets, corpus construction from new repositories, manual annotation with model predictions hidden during binary labeling, and classifier fine-tuning using exported annotations. The release provides aggregate statistics for six SDK datasets comprising 39,593 sentence-level code comment units (SCCUs), of which 17,517 are browsable in the web interface. Applied to three additional application-layer Qiskit repositories outside the original CC4Q dataset, the pipeline ran without code changes and produced 11,517 SCCUs. Manual verification of 379 SCCUs showed 85.0% agreement between human annotations and the binary classifier's predictions.
With the advance of quantum computing, quantum software becomes critical for exploring the full potential of quantum computing systems. Recently, quantum software engineering (QSE) becomes an emerging area attracting more and more attention. However, it is not clear what are the challenges and opportunities of quantum computing facing the software engineering community. This work aims to understand the QSE-related challenges perceived by developers. We perform an empirical study on Stack Exchange forums where developers post-QSE-related questions & answers and Github issue reports where developers raise QSE-related issues in practical quantum computing projects. Based on an existing taxonomy of question types on Stack Overflow, we first perform a qualitative analysis of the types of QSE-related questions asked on Stack Exchange forums. We then use automated topic modeling to uncover the topics in QSE-related Stack Exchange posts and GitHub issue reports. Our study highlights some particularly challenging areas of QSE that are different from that of traditional software engineering, such as explaining the theory behind quantum computing code, interpreting quantum program outputs, and bridging the knowledge gap between quantum computing and classical computing, as well as their associated opportunities.
Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit – a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.
Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.
Quantum computing is no longer only a scientific interest but is rapidly becoming an industrially available technology that can potentially overcome the limits of classical computation. Over the last years, all major companies have provided frameworks and programming languages that allow developers to create their quantum applications. This shift has led to the definition of a new discipline called quantum software engineering, which is demanded to define novel methods for engineering large-scale quantum applications. While the research community is successfully embracing this call, we notice a lack of systematic investigations into the state of the practice of quantum programming. Understanding the challenges that quantum developers face is vital to precisely define the aims of quantum software engineering. Hence, in this paper, we first mine all the GitHub repositories that make use of the most used quantum programming frameworks currently on the market and then conduct coding analysis sessions to produce a taxonomy of the purposes for which quantum technologies are used. In the second place, we conduct a survey study that involves the contributors of the considered repositories, which aims to elicit the developers' opinions on the current adoption and challenges of quantum programming. On the one hand, the results highlight that the current adoption of quantum programming is still limited. On the other hand, there are many challenges that the software engineering community should carefully consider: these do not strictly pertain to technical concerns but also socio-technical matters.
Over the last few years, quantum computing has been growing at an exponential pace. Every day, new techniques, frameworks, modeling, and programming languages are emerging that aim to facilitate the development of quantum software, which is key to achieving the promising applications of quantum computing. However, which of these are actively used and the degree of satisfaction of researchers and developers regarding these quantum software frameworks and languages is not known. To address this, we conducted a survey to characterize which modeling tools and which quantum programming languages are used during the quantum software lifecycle. Researchers in academia and industry developers were surveyed, and a total of 57 responses were collected. The results indicate that during quantum software development, some models and diagrams are used to guide development. In addition, the survey results show what quantum programming languages are the most used alongside the classical programming languages employed to build hybrid programs, among other important insights. The implications of this survey are: (i) to find out what the current trends are within quantum software development and (ii) to find out what the needs are of quantum software developers with respect to current modeling and programming languages and tools.