The DisCoCirc framework for natural language processing allows the construction of compositional models of text, by combining units for individual words together according to the grammatical structure of the text to be read by the model. The compositional nature of a model can give rise to two things: compositional generalisation - the ability of a model to generalise outside its training distribution by learning compositional rules underpinning the entire data distribution - and compositional understandability - making sense of how the model works by inspecting its modular components in isolation and the processes through which they are combined. We present a proof-of-concept implementation of a DisCoCirc model on a trapped-ion quantum computer, solving a hand-crafted question answering task. DisCoCirc allows a quantum model to be trained in classical simulation on small instances, while larger instances at test time are evaluated on a quantum computer. This avoids the barren plateau pitfalls of other common QML setups. We find that the accuracy of the models decay by at most 5% when evaluated on the Test set: the model compositionally generalises. Secondly, we consider how well DisCoCirc models can learn different types of compositional behaviour. We compare both quantum circuit based models, as well as classical neural networks under a series of training paradigms, on a dataset derived from one of the bAbI tasks, extended to test a series of aspects of compositionality. Both architectures score within 5% of one another on the productivity and substitutivity tasks, but differ for the systematicity and overgeneralisation tasks. Overall, we find the neural models are more prone to overfitting the Train data. Finally, we inspect the trained models, both by comparing them to manuallyconstructed perfect compositional models, and by considering how the model components interact with one another, explaining how the models behave and exhibiting the models’ compositional understandability.
The DisCoCirc framework for natural language processing allows the construction of compositional models of text, by combining units for individual words together according to the grammatical structure of the text to be read by the model. The compositional nature of a model can give rise to two things: compositional generalisation - the ability of a model to generalise outside its training distribution by learning compositional rules underpinning the entire data distribution - and compositional understandability - making sense of how the model works by inspecting its modular components in isolation and the processes through which they are combined. We present a proof-of-concept implementation of a DisCoCirc model on a trapped-ion quantum computer, solving a hand-crafted question answering task. DisCoCirc allows a quantum model to be trained in classical simulation on small instances, while larger instances at test time are evaluated on a quantum computer. This avoids the barren plateau pitfalls of other common QML setups. We find that the accuracy of the models decay by at most 5% when evaluated on the Test set: the model compositionally generalises. Secondly, we consider how well DisCoCirc models can learn different types of compositional behaviour. We compare both quantum circuit based models, as well as classical neural networks under a series of training paradigms, on a dataset derived from one of the bAbI tasks, extended to test a series of aspects of compositionality. Both architectures score within 5% of one another on the productivity and substitutivity tasks, but differ for the systematicity and overgeneralisation tasks. Overall, we find the neural models are more prone to overfitting the Train data. Finally, we inspect the trained models, both by comparing them to manuallyconstructed perfect compositional models, and by considering how the model components interact with one another, explaining how the models behave and exhibiting the models’ compositional understandability.
We present the first implementation of text-level quantum natural language processing, a field where quantum computing and AI have found a fruitful intersection. We focus on the QDisCoCirc model, which is underpinned by a compositional approach to rendering AI interpretable: the behaviour of the whole can be understood in terms of the behaviour of parts, and the way they are put together. Interpretability is crucial for understanding the unwanted behaviours of AI. By leveraging the compositional structure in the model's architecture, we introduce a novel setup which enables 'compositional generalisation': we classically train components which are then composed to generate larger test instances, the evaluation of which asymptotically requires a quantum computer. Another key advantage of our approach is that it bypasses the trainability challenges arising in quantum machine learning. The main task that we consider is the model-native task of question-answering, and we handcraft toy scale data that serves as a proving ground. We demonstrate an experiment on Quantinuum's H1-1 trapped-ion quantum processor, which constitutes the first proof of concept implementation of scalable compositional QNLP. We also provide resource estimates for classically simulating the model. The compositional structure allows us to inspect and interpret the word embeddings the model learns for each word, as well as the way in which they interact. This improves our understanding of how it tackles the question-answering task. As an initial comparison with classical baselines, we considered transformer and LSTM models, as well as GPT-4, none of which succeeded at compositional generalisation.
Quantum computing and AI have found a fruitful intersection in the field of natural language processing. We focus on the recently proposed DisCoCirc framework for natural language, and propose a quantum adaptation, QDisCoCirc. This is motivated by a compositional approach to rendering AI interpretable: the behavior of the whole can be understood in terms of the behavior of parts, and the way they are put together. For the model-native primitive operation of text similarity, we derive quantum algorithms for fault-tolerant quantum computers to solve the task of question-answering within QDisCoCirc, and show that this is BQP-hard; note that we do not consider the complexity of question-answering in other natural language processing models. Assuming widely-held conjectures, implementing the proposed model classically would require super-polynomial resources. Therefore, it could provide a meaningful demonstration of the power of practical quantum processors. The model construction builds on previous work in compositional quantum natural language processing. Word embeddings are encoded as parameterized quantum circuits, and compositionality here means that the quantum circuits compose according to the linguistic structure of the text. We outline a method for evaluating the model on near-term quantum processors, and elsewhere we report on a recent implementation of this on quantum hardware. In addition, we adapt a quantum algorithm for the closest vector problem to obtain a Grover-like speedup in the fault-tolerant regime for our model. This provides an unconditional quadratic speedup over any classical algorithm in certain circumstances, which we will verify empirically in future work.
Language models based on neural network primarily focus on learning patterns and representations from the input data, without explicitly modeling the grammatical structure of the text, thus often lacks of the consideration of syntax information. The distributional compositional categorical (DisCoCat) model is a framework that can both capture the distributional information of words and the syntactic structure of sentence. Thus, we research the application of DisCoCat models in both quantum and classical natural language processing. Specifically, in classical and quantum natural language processing, we respectively converted sentences into tensor networks and quantum circuits based on the DisCoCat model to get representation containing both semantic and syntactic information of language in classical and quantum computer. We conduct experiment for a text binary classification task in both classical and quantum natural language processing setting to verify the effectiveness of DisCoCat model. The experimental results shown that the combination of DisCoCat with quantum computing has significant potential.
This thesis introduces quantum natural language processing (QNLP) models based on a simple yet powerful analogy between computational linguistics and quantum mechanics: grammar as entanglement. The grammatical structure of text and sentences connects the meaning of words in the same way that entanglement structure connects the states of quantum systems. Category theory allows to make this language-to-qubit analogy formal: it is a monoidal functor from grammar to vector spaces. We turn this abstract analogy into a concrete algorithm that translates the grammatical structure onto the architecture of parameterised quantum circuits. We then use a hybrid classical-quantum algorithm to train the model so that evaluating the circuits computes the meaning of sentences in data-driven tasks. The implementation of QNLP models motivated the development of DisCoPy (Distributional Compositional Python), the toolkit for applied category theory of which the first chapter gives a comprehensive overview. String diagrams are the core data structure of DisCoPy, they allow to reason about computation at a high level of abstraction. We show how they can encode both grammatical structures and quantum circuits, but also logical formulae, neural networks or arbitrary Python code. Monoidal functors allow to translate these abstract diagrams into concrete computation, interfacing with optimised task-specific libraries. The second chapter uses DisCopy to implement QNLP models as parameterised functors from grammar to quantum circuits. It gives a first proof-of-concept for the more general concept of functorial learning: generalising machine learning from functions to functors by learning from diagram-like data. In order to learn optimal functor parameters via gradient descent, we introduce the notion of diagrammatic differentiation: a graphical calculus for computing the gradients of parameterised diagrams.
Abstract Natural language consists of words, sentences, and larger units. Guided by grammatical structure, words compose with each other to form sentences. Similarly, guided by discourse structure, sentences compose with each other to form dialogues and documents. Classical machine learning algorithms have achieved significant success in learning the meanings of words. When it comes to meanings of sentences and discourse units, they however fall short of being compositional. The DisCoCat model of meaning, introduced by Clark, Coecke, and Sadrzadeh in 2010, provides a solution—at the sentence level—using higher-order tensors, the learning of which has been a challenge. A recent initiative known as Quantum Natural Language Processing (QNLP) introduces a translation between the DisCoCat tensors and Variational Quantum Circuits (VQC). This offers the potential of learning these higher-order tensors more efficiently when the circuits are executed on quantum computers. In previous work, we lifted the DisCoCat framework from the sentence level to the discourse level using a Fock space semantics. In this paper, we extend the DisCoCat-VQC translation to this semantics and experiment with it in a discourse task. We develop a massive dataset with 16,400 entries inspired by a major coreference resolution task, known as the Winograd Schema Challenge, proposed as a test of machine intelligence. Noisy and noiseless simulations were executed on IBMQ software, and the parameters of the discourse VQCs were learnt. The model converged to 77.5% accuracy, surpassing a Bag-of-Words model that neglects any structure. It also outperformed 2 out of 3 state-of-the-art classical coreference resolution architectures. These findings highlight the significant potential of quantum machine learning in advancing discourse analysis and structured natural language processing.
Quantum approaches to natural language processing (NLP) are redefining how linguistic information is represented and processed. While traditional hybrid quantum-classical models rely heavily on classical neural networks, recent advancements propose a novel framework, DisCoCirc, capable of directly encoding entire documents as parameterised quantum circuits (PQCs), besides enjoying some additional interpretability and compositionality benefits. Following these ideas, this paper introduces an efficient methodology for converting large-scale texts into quantum circuits using tree-like representations of pregroup diagrams. Exploiting the compositional parallels between language and quantum mechanics, grounded in symmetric monoidal categories, our approach enables faithful and efficient encoding of syntactic and discourse relationships in long and complex texts (up to 6410 words in our experiments) to quantum circuits. The developed system is provided to the community as part of the augmented open-source quantum NLP package lambeq Gen II.