The approach to question answering is challenging because it usually requires finding useful information from within and between question and answer sentences for sentence semantic matching. The key information mined from existing question and answer sentences, as a supplement to semantic information, maybe helpful for this task. However, capturing the intra-sentence and inter-sentence semantic interactions is somewhat difficult given the implicit interrelationships between question and answer sentences. Although the learning effect of multi-layer neural network is good, it lacks explanatory and theoretical support. This paper mainly studies how to select the best answer sentence from a set of question and answer sentences, mine information more effectively, perform semantic matching, and have a certain interpretability. Therefore, we propose a hierarchical attention network (QHAN) based on the mathematical framework of quantum theory, which integrates the attention mechanism under quantum measurement and density matrix (DMATT) and the attention mechanism under quantum weak measurement and weak value (WMATT) in one in a unified model framework. QHAN can not only discover key sentences in a set of question and answer sentences and key words in a sentence, but also perform sentence semantic matching quickly and accurately. Furthermore, QHAN has the advantage of interpretability in terms of models due to the physical meaning and attentional full weight distribution implied by quantum theory. Extensive experiments on the question answering dataset show that our method is comparable to the baselines and can explain why the selected question and answer sentence is the best option in terms of intra-sentence and inter-sentence attention distributions.
Language Modeling (LM) is a fundamental research topic in a range of areas. Recently, inspired by quantum theory, a novel Quantum Language Model (QLM) has been proposed for Information Retrieval (IR). In this paper, we aim to broaden the theoretical and practical basis of QLM. We develop a Neural Network based Quantum-like Language Model (NNQLM) and apply it to Question Answering. Specifically, based on word embeddings, we design a new density matrix, which represents a sentence (e.g., a question or an answer) and encodes a mixture of semantic subspaces. Such a density matrix, together with a joint representation of the question and the answer, can be integrated into neural network architectures (e.g., 2-dimensional convolutional neural networks). Experiments on the TREC-QA and WIKIQA datasets have verified the effectiveness of our proposed models.
Given the query, key and value matrices Q,K,V∈Rn×d, the attention module is defined as Att(Q,K,V)=D−1AV where A=exp(QK⊤/d) with exp(⋅) applied entrywise, D=diag(A1n). The attention module is the backbone of modern transformers and large language models, but explicitly forming the softmax matrix D−1A incurs Ω(n2) time, motivating numerous approximation schemes that reduce runtime to O(nd) via sparsity or low-rank factorization.
We propose a quantum data structure that approximates any row of Att(Q,K,V) using only row queries to Q,K,V. Our algorithm preprocesses these matrices in O(ε−1n0.5(sλ2.5+sλ1.5d+α0.5d)) time, where ε is the target accuracy, sλ is the λ-statistical dimension of the exponential kernel defined by Q and K, and α measures the row distortion of V that is at most d/srank(V), the stable rank of V. Each row query can be answered in O(sλ2+sλd) time.
To our knowledge, this is the first quantum data structure that approximates rows of the attention matrix in sublinear time with respect to n. Our approach relies on a quantum Nyström approximation of the exponential kernel, quantum multivariate mean estimation for computing D, and quantum leverage score sampling for the multiplication with V.