Sarcasm detection in unimodal or multimodal setting is a very complex task. Sarcasm, emotion, and sentiment are related to each other, and hence any multitask model could be an effective way to leverage the interdependence among these tasks. In order to better represent these clandestine associations, we avoid solely relying on traditional machine learning methods to encode the relationships between the modalities. In this article, we propose a hybrid quantum model that banks upon the low computational complexity and robust representational power of a variational quantum circuit (VQC) and the tried and tested dense neural network to tackle sentiment, emotion, and sarcasm classification simultaneously. We empirically establish that the quantum properties like superposition, entanglement, and interference will better capture and replicate not only the cross-modal interactions between text, acoustics, and visuals but also the correlations between the three responses. We consider the extended MUStARD dataset to evaluate our proposed hybrid model. The results show that our proposed hybrid quantum framework yields more promising results for the primary task of sarcasm detection with the help of the two secondary classification tasks,viz.sentiment and emotion.
Sarcasm detection in unimodal or multimodal setting is a very complex task. Sarcasm, emotion, and sentiment are related to each other, and hence any multitask model could be an effective way to leverage the interdependence among these tasks. In order to better represent these clandestine associations, we avoid solely relying on traditional machine learning methods to encode the relationships between the modalities. In this article, we propose a hybrid quantum model that banks upon the low computational complexity and robust representational power of a variational quantum circuit (VQC) and the tried and tested dense neural network to tackle sentiment, emotion, and sarcasm classification simultaneously. We empirically establish that the quantum properties like superposition, entanglement, and interference will better capture and replicate not only the cross-modal interactions between text, acoustics, and visuals but also the correlations between the three responses. We consider the extended MUStARD dataset to evaluate our proposed hybrid model. The results show that our proposed hybrid quantum framework yields more promising results for the primary task of sarcasm detection with the help of the two secondary classification tasks,viz.sentiment and emotion.
Sentiment, emotion, and sarcasm analysis in multimodal dialogues is crucial for understanding the underlying intentions and attitudes expressed by individuals. Traditional methods often struggle to capture the full intensity of these polarities, leading to less accurate results. To address this limitation, we propose a quantum-inspired approach leveraging Quantum Neural Networks (QNNs) for enhanced classification and intensity analysis. A key component of our method is the Variational Quantum Eigensolver (VQE), a hybrid quantum-classical algorithm that optimizes the parameters of the QNN by minimizing the eigenvalues of a Hamiltonian system. This optimization enables the network to learn complex relationships in multimodal data more effectively. Our approach surpasses state-of-the-art methods, achieving up to 7.5 % higher accuracy and 6.8 % greater precision. Experiments on benchmark datasets such as MUStARD, Memotion, CMU-MOSEI, and MELD demonstrate its effectiveness, with an F1-score of 87.3 % on CMU-MOSEI. This method is particularly beneficial in domains like social media, customer support, and entertainment, where both verbal and non-verbal cues play a critical role in accurate sentiment analysis .
Sarcasm and sentiment embody intrinsic uncertainty of human cognition, making joint detection of multi-modal sarcasm and sentiment a challenging task.In view of the advantages of quantum probability (QP) in modeling such uncertainty, this paper explores the potential of QP as a mathematical framework and proposes a QP driven multi-task (QPM) learning framework.The QPM framework involves a complex-valued multi-modal representation encoder, a quantum-like fusion network and a quantum measurement mechanism.Each multimodal (e.g., textual, visual) utterance is first encoded as a quantum superposition of a set of basis terms using a complex-valued representation.Then, the quantum-like fusion network leverages quantum state composition and quantum interference to model the contextual interaction between adjacent utterances and the correlations across modalities respectively.Finally, quantum incompatible measurements are performed on the multi-modal representation of each utterance to yield the probabilistic outcomes of sarcasm and sentiment recognition.Experimental results show the state-of-the-art performance of our model.
Sarcasm, sentiment, and emotion are three typical kinds of spontaneous affective responses of humans to external events and they are tightly intertwined with each other. Such events may be expressed in multiple modalities (e.g., linguistic, visual and acoustic), e.g., multi-modal conversations. Joint analysis of humans’ multi-modal sarcasm, sentiment, and emotion is an important yet challenging topic, as it is a complex cognitive process involving both cross-modality interaction and cross-affection correlation. From the probability theory perspective, cross-affection correlation also means that the judgments on sarcasm, sentiment, and emotion are incompatible. However, this exposed phenomenon cannot be sufficiently modelled by classical probability theory due to its assumption of compatibility. Neither do the existing approaches take it into consideration. In view of the recent success of quantum probability (QP) in modeling human cognition, particularly contextual incompatible decision making, we take the first step towards introducing QP into joint multi-modal sarcasm, sentiment, and emotion analysis. Specifically, we propose aQUantum probabIlity driven multi-modal sarcasm, sEntiment and emoTion analysis framework, termed QUIET. Extensive experiments on two datasets and the results show that the effectiveness and advantages of QUIET in comparison with a wide range of the state-of-the-art baselines. We also show the great potential of QP in multi-affect analysis.
Multimodal emotion recognition can benefit from combining complementary information from facial images, speech, and text.However, integrating heterogeneous modalities obtained from independently collected datasets introduces challenges in data alignment and label consistency.This study presents a hybrid quantum-classical framework that combines modality-specific classical encoders with a quantum re-uploading variational classifier for six-class emotion recognition.Facial images, speech samples, and textual samples were obtained from separate publicly available datasets and aligned at the emotion-label level to construct a unified multimodal experimental setting.Image, speech, and text encoders generated modality-specific representations that were fused through a classical neural network and projected into a six-dimensional quantum input space.A six-qubit quantum circuit with three re-uploading layers repeatedly encoded the fused features before six-class classification.On a test set of 6,347 samples, the classical Image-Speech-Text fusion model achieved 73.94% accuracy and a 70.64% macro F1-score, whereas the proposed quantum reuploading model achieved 72.57% accuracy and a 68.76% macro F1-score.The quantum model used 8,706 trainable parameters compared with 15,440,006 for the classical multimodal model, representing a 99.94% reduction in trainable parameters.The results indicate that quantum re-uploading can provide competitive performance for multimodal emotion recognition while highlighting the limitations of label-level alignment across independently collected datasets.
Sentiment analysis finds widespread applications in health, marketing, finance, stock markets, media, and politics. To analyze attitudes and emotions in textual data, handling large datasets and significant computational power is essential. Traditional computing methods struggle with the growing data volume, prompting interest in quantum computing as a promising alternative with its inherent high-speed processing capacity. This study focuses on sentiment analysis applied to texts derived from bilateral conversation dialogues. The primary objective is to categorize emotions within the text as positive, neutral, or negative, while concurrently identifying the speaker. To achieve this, a novel quantum-classical hybrid approach is proposed. The quantum side of this approach includes the variational quantum circuit (VQC). On the classical side, preprocessing of the data set, feature extraction with a model containing LSTM, and optimizing the parameters of VQC are performed. The proposed approach was trained and tested using a data set containing bilateral conversations. As a result of the tests, the proposed approach achieved a higher accuracy rate compared to studies using the classical approach. Thus, the effectiveness of the proposed approach is confirmed.
This article proposes a quantum multimodal neural network (QMNN) model that can be implemented on parameterized quantum circuits (PQCs), providing a novel avenue for processing multimodal data and performing advanced multimodal sentiment analysis tasks. The comprehensive QMNN model is structured into four fundamental blocks: multimodal data preprocessing, unimodal feature extraction, multimodal feature fusion, and optimization. Through these blocks, multimodal data are initially preprocessed and encoded into quantum states. Subsequently, visual and textual features are extracted from the quantum states and are then integrated to learn the interactions between different modalities. Finally, the model parameters are fine-tuned to optimize the sentiment analysis performance. Simulation results confirm that QMNN surpasses state-of-the-art baselines, using significantly lower input dimensions and substantially fewer parameters than classical models. Furthermore, the entanglement, integrity, robustness, and scalability of the model are analyzed in depth. Internally, the strong entanglement within the multimodal fusion block enhances interactions between textual and visual features, and the integrity of the model reflects the indispensable contribution of each component to the overall performance. Externally, robustness ensures the model operates stably under noisy conditions and incomplete inputs, and scalability enables it to efficiently adapt to varying architectural depths and widths. The above simulation results and performance analyses showcase the comprehensive strength of our proposed model.