Video data is multimodal in its nature, where an utterance can involve linguistic, visual and acoustic information. Therefore, a key challenge for video sentiment analysis is how to combine different modalities for sentiment recognition effectively. The latest neural network approaches achieve state-of-the-art performance, but they neglect to a large degree of how humans understand and reason about sentiment states. By contrast, recent advances in quantum probabilistic neural models have achieved comparable performance to the state-of-the-art, yet with better transparency and increased level of interpretability. However, the existing quantum-inspired models treat quantum states as either a classical mixture or as a separable tensor product across modalities, without triggering their interactions in a way that they are correlated or non-separable (i.e., entangled). This means that the current models have not fully exploited the expressive power of quantum probabilities. To fill this gap, we propose a transparent quantum probabilistic neural model. The model induces different modalities to interact in such a way that they may not be separable, encoding crossmodal information in the form of non-classical correlations. Comprehensive evaluation on two benchmarking datasets for video sentiment analysis shows that the model achieves significant performance improvement. We also show that the degree of non-separability between modalities optimizes the post-hoc interpretability.
Video data is multimodal in its nature, where an utterance can involve linguistic, visual and acoustic information. Therefore, a key challenge for video sentiment analysis is how to combine different modalities for sentiment recognition effectively. The latest neural network approaches achieve state-of-the-art performance, but they neglect to a large degree of how humans understand and reason about sentiment states. By contrast, recent advances in quantum probabilistic neural models have achieved comparable performance to the state-of-the-art, yet with better transparency and increased level of interpretability. However, the existing quantum-inspired models treat quantum states as either a classical mixture or as a separable tensor product across modalities, without triggering their interactions in a way that they are correlated or non-separable (i.e., entangled). This means that the current models have not fully exploited the expressive power of quantum probabilities. To fill this gap, we propose a transparent quantum probabilistic neural model. The model induces different modalities to interact in such a way that they may not be separable, encoding crossmodal information in the form of non-classical correlations. Comprehensive evaluation on two benchmarking datasets for video sentiment analysis shows that the model achieves significant performance improvement. We also show that the degree of non-separability between modalities optimizes the post-hoc interpretability.
A long-standing goal in the field of Artificial Intelligence (AI) is to develop systems that can perceive and understand human multimodal language. This requires both the consideration of context in the form of surrounding utterances in a conversation, i.e., context modelling, as well as the impact of different modalities (e.g., linguistic, visual acoustic), i.e., multimodal fusion. In the last few years, significant strides have been made towards the interpretation of human language due to simultaneous advancement in deep learning, data gathering and computing infrastructure. AI models have been investigated to either model interactions across distinct modalities, i.e., linguistic, visual and acoustic, or model interactions across parties in a conversation, achieving unprecedented levels of performance. However, AI models are often designed with only performance as their design target, leaving aside other essential factors such as transparency, interpretability, and how humans understand and reason about cognitive states. In line with this observation, in this dissertation, we develop quantum probabilistic neural models and techniques that allow us to capture rational and irrational cognitive biases, without requiring a priori understanding and identification of them. First, we present a comprehensive empirical comparison of state-of-the-art (SOTA) modality fusion strategies for video sentiment analysis. The findings provide us helpful insights into the development of more effective modality fusion models incorporating quantum-inspired components. Second, we introduce an end-to-end complex-valued neural model for video sentiment analysis, simulating quantum procedural steps, outside of physics, into the neural network modelling paradigm. Third, we investigate non-classical correlations across different modalities. In particular, we describe a methodology to model interactions between image and text for an information retrieval scenario. The results provide us with theoretical and empirical insights to develop a transparent end-to-end probabilistic neural model for video emotion detection in conversations, capturing non-classical correlations across distinct modalities. Fourth, we introduce a theoretical framework to model user's cognitive states underlying their multimodal decision perspectives, and propose a methodology to capture interference of modalities in decision making. Overall, we show that our models advance the SOTA on various affective analysis tasks, achieve high transparency due to the mapping to quantum physics meanings, and improve post-hoc interpretability, unearthing useful and explainable knowledge about cross-modal interactions.
Video sentiment analysis as a decision-making process is inherently complex, involving the fusion of decisions from multiple modalities and the so-caused cognitive biases. Inspired by recent advances in quantum cognition, we show that the sentiment judgment from one modality could be incompatible with the judgment from another, i.e., the order matters and they cannot be jointly measured to produce a final decision. Thus the cognitive process exhibits ``quantum-like'' biases that cannot be captured by classical probability theories. Accordingly, we propose a fundamentally new, quantum cognitively motivated fusion strategy for predicting sentiment judgments. In particular, we formulate utterances as quantum superposition states of positive and negative sentiment judgments, and uni-modal classifiers as mutually incompatible observables, on a complex-valued Hilbert space with positive-operator valued measures. Experiments on two benchmarking datasets illustrate that our model significantly outperforms various existing decision level and a range of state-of-the-art content-level fusion approaches. The results also show that the concept of incompatibility allows effective handling of all combination patterns, including those extreme cases that are wrongly predicted by all uni-modal classifiers.
Quantum theory provides non-classical principles, such as superposition and entanglement, that inspires promising paradigms in machine learning. However, most existing quantum-inspired fusion models rely solely on unitary or unitary-like transformations to generate quantum entanglement. While theoretically expressive, such approaches often suffer from training instability and limited generalizability. In this work, we propose a Quantum-inspired Neural Network with Quantum Jump (QiNN-QJ) for multimodal entanglement modelling. Each modality is firstly encoded as a quantum pure state, after which a differentiable module simulating the QJ operator transforms the separable product state into the entangled representation. By jointly learning Hamiltonian and Lindblad operators, QiNN-QJ generates controllable cross-modal entanglement among modalities with dissipative dynamics, where structured stochasticity and steady-state attractor properties serve to stabilize training and constrain entanglement shaping. The resulting entangled states are projected onto trainable measurement vectors to produce predictions. In addition to achieving superior performance over the state-of-the-art models on benchmark datasets, including CMU-MOSI, CMU-MOSEI, and CH-SIMS, QiNN-QJ facilitates enhanced post-hoc interpretability through von-Neumann entanglement entropy. This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations.
This article proposes a quantum multimodal neural network (QMNN) model that can be implemented on parameterized quantum circuits (PQCs), providing a novel avenue for processing multimodal data and performing advanced multimodal sentiment analysis tasks. The comprehensive QMNN model is structured into four fundamental blocks: multimodal data preprocessing, unimodal feature extraction, multimodal feature fusion, and optimization. Through these blocks, multimodal data are initially preprocessed and encoded into quantum states. Subsequently, visual and textual features are extracted from the quantum states and are then integrated to learn the interactions between different modalities. Finally, the model parameters are fine-tuned to optimize the sentiment analysis performance. Simulation results confirm that QMNN surpasses state-of-the-art baselines, using significantly lower input dimensions and substantially fewer parameters than classical models. Furthermore, the entanglement, integrity, robustness, and scalability of the model are analyzed in depth. Internally, the strong entanglement within the multimodal fusion block enhances interactions between textual and visual features, and the integrity of the model reflects the indispensable contribution of each component to the overall performance. Externally, robustness ensures the model operates stably under noisy conditions and incomplete inputs, and scalability enables it to efficiently adapt to varying architectural depths and widths. The above simulation results and performance analyses showcase the comprehensive strength of our proposed model.
For Urdu sentiment analysis, we present a quantum-enhanced multimodal system that combines a variational quantum circuit (VQC) classifier with traditional feature extractors. To create a cohesive embedding, a cross-modal transformer model first processes raw text, audio, and visual information. On a noisy intermediate-scale quantum (NISQ) processor, these fused embeddings are then angle-encoded into a parameterized quantum state. Simultaneous perturbation stochastic approximation (SPSA) is used to improve the trainable rotation gates of the VQC under realistic hardware-inspired noise models. Our quantum-enhanced model outperforms the classical cross-modal baseline(82.4%/79.6%)and a purely classical MLP head (83.1%/80.8%) with an overall accuracy of 84.2% and a macro-F1 score of 81.7% when tested on UMSA-U, a curated dataset of 6,000 Urdu video clips with aligned transcripts, audio, and facial expression annotations. The findings show that noise-robust SPSA tuning maintains performance on simulated hardware error rates up to 5%, while quantum feature encoding produces more separable decision boundaries. These findings establish a new standard for Urdu sentiment recognition on NISQ devices and demonstrate the potential of variational quantum classifiers for low-resource, multimodal language solutions.
We introduce a quantum-enhanced multimodal framework for Urdu sentiment analysis that integrates classical feature extractors with a variational quantum circuit (VQC) classifier. Raw text, audio, and visual signals are first processed by a cross-modal transformer to produce a unified embedding. These fused embeddings are then angle-encoded into a parameterized quantum state on a noisy intermediate-scale quantum (NISQ) processor. The VQC's trainable rotation gates are optimized via simultaneous perturbation stochastic approximation (SPSA) under realistic hardware-inspired noise models. Evaluated on UMSA-U, a curated dataset of 6,000 Urdu video clips with aligned transcripts, audio, and facial-expression annotations, our quantum-enhanced model achieves 84.2% overall accuracy and a macro-F1 score of 81.7% improving over both the classical cross-modal baseline (82.4%/79.6%) and a purely classical MLP head (83.1%/80.8%). Ablation studies demonstrate that quantum feature encoding yields more separable decision boundaries, and noise-robust SPSA tuning sustains performance on simulated hardware error rates up to 5%. These results showcase the promise of variational quantum classifiers for low-resource, multimodal language tasks and set a new benchmark for Urdu sentiment understanding on NISQ devices.