Multimodal emotion recognition can benefit from combining complementary information from facial images, speech, and text.However, integrating heterogeneous modalities obtained from independently collected datasets introduces challenges in data alignment and label consistency.This study presents a hybrid quantum-classical framework that combines modality-specific classical encoders with a quantum re-uploading variational classifier for six-class emotion recognition.Facial images, speech samples, and textual samples were obtained from separate publicly available datasets and aligned at the emotion-label level to construct a unified multimodal experimental setting.Image, speech, and text encoders generated modality-specific representations that were fused through a classical neural network and projected into a six-dimensional quantum input space.A six-qubit quantum circuit with three re-uploading layers repeatedly encoded the fused features before six-class classification.On a test set of 6,347 samples, the classical Image-Speech-Text fusion model achieved 73.94% accuracy and a 70.64% macro F1-score, whereas the proposed quantum reuploading model achieved 72.57% accuracy and a 68.76% macro F1-score.The quantum model used 8,706 trainable parameters compared with 15,440,006 for the classical multimodal model, representing a 99.94% reduction in trainable parameters.The results indicate that quantum re-uploading can provide competitive performance for multimodal emotion recognition while highlighting the limitations of label-level alignment across independently collected datasets.
Multimodal emotion recognition can benefit from combining complementary information from facial images, speech, and text.However, integrating heterogeneous modalities obtained from independently collected datasets introduces challenges in data alignment and label consistency.This study presents a hybrid quantum-classical framework that combines modality-specific classical encoders with a quantum re-uploading variational classifier for six-class emotion recognition.Facial images, speech samples, and textual samples were obtained from separate publicly available datasets and aligned at the emotion-label level to construct a unified multimodal experimental setting.Image, speech, and text encoders generated modality-specific representations that were fused through a classical neural network and projected into a six-dimensional quantum input space.A six-qubit quantum circuit with three re-uploading layers repeatedly encoded the fused features before six-class classification.On a test set of 6,347 samples, the classical Image-Speech-Text fusion model achieved 73.94% accuracy and a 70.64% macro F1-score, whereas the proposed quantum reuploading model achieved 72.57% accuracy and a 68.76% macro F1-score.The quantum model used 8,706 trainable parameters compared with 15,440,006 for the classical multimodal model, representing a 99.94% reduction in trainable parameters.The results indicate that quantum re-uploading can provide competitive performance for multimodal emotion recognition while highlighting the limitations of label-level alignment across independently collected datasets.
Multimodal emotion recognition can benefit from combining complementary information from facial images, speech, and text. However, integrating heterogeneous modalities obtained from independently collected datasets introduces challenges in data alignment and label consistency. This study presents a hybrid quantum–classical framework that combines modality-specific classical encoders with a quantum re-uploading variational classifier for six-class emotion recognition. Facial images, speech samples, and textual samples were obtained from separate publicly available datasets and aligned at the emotion-label level to construct a unified multimodal experimental setting. Image, speech, and text encoders generated modality-specific representations that were fused through a classical neural network and projected into a six-dimensional quantum input space. A six-qubit quantum circuit with three re-uploading layers repeatedly encoded the fused features before six-class classification. On a test set of 6,347 samples, the classical Image–Speech–Text fusion model achieved 73.94% accuracy and a 70.64% macro F1-score, whereas the proposed quantum re-uploading model achieved 72.57% accuracy and a 68.76% macro F1-score. The quantum model used 8,706 trainable parameters compared with 15,440,006 for the classical multimodal model, representing a 99.94% reduction in trainable parameters. The results indicate that quantum re-uploading can provide competitive performance for multimodal emotion recognition while highlighting the limitations of label-level alignment across independently collected datasets.
Emotion recognition seeks to identify a person’s state by examining signals such as facial expressions, speech, physiological signals and conversational content. Yet accurate emotion recognition presents obstacles because these signals can be complex, noisy, nonlinear and heavily affected by surrounding context. This paper investigates quantum-based representations combined with machine learning techniques to enhance emotion classification. The research examines emotion sources, including facial expressions, speech, EEG signals and conversational data. It explores how quantum-inspired techniques can merge with established deep learning methods to produce representations. The design presented fuses a neural network with a four qubit ZZFeatureMap component enabling the system to learn data from both conventional and quantum derived feature spaces. Experiments examined EEG segments for each participant and the system achieved average recognition rates of seventy nine point six percent for valence and eighty one point five percent for arousal across ten repeated trials. These findings show that quantum derived features can add information when combined with conventional deep learning methods. Nevertheless because experiments used simulated circuits the results do not prove a performance advantage over conventional techniques on actual quantum computers. The study also highlights difficulties such as quantum noise, computational scalability, experimental assessment, data privacy and real world deployment.
Speech emotion recognition (SER) remains fragile in real-world conditions because emotional cues are subtle, speaker-dependent, and easily confounded by recording variability, while high-performing deep models typically rely on large and carefully curated training sets. Quantum machine learning offers an alternative way to introduce nonlinear correlation modeling with compact modules, yet existing quantum SER studies remain limited, and the impact of circuit structure is not well understood. This paper presents HQTN-SER, a hybrid quantum-classical framework that investigates how quantum tensor network connectivity can support SER under small-qubit settings. HQTN-SER introduces (i) an MPS-inspired quantum tensor network module that enforces structured interactions to model correlations in speech representations with a small number of trainable parameters, and (ii) a fusion strategy that combines quantum measurement features with a learned classical latent embedding for end-to-end emotion classification. We evaluate HQTN-SER on three public benchmarks (RAVDESS, SAVEE, and MDER) under a unified preprocessing and training protocol. The proposed model achieves test accuracies of 80.09\% on RAVDESS, 78.12\% on SAVEE, and 73.51\% on MDER, with corresponding macro-F1 scores of 0.8012, 0.7826, and 0.7351, respectively. The model also exhibits stable convergence while using only three or four qubits, suggesting that MPS-inspired circuit structure can be an effective and hardware-aware design choice for quantum-assisted SER. These results provide a compact experimental baseline for studying structured quantum modules in affective computing.
Abstract This study explores Quantum AI’s potential in enhancing Speech Emotion Recognition (SER) systems. Our primary objective is to evaluate the performance of quantum-inspired algorithms compared to classical machine-learning approaches in accurately identifying and classifying emotions from speech signals. We hypothesise that quantum computing techniques can improve the efficiency and accuracy of emotion recognition, particularly in handling complex, highdimensional acoustic data. We developed a hybrid quantum-classical model that combines quantum-inspired neural networks with traditional feature extraction techniques. Our approach utilises quantum circuits for data encoding and quantum variational algorithms for classification. We implemented quantum versions of support vector machines (QSVM) and quantum approximate optimisation algorithms (QAOA) for emotion classification tasks. These quantum methods were compared against a hybrid Convolutional Neural Network and Long Short Term (LSTM). The hybrid network is called a Convolutional Long Short Term network(CLSTM). The study employed a custom dataset for speech emotion recognition. We prepossessed the audio data to extract relevant acoustic features, including mel-frequency cepstral coefficients (MFCCs), pitch, and energy. In contrast, our implemented quantum-inspired model for this corpus showed lower performance, with the highest training accuracy of 30%, struggling with most emotions but performing best with ’Trust’. These varied results across different implementations suggest that while Quantum AI techniques show promise in advancing speech emotion recognition, their effectiveness may depend on the specific dataset, language, and model architecture used. Finally, the speed of a simulated quantum network is shown to outperform current methods in terms of matrix multiplication by leaps and bounds.
This paper presents a quantum convolutional neural network (QCNN) that classifies the happy and sad emotions from the fabled facial emotion recognition dataset (FER), FER-2013. The model uses the data re-uploading concept on the qubit(s) of the parameterized quantum circuits (PQCs) in the quantum convolutional layer of the hybrid QCNN model, which achieved a 70.25 % validation accuracy on the test set. Later, a model averaging ensemble approach was implemented which yielded results motivating further investigation.
Quantum machine learning (QML) offers a promising avenue for advancing representation learning in complex signal domains. In this study, we investigate the use of parameterised quantum circuits (PQCs) for speech emotion recognition (SER)-a challenging task due to the subtle temporal variations and overlapping affective states in vocal signals. We propose a hybrid quantum-classical architecture that integrates PQCs into a conventional convolutional neural network (CNN), leveraging quantum properties such as superposition and entanglement to enrich emotional feature representations. Experimental evaluations on three benchmark datasets IEMOCAP, RECOLA, and MSP-IMPROV-demonstrate that our hybrid model achieves improved classification performance relative to a purely classical CNN baseline, with over 50% reduction in trainable parameters. Furthermore, Adjusted Rand Index (ARI) analysis demonstrates that the quantum model yields feature representations with improved alignment to true emotion classes compared with the classical model, reinforcing the observed performance gains. This work provides early evidence of the potential for QML to enhance emotion recognition and lays the foundation for future quantum-enabled affective computing systems.