An Evaluation of Multi-Modal Approaches for Sentiment Analysis using Deep Learning

Abstract

This thesis presents an evaluation of eight deep-learning-based multimodal models for sentiment analysis, applied to the CMU Multimodal Opinion Sentiment Expression in Recordings (CMU MOSEI) and CMU Multimodal Sentiment Opinions (CMU MOSI) benchmark datasets. The study delves into the realm of multimodal sentiment analysis by integrating text, audio and visual data. Through rigorous experimentation and analysis, key observations are drawn regarding the performance and effectiveness of each model. To evaluate the models’ performance in classifying positive and negative sentiment across various modalities, the study employs a range of metrics including accuracy, F1-score, loss, mean absolute error (MAE), and correlation. Beyond demonstrating the promise of multimodal sentiment analysis, this research offers valuable knowledge. It sheds light on both the optimal configurations for models and how incorporating various modalities impacts the accuracy of sentiment classification.