Personalized Multimodal Sentiment Analysis Under Uncertain Modalities Missing via Pretraining and Online Learning
Date
Supervisor
Item type
Journal Article
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
Elsevier BV
Abstract
Currently, multimodal sentiment analysis (MSA) for personalized users under uncertain modalities missing has become a new challenging problem. To address this issue, we propose a two-step idea. First, we propose an effective MSA model under uncertain modalities missing and train it with some public datasets, thus to enable the model to possess better preliminary MSA ability. Then, we make the pretrained model to continuously learn user’s personalized characteristics with online learning methods, thereby enable the model grow into a robust model for personalized MSA. Based on this idea, we propose a Personalized MSA model under uncertain modalities missing via Pretraining and Online Learning (termed as PMSAPO). For Personalized MSA under uncertain modalities missing, PMSAPO firstly generates the fused modality and allocate weights for each modality with a Fully Connected Neural Network Evaluation Module. Then, PMSAPO completes the final sentiment classification based on the fusion modality with a Joint feature optimization module. For the pretrained PMSAPO, we make it autonomously learn the personalized users via our proposed online learning techniques, including an online meta-learning method, a learning rate adaptive adjustment strategy, and a dynamic weight assignment strategy for sample data. Finally, based on three public benchmark datasets (IEMOCAP, MELD and CMU-MOSI), we conduct extensive experiments and prove that PMSAPO completely outperforms the Twelve state-of-the-art baseline models. (Code is available at https://github.com/SHX-AI/PMSAPO.)Description
Keywords
46 Information and Computing Sciences, 4603 Computer Vision and Multimedia Computation, 4605 Data Management and Data Science, 4608 Human-Centred Computing, 4611 Machine Learning, Networking and Information Technology R&D (NITRD), Machine Learning and Artificial Intelligence, 08 Information and Computing Sciences, Artificial Intelligence & Image Processing, 4602 Artificial intelligence, Multimodal sentiment analysis, Uncertain modalities missing, Pretraining, Online learning
Source
Knowledge-Based Systems, ISSN: 0950-7051 (Print), Elsevier BV, 114287-114287. doi: 10.1016/j.knosys.2025.114287
Publisher's version
Rights statement
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2025 Elsevier B.V. All rights are reserved, including those for text and data mining, AI training, and similar technologies.