Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition Under Uncertain Missing Modalities
Loading...
Files
Size: 3.99 MB, File format: Adobe PDF
Date
Authors
Shang, Ling
Liu, Zhizhong
Wu, Yuxuan
Song, Xiaoyu
Yu, Jian
Sheng, Quan Z
Supervisor
Item type
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
MDPI AG
Abstract
Currently, uncertain missing modalities pose new challenges to multimodal intent recognition. To tackle this issue, this work proposes an Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition model under Uncertain Missing Modalities (named ABGFMIR). Firstly, ABGFMIR extracts the features of each modality (text, audio, visual) with the LSTM network, respectively. Secondly, ABGFMIR narrows the distances between audio, visual and text modality based on Central Moment Discrepancy (CMD), and then performs multimodal feature fusion through an attention bidirectional gated fusion method. Then, corresponding attention level prompts are generated based on the uncertain missing modalities situations of the current sample. The fused multimodal features are then input into the Transformer encoder and decoder, and the prompts are injected into the keys and values of the multihead self-attention layer to guide the Transformer to focus on the missing modes and dynamically adjust the attention distribution, enhancing the robustness of ABGFMIR to different missing modes. Finally, the features produced by the Transformer are fed into the classification layer for intent recognition. Simultaneously, the pre-trained model (AGNN) that trained with the complete modality is employed in the classification layer to guide the main module of ABGFMIR. Two public benchmark datasets (MIntRec and EMOTyDA) are adopted for performance verification. Compared with the other five baseline models, on the MIntRec dataset, ABGFMIR improved accuracy by an average of 2.68 and improved F1 values by an average of 3.24. On the EMOTyDA dataset, ABGFMIR improved accuracy by an average of 2.28 and improved F1 values by an average of 3.44.
Description
Keywords
4605 Data Management and Data Science, 46 Information and Computing Sciences, 08 Information and Computing Sciences, 46 Information and computing sciences, multimodal intent recognition, uncertain missing modalities, transformer, LSTM, multimodal feature fusion
Source
Information, ISSN: 2078-2489 (Print); 2078-2489 (Online), MDPI AG, 17(8), 728-728. doi: 10.3390/info17080728
Publisher's version
Rights statement
Creative Commons Attribution (CC BY)
Permanent link
Endorsement
Review
Supplemented By
Referenced By
Creative Commons license
Except where otherwise noted, this item's license is described as Creative Commons Attribution (CC BY)