Effects of Similarity Score Functions in Attention Mechanisms on the Performance of Neural Question Answering Systems
Loading...
Files
Size: 1.81 MB, File format: Adobe PDF
Date
Authors
Shen, Y
Lai, EM-K
Mohaghegh, M
Supervisor
Item type
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
Springer Science and Business Media LLC
Abstract
Attention mechanisms have been incorporated into many neural network-based natural language processing (NLP) models. They enhance the ability of these models to learn and reason with long input texts. A critical part of such mechanisms is the computation of attention similarity scores between two elements of the texts using a similarity score function. Given that these models have different architectures, it is difficult to comparatively evaluate the effectiveness of different similarity score functions. In this paper, we proposed a baseline model that captures the common components of recurrent neural network-based Question Answering (QA) systems found in the literature. By isolating the attention function, this baseline model allows us to study the effects of different similarity score functions on the performance of such systems. Experimental results show that a trilinear function produced the best results among the commonly used functions. Based on these insights, a new T-trilinear similarity function is proposed which achieved the higher predictive EM and F1 scores than these existing functions. A heatmap visualization of the attention score matrix explains why this T-trilinear function is effective.
Description
Keywords
Attention mechanism, Question answering, Deep learning, Natural language processing
Source
Neural Process Lett (2022). https://doi.org/10.1007/s11063-021-10730-4
Publisher's version
Rights statement
Creative Commons Attribution 4.0 International
© The Author(s) 2022. Open access.
© The Author(s) 2022. Open access.
Permanent link
Endorsement
Review
Supplemented By
Referenced By
Creative Commons license
Except where otherwise noted, this item's license is described as Creative Commons Attribution 4.0 International