Show simple item record

dc.contributor.authorShen, Yen_NZ
dc.contributor.authorLai, EM-Ken_NZ
dc.contributor.authorMohaghegh, Men_NZ
dc.date.accessioned2022-02-20T22:58:23Z
dc.date.available2022-02-20T22:58:23Z
dc.identifier.citationNeural Process Lett (2022). https://doi.org/10.1007/s11063-021-10730-4
dc.identifier.issn1370-4621en_NZ
dc.identifier.issn1573-773Xen_NZ
dc.identifier.urihttp://hdl.handle.net/10292/14925
dc.description.abstractAttention mechanisms have been incorporated into many neural network-based natural language processing (NLP) models. They enhance the ability of these models to learn and reason with long input texts. A critical part of such mechanisms is the computation of attention similarity scores between two elements of the texts using a similarity score function. Given that these models have different architectures, it is difficult to comparatively evaluate the effectiveness of different similarity score functions. In this paper, we proposed a baseline model that captures the common components of recurrent neural network-based Question Answering (QA) systems found in the literature. By isolating the attention function, this baseline model allows us to study the effects of different similarity score functions on the performance of such systems. Experimental results show that a trilinear function produced the best results among the commonly used functions. Based on these insights, a new T-trilinear similarity function is proposed which achieved the higher predictive EM and F1 scores than these existing functions. A heatmap visualization of the attention score matrix explains why this T-trilinear function is effective.en_NZ
dc.languageenen_NZ
dc.publisherSpringer Science and Business Media LLCen_NZ
dc.relation.urihttps://link.springer.com/article/10.1007/s11063-021-10730-4
dc.rights© The Author(s) 2022. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/
dc.subjectAttention mechanism; Question answering; Deep learning; Natural language processing
dc.titleEffects of Similarity Score Functions in Attention Mechanisms on the Performance of Neural Question Answering Systemsen_NZ
dc.typeJournal Article
dc.rights.accessrightsOpenAccessen_NZ
dc.identifier.doi10.1007/s11063-021-10730-4en_NZ
pubs.elements-id448963
aut.relation.journalNeural Processing Lettersen_NZ


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record