Effects of Similarity Score Functions in Attention Mechanisms on the Performance of Neural Question Answering Systems
| aut.relation.journal | Neural Processing Letters | en_NZ |
| aut.researcher | Lai, Edmund | |
| dc.contributor.author | Shen, Y | en_NZ |
| dc.contributor.author | Lai, EM-K | en_NZ |
| dc.contributor.author | Mohaghegh, M | en_NZ |
| dc.date.accessioned | 2022-02-20T22:58:23Z | |
| dc.date.available | 2022-02-20T22:58:23Z | |
| dc.description.abstract | Attention mechanisms have been incorporated into many neural network-based natural language processing (NLP) models. They enhance the ability of these models to learn and reason with long input texts. A critical part of such mechanisms is the computation of attention similarity scores between two elements of the texts using a similarity score function. Given that these models have different architectures, it is difficult to comparatively evaluate the effectiveness of different similarity score functions. In this paper, we proposed a baseline model that captures the common components of recurrent neural network-based Question Answering (QA) systems found in the literature. By isolating the attention function, this baseline model allows us to study the effects of different similarity score functions on the performance of such systems. Experimental results show that a trilinear function produced the best results among the commonly used functions. Based on these insights, a new T-trilinear similarity function is proposed which achieved the higher predictive EM and F1 scores than these existing functions. A heatmap visualization of the attention score matrix explains why this T-trilinear function is effective. | en_NZ |
| dc.identifier.citation | Neural Process Lett (2022). https://doi.org/10.1007/s11063-021-10730-4 | |
| dc.identifier.doi | 10.1007/s11063-021-10730-4 | en_NZ |
| dc.identifier.issn | 1370-4621 | en_NZ |
| dc.identifier.issn | 1573-773X | en_NZ |
| dc.identifier.uri | https://hdl.handle.net/10292/14925 | |
| dc.language | en | en_NZ |
| dc.publisher | Springer Science and Business Media LLC | en_NZ |
| dc.relation.uri | https://link.springer.com/article/10.1007/s11063-021-10730-4 | |
| dc.rights | © The Author(s) 2022. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ | |
| dc.rights.accessrights | OpenAccess | en_NZ |
| dc.subject | Attention mechanism; Question answering; Deep learning; Natural language processing | |
| dc.title | Effects of Similarity Score Functions in Attention Mechanisms on the Performance of Neural Question Answering Systems | en_NZ |
| dc.type | Journal Article | |
| pubs.elements-id | 448963 | |
| pubs.organisational-data | /AUT | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies/Faculty Central | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies/School of Engineering, Computer & Mathematical Sciences | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies/School of Engineering, Computer & Mathematical Sciences/Centre for Artificial Intelligence Research | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies/School of Engineering, Computer & Mathematical Sciences/Centre for Robotics & Vision | |
| pubs.organisational-data | /AUT/Faculty of Design & Creative Technologies/School of Engineering, Computer & Mathematical Sciences/Science, Technology, Engineering, & Mathematics Tertiary Education Centre | |
| pubs.organisational-data | /AUT/PBRF | |
| pubs.organisational-data | /AUT/PBRF/PBRF Design and Creative Technologies | |
| pubs.organisational-data | /AUT/PBRF/PBRF Design and Creative Technologies/PBRF ECMS |