Anomaly Detection in Text Data Sets Using Character-Level Representation

Mohaghegh, Mahsa; Abdurakhmanov, Amantay

Anomaly Detection in Text Data Sets Using Character-Level Representation

Files

Journal article(610.87 KB)

Date

2021-04-28

Authors

Mohaghegh, Mahsa

Abdurakhmanov, Amantay

Item type

Journal Article

Publisher

Institute of Physics (IoP)

Abstract

This paper proposes a character-level representation of unsupervised text data sets for anomaly detection problems. An empirical examination of the character-level text representation was conducted to demonstrate the ability to separate outlying and normal records using an ensemble of multiple classic numerical anomaly classifiers. Experimental results obtained on two different data sets confirmed the applicability of the developed unsupervised model to detect outlying instances in various real-world scenarios, providing the opportunity to quickly assess a large amount of textual data in terms of information consistency and conformity without knowledge of the data content itself.