Repository logo
 

Log Data Anomaly Detection and Analysis for an AIOps System

Files

Thesis(14.82 MB)

Date

2022

Authors

Supervisor

Ma, Jing
Lai, Edmund

Item type

Thesis

Degree name

Master of Computer and Information Sciences

Journal Title

Journal ISSN

Volume Title

Publisher

Auckland University of Technology

Abstract

With the rapid development of information technology and the increasing scale of networks, the security, efficiency, and high-quality operation and maintenance of IT systems are areas of concern. In order to reduce the burden on human operators, AIOps (Artificial Intelligence for IT Operations), which attempts to combine artificial intelligence techniques with IT operation and maintenance, has emerged as a promising approach. This thesis focuses on anomaly detection and analysis through computer logs. The aim is to construct an AIOps system model based on this study by deconstructing log data through the analysis of historical log data, clarifying the algorithm’s feasibility. The following objectives have been achieved to address the mentioned issues. First, the state of the logging research field, operation and maintenance concepts and ideas on AIOps are analysed. It then analysed the characteristics from different system logs in AIOps scenarios and design a log detection framework, including collecting log data, decoding them, extracting them by LDA(Latent Dirichlet Allocation) topic model. Using “T-SNE"(t-distributed stochastic neighbor embedding) reduces the high-dimensional features to two-dimensional to observe the grouping effect. Three unsupervised algorithms, K-means, DBSCAN and LOF, were chosen to train models for log anomaly detection. They will select the optimal clusters, reduce redundant features and improve model performance. Numerical experiment results show K-means performs better in several tests and can delineate more finely and detect log anomalies earlier. Finally, an architecture diagram of the AIOps operating and management system was designed, and a preliminary requirements analysis of the AIOps system was conducted.

Description

Keywords

Source

DOI

Publisher's version

Rights statement

Permanent link

https://hdl.handle.net/10292/15324

Collections

Masters Theses