Log Data Anomaly Detection and Analysis for an AIOps System
| aut.embargo | Yes | en_NZ |
| aut.filerelease.date | 2023-01-28 | |
| aut.thirdpc.contains | No | en_NZ |
| dc.contributor.advisor | Ma, Jing | |
| dc.contributor.advisor | Lai, Edmund | |
| dc.contributor.author | Huang, Yubo | |
| dc.date.accessioned | 2022-07-28T04:39:28Z | |
| dc.date.available | 2022-07-28T04:39:28Z | |
| dc.date.copyright | 2022 | |
| dc.date.issued | 2022 | |
| dc.date.updated | 2022-07-20T14:00:39Z | |
| dc.description.abstract | With the rapid development of information technology and the increasing scale of networks, the security, efficiency, and high-quality operation and maintenance of IT systems are areas of concern. In order to reduce the burden on human operators, AIOps (Artificial Intelligence for IT Operations), which attempts to combine artificial intelligence techniques with IT operation and maintenance, has emerged as a promising approach. This thesis focuses on anomaly detection and analysis through computer logs. The aim is to construct an AIOps system model based on this study by deconstructing log data through the analysis of historical log data, clarifying the algorithm’s feasibility. The following objectives have been achieved to address the mentioned issues. First, the state of the logging research field, operation and maintenance concepts and ideas on AIOps are analysed. It then analysed the characteristics from different system logs in AIOps scenarios and design a log detection framework, including collecting log data, decoding them, extracting them by LDA(Latent Dirichlet Allocation) topic model. Using “T-SNE"(t-distributed stochastic neighbor embedding) reduces the high-dimensional features to two-dimensional to observe the grouping effect. Three unsupervised algorithms, K-means, DBSCAN and LOF, were chosen to train models for log anomaly detection. They will select the optimal clusters, reduce redundant features and improve model performance. Numerical experiment results show K-means performs better in several tests and can delineate more finely and detect log anomalies earlier. Finally, an architecture diagram of the AIOps operating and management system was designed, and a preliminary requirements analysis of the AIOps system was conducted. | en_NZ |
| dc.identifier.uri | https://hdl.handle.net/10292/15324 | |
| dc.language.iso | en | en_NZ |
| dc.publisher | Auckland University of Technology | |
| dc.rights.accessrights | OpenAccess | |
| dc.title | Log Data Anomaly Detection and Analysis for an AIOps System | en_NZ |
| dc.type | Thesis | en_NZ |
| thesis.degree.grantor | Auckland University of Technology | |
| thesis.degree.level | Masters Theses | |
| thesis.degree.name | Master of Computer and Information Sciences | en_NZ |