2026-09-23T20:04:48Zhttps://openrepository.aut.ac.nz/server/oai/requestoai:openrepository.aut.ac.nz:10292/153242026-06-13T16:39:42Zcom_10292_3col_10292_5
AUT Research Repository
advisor
Ma, Jing
advisor
Lai, Edmund
author
Huang, Yubo
2022-07-28T04:39:28Z
2022-07-28T04:39:28Z
2022
https://hdl.handle.net/10292/15324
With the rapid development of information technology and the increasing scale of networks, the security, efficiency, and high-quality operation and maintenance of IT systems are areas of concern. In order to reduce the burden on human operators, AIOps (Artificial Intelligence for IT Operations), which attempts to combine artificial intelligence techniques with IT operation and maintenance, has emerged as a promising approach.
This thesis focuses on anomaly detection and analysis through computer logs. The aim is to construct an AIOps system model based on this study by deconstructing log data through the analysis of historical log data, clarifying the algorithm’s feasibility. The following objectives have been achieved to address the mentioned issues.
First, the state of the logging research field, operation and maintenance concepts and ideas on AIOps are analysed. It then analysed the characteristics from different system logs in AIOps scenarios and design a log detection framework, including collecting log data, decoding them, extracting them by LDA(Latent Dirichlet Allocation) topic model. Using “T-SNE"(t-distributed stochastic neighbor embedding) reduces the high-dimensional features to two-dimensional to observe the grouping effect. Three unsupervised algorithms, K-means, DBSCAN and LOF, were chosen to train models for log anomaly detection. They will select the optimal clusters, reduce redundant features and improve model performance. Numerical experiment results show K-means performs better in several tests and can delineate more finely and detect log anomalies earlier. Finally, an architecture diagram of the AIOps operating and management system was designed, and a preliminary requirements analysis of the AIOps system was conducted.
en
Log Data Anomaly Detection and Analysis for an AIOps System
Thesis
PHA+PGI+REVDTEFSQVRJT048L2I+PC9wPg0KDQo8cD5UaGlzIHRoZXNpcyBpcyBteSBvd24gd29yayBhbmQsIHRvIHRoZSBiZXN0IG9mIG15IGtub3dsZWRnZSBhbmQgYmVsaWVmLCBpdCBjb250YWluczo8L3A+DQogICA8dWw+DQoJPGxpPiBubyBtYXRlcmlhbCBwcmV2aW91c2x5IHB1Ymxpc2hlZCBvciB3cml0dGVuIGJ5IGFub3RoZXIgcGVyc29uLCBleGNlcHQgd2hlcmUgdGhpczxicj5pcyBleHBsaWNpdGx5IGRlZmluZWQgaW4gdGhlIGFja25vd2xlZGdlbWVudHM7PC9saT4NCiAgICAgICAgPGxpPm5vIG1hdGVyaWFsIHdoaWNoIHRvIGEgc3Vic3RhbnRpYWwgZXh0ZW50IGhhcyBiZWVuIGFjY2VwdGVkIGZvciB0aGUgcXVhbGlmaWNhdGlvbjxicj5vZiBhbnkgb3RoZXIgZGVncmVlIG9yIGRpcGxvbWEgb2YgYSB1bml2ZXJzaXR5IG9yIG90aGVyIGluc3RpdHV0aW9uPGJyPm9mIGhpZ2hlciBsZWFybmluZy48L2xpPg0KICAgPC91bD4NCjxwPjxiPlRoaXJkIFBhcnR5IENvcHlyaWdodCBTdGF0ZW1lbnQ8L3A+PC9iPg0KCTxwPkkgaGF2ZSBlaXRoZXIgdXNlZCBubyBzdWJzdGFudGlhbCBwb3J0aW9ucyBvZiB0aGlyZCBwYXJ0eSBjb3B5cmlnaHQgbWF0ZXJpYWwsIGluY2x1ZGluZzxicj5jaGFydHMsIGRpYWdyYW1zLCBncmFwaHMsIHBob3RvZ3JhcGhzIG9yIG1hcHMsIGluIG15IHRoZXNpcyBvciBJIGhhdmUgb2J0YWluZWQgcGVybWlzc2lvbjxicj5mb3Igc3VjaCBtYXRlcmlhbCB0byBiZSBtYWRlIGFjY2Vzc2libGUgd29ybGR3aWRlIHZpYSB0aGUgaW50ZXJuZXQuIElmIHBlcm1pc3Npb24gaGFzPGJyPm5vdCBiZWVuIG9idGFpbmVkLCBJIGhhdmUgcmVtb3ZlZCB0aGUgdGhpcmQgcGFydHkgY29weXJpZ2h0IG1hdGVyaWFsIGZyb20gdGhlIGRpZ2l0YWwgY29weS48L3A+
URL
https://openrepository.aut.ac.nz/bitstreams/42acd962-123c-4890-8c3b-f43fb5ff7ec4/download
File
MD5
99b44472afac647ff6f057f05e110ab5
15541867
application/pdf
Yubo_Huang_Thesis.pdf
URL
https://openrepository.aut.ac.nz/bitstreams/a56bff63-4f91-4130-a967-1f6f2cc8e592/download
File
MD5
6704c68c7d3fccc8a45a9b0f4b6c67c9
607
text/plain
Yubo_Huang_Thesis.pdf.txt