Show simple item record

dc.contributor.advisorRussel, Pears
dc.contributor.authorGreen, Brian
dc.date.accessioned2015-11-13T06:27:29Z
dc.date.available2015-11-13T06:27:29Z
dc.date.copyright2015
dc.date.created2015
dc.identifier.urihttp://hdl.handle.net/10292/9214
dc.description.abstractLog files play an important part in the day to day running of many systems and services, allowing administrators and other users to gain insights into operational, performance or even security issues but it is now impractical with the volume of files today to manually examine them. Existing tools in this space largely work by detecting anomalies from log files that have already been stored or by comparing them against known errors (signatures). By data mining log file streams for the detection of anomalies instead, it will allow administrators to reduce the time required to detect anomalies significantly with no signatures or complex settings needing to be maintained. This paper presents the experimental work undertaken to define a generic, practical and scalable method for anomaly detection in streaming log files by detecting the change to the mix of log events occurring. This was achieved by following a modified CRISP-DM (Cross Industry Standard Process for Data Mining) methodology enabling a broader more flexible approach to the data mining process. By taking this approach, a solution was developed that employs common log file features together with a weighted earth mover distance metric. This enabled a framework to be developed that can be broadly applied to many log file types. By setting a simple percentile threshold indicating an acceptable level of change, anomaly detection in streaming log files can be achieved.en_NZ
dc.language.isoenen_NZ
dc.publisherAuckland University of Technology
dc.subjectData miningen_NZ
dc.subjectLogsen_NZ
dc.subjectStreamingen_NZ
dc.subjectAnomaliesen_NZ
dc.subjectExpermentalen_NZ
dc.titleData mining log file streams for the detection of anomaliesen_NZ
dc.typeThesis
thesis.degree.grantorAuckland University of Technology
thesis.degree.grantorAuckland University of Technology
thesis.degree.levelMasters Theses
thesis.degree.nameMaster of Computer and Information Sciencesen_NZ
thesis.degree.discipline
dc.rights.accessrightsOpenAccess
dc.date.updated2015-11-13T03:59:22Z


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record