What accuracy statistics really measure
Files
Date
Supervisor
Item type
Journal Article
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
IEEE
Abstract
Provides the software estimation research community with a better understanding of the meaning of, and relationship between, two statistics that are often used to assess the accuracy of predictive models: the mean magnitude relative error (MMRE) and the number of predictions within 25% of the actual, pred(25). It is demonstrated that MMRE and pred(25) are, respectively, measures of the spread and the kurtosis of the variable z, where z=estimate/actual. Thus, z is considered to be a measure of accuracy, and statistics such as MMRE and pred(25) to be measures of properties of the distribution of z. It is suggested that measures of the central location and skewness of z, as well as measures of spread and kurtosis, are necessary. Furthermore, since the distribution of z is non-normal, non-parametric measures of these properties may be needed. For this reason, box-plots of z are useful alternatives to simple summary metrics. It is also noted that the simple residuals are better behaved than the z variable, and could also be used as the basis for comparing prediction systemsDescription
Keywords
Accuracy measures, Accuracy statistics, Box-plots, Central location measure, Kurtosis measure, Mean magnitude relative error, Nonnormal distribution, Nonparametric measures, Prediction number, Prediction systems comparison, Predictive models, Residuals, Skewness measure, Software estimation, Spread measure, Statistical distribution properties, Summary metrics
Source
IEE Proceedings: Software, vol.148(3), pp.81-85
Publisher's version
Rights statement
Copyright © 2001 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.