Assessing the Performance of Classification Methods

A large number of measures have been developed for evaluating the performance of classification rules. Some of these have been developed to meet the practical requirements of specific applications, but many others—which here we call “classification accuracy” criteria—represent different ways of balancing the different kinds of misclassification which may be made. This paper reviews classification accuracy criteria. However, the literature is now so large and diverse that a comprehensive list, covering all the measures and their variants, would probably be impossible. Instead, this paper embeds such measures in general framework, spanning the possibilities, and draws attention to relationships between them. Important points to note are, firstly, that different performance measures, by definition, measure different aspects of performance; secondly, that one should therefore carefully choose a measure to match the objectives of one's study; and, thirdly, that empirical comparisons between instruments measuring different aspects are of limited value.