We address the problem of comparing the performance of classifiers. In this paper we study techniques for generating and evaluating confidence bands on ROC curves. Historically this has been done using one-dimensional confidence intervals by freezing one variable—the false-positive rate, or threshold on the classification scoring function. We adapt two prior methods and introduce a new radial s...

In this paper we review the Receiver Operating Characteristic ROC curve and the test statistic in relation to the analysis of a confusion matrix We then show how these two methods are related and propose an extension to the ROC curve so that it shows contours of values These contours can be used to provide further insight into the appropriate setting of the decision threshold for a particular a...

Receiver operating characteristic (ROC) curves are useful statistical tools for medical diagnostic testing. It has been proved its capability to assess diagnostic marker’s ability to distinguish between healthy and diseased subjects and to compare different diagnostic markers. In this paper we introduce non parametric ROC summary statistics to assess a ROC curve across the entire range of FPFs ...

Declaration This dissertation is submitted to the University of Bristol in accordance with the requirements of the degree of Bachelor of Science in the Faculty of Engineering. It has not been submitted for any other degree or diploma of any examining body. Except where specifically acknowledged, it is all the work of the Author. 3 ABSTRACT Machine Learning applications require learning algorith...

Introduction Classification models and in particular binary classification models are ubiquitous in many branches of science and business. Consider, for example, classification models in bioinformatics that classify catalytic protein structures as being in an active or inactive conformation. As an example from the field of medical informatics we might consider a classification model that, given...

In this paper we investigate methods to detect and repair concavities in ROC curves by manipulating model predictions. The basic idea is that, if a point or a set of points lies below the line spanned by two other points in ROC space, we can use this information to repair the concavity. This effectively builds a hybrid model combining the two better models with an inversion of the poorer models...

• This work overviews some developments on the estimation of the Receiver Operating Characteristic (ROC) curve. Estimation methods in this area are constantly being developed, adjusted and extended, and it is thus impossible to cover all topics and areas of application in a single paper. Here, we focus on some frequentist and Bayesian methods which have been mostly employed in the medical setti...

This paper is devoted to thoroughly investigating how to bootstrap the ROC curve, a widely used visual tool for evaluating the accuracy of test/scoring statistics in the bipartite setup. The issue of confidence bands for the ROC curve is considered and a resampling procedure based on a smooth version of the empirical distribution called the ”smoothed bootstrap” is introduced. Theoretical argume...

The accurate medical diagnostic of a disease condition is fundamental for a correct medical decision. Disease screening programs are based, in general, in diagnostic tests which provide a binary response: a subject is classified as positive, if the test result is above a given threshold, and negative, otherwise. Therefore, false positive and false negative classifications can be generated. The ...

The limitations of diagnostic "accuracy" as a measure of decision performance require introduction of the concepts of the "sensitivity" and "specificity" of a diagnostic test. These measures and the related indices, "true positive fraction" and "false positive fraction," are more meaningful than "accuracy," yet do not provide a unique description of diagnostic performance because they depend on...