Receiver Operating Characteristic

The Receiver Operating Characteristic ( ROC) - curve or limit curve optimization is a method for evaluating and optimizing analysis strategies. The ROC curve is a function of the visual efficiency of the error rate for different parameter values ​​of the application and consists of a signal detection theory.

It can be used to find the best possible value of a parameter, such as a dichotomous (semi-) quantitative trait or two-class classification problem.

Calculation of the ROC curve

For any parameter value (eg, transmission speed, frequency, ... ) we determine the resulting relative frequency distributions in the form of sensitivity (true positive rate ) and false-positive rate. In a chart, you wear sensitivity (true positive rate ) as the ordinate and false positive rate as the abscissa. The parameter value itself does not appear here, but can be used as a labeling of the points. It typically results in a curved, ascending curve.

Interpretation of the ROC curve

A ROC curve close to the diagonal line indicates a random process: values ​​close to the diagonal represent a same hit rate and false positive rate, which corresponds to the expected hit frequency of a random process. The ideal ROC curve first rises vertically (the hit rate is close to 100%, while the error rate initially remains close to 0% ), and only then the false positive rate increases. A ROC curve, which remains well below the diagonal, indicates that the values ​​were interpreted incorrectly. Instead, a signal can be seen, noise is detected and the signal is filtered.

Application as an optimization method

The theoretical optimum (in the sense of a compromise from hit and error rate) of the tested value is then determined visually from the contact point of a 45 ° rising tangent to the ROC curve, provided that the axes are scaled uniformly. Otherwise, the increase must be tangent to the diagonal equal.

Draw the test values ​​(for example, a function of the FP rate) in the same diagram, there is the limit as a solder the contact point of the tangent to the curve of test values ​​. Alternatively, the points of the curve can be labeled with the test value. Mathematically, look for the test value with the highest Youden index. This is calculated from (calculated with relative values ​​).

An alternative method, which finds application especially in information retrieval, is the consideration of recall and precision.

Application as a quality

An ROC curve can be used as a quality measure. This is often in the field of information retrieval the case. In order to evaluate regardless of the test value, the ROC curve for all or a sample of test values ​​is calculated.

At the ROC curve to calculate the area under the curve ( "Area under curve ", ROC AUC). This value can be between 0 and 1, but 0.5 is the worst value. As described above, a ROC curve is close to the expected result of the diagonals of a random process that has an area of 0.5. The previously described as optimal curve has an area of ​​between 0.5 and 1, the curve with the surface of less than 0.5 can be used in information theory, but ultimately be just as good if you are the result reversed accordingly interpreted ( " positive" and " negative" reversed).

The key advantage of using the ROC - AUC compared to for example the pure misclassification rate is that here the parameter value is omitted, while the latter can always be calculated only for a single specific parameter value. A high ROC AUC clearly means " for a suitable choice of the parameter, the result is good."

Example

In information retrieval can be evaluated here, for example, the quality of a search result. "Positive" is this a suitable search result, "negative" an unsuitable. The assay value is the number of the requested results. If the database contains 10 relevant and 90 irrelevant documents, and a method has been found in the first 12 results relevant 7, the ROC curve passes through the point. This is calculated for all possible numbers of results ( 0-100).

The problem as an optimization problem would be: " What is the optimal number of results should I consider? "

The problem as a quality measure would be: " Regardless of how many results I want to get, how well is the search function? "

Of course, in this example, both questions are of limited use.

Intuition in machine learning

In machine learning, ROC curves are used to evaluate the Klassifikatorperformance. The misclassification rate for a growing set of instances is determined, from the instances for which the classifier is safest (because, for example, the greatest distance to the separating function of a support vector machine have ).

By way of example, one can imagine an examiner who can first answer the questions where this feels safest to the DUT. During the test can create a ROC curve of the examiners. Good specimens give only then to the end of the test wrong answers, which can be easily read from the ROC curve.