Performance Analysis of Classification Models

 In a machine learning Algorithm, once the model is built, the next step is the use of various performance criteria to evaluate Machine learning Models.

In the Classification model output is a discrete value therefore for classification performance analysis following metrics are used  

1. Confusion matrix
2. Accuracy
3. Precision
4. Recall (sensitivity)
5. Specificity
6. ROC curve (AUC) ROC Area Under Curve is useful when we are not concerned about whether the small dataset/class of dataset is positive or not, in contrast to the F1 score where the class being positive is important.
7. F-score(F1 score is useful when the size of the positive class is relatively small)

Performance metrics should be chosen based on the problem domain, project goals, and objectives. 

A confusion matrix

A confusion matrix is a table that is used to describe the performance of an algorithm (or "classifier") on a set of test data for which the true values/targets are known.
For binary classifier, the confusion matrix is as shown below: 

 

    Figure1: Confusion Matrix

Table 1: Confusion Matrix Values

 

Total Samples=1000	Predicted Yes	Predicted No
Actual Yes	TP(800)	FN(40)
Actual No	FP(60)	TN(100)

 

         Is the Machine Learning model used to predict patient is diabetic or not?

From the Above two tables:  è "Yes": means He is Diabetic.

                            è  "no": means He is not a Diabetic.

Suppose total number of patients =1000(Total Samples)

Out of 1000samples

 800 patients are diabetic

100 are Diabetic

100 are the wrong result

 

• Correctly Predicted 900
• Wrongly Predicted 100
• Basic terms of confusion Matrix

Predicted correctly: The diagonal element gives the correct prediction 

True positives (TP): These are cases in which the model predicted yes (they have the disease), and they do have diabetic

True negatives (TN): The model predicted no, and they don't have diabetic 

Wrong Prediction.

False positives (FP): The model predicted yes, but they don't actually have the diabetic (Also known as a "Type I error.")

False negatives (FN): The model predicted no, but they actually do have diabetes (Also known as a "Type II error.")

 

For a balanced type of data, Accuracy is used for performance analysis.

Balance Data: In binary classification equal number of samples belongs to class I and class II.

• Accuracy

Accuracy=(TP+TN)/(TP+FP+FN+TN) = (800+100)/1000 = 0.9

(90% of Samples are correctly Classified) 

Error Rate=(FP+FN)/TP+TN+FN+FP = (40+60)/1000 = 0.1 

Accuracy=1- Error Rate

• Recall / Sensitivity/TPR

·         Recall=Sensitivity:=TPR=TP/TP+FN(Actual Positive)

Sensitivity=TP/actual yes = 800/840= 0.95 

TPR=True Positive Rate

·         False Positive Rate: Type one error 

FP/FP+TN= 60/ (60+100) 

Specificity measures the proportion of actual negative instances that are correctly identified by the model as negative. It represents the ability of the model to correctly identify negative instances

   FPR=1-Specificity

·         True Negative Rate: 

TN/TN+FP=100/100+60 

• Precision

      Pecision=TP/TP+FP(predicted yes)= 800/800+60

• F1 Score: The F1-score is a harmonic mean of Precision and Recall, and so it gives a combined idea about these two metrics. It is maximum when Precision is equal to Recall. This is a weighted average of the true positive rate (recall) and precision.  

F1Score=2*precision*recall/(Precision+Recall) 

b=1 precision and recall both are important FP and FN Both have an impact 

If FPR is imp then b=0.5 between 1 and 0

FNR  is imp then b=2 

•  AUC-ROC

    The Receiver Operator Characteristic (ROC) curve is an evaluation metric for binary classification problems. It is a probability curve that plots the TPR against FPR at various threshold values. The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve.

    In a ROC curve, a higher X-axis value indicates a higher number of False positives than     True negatives. While a higher Y-axis value indicates a higher number of True positives than False negatives. So, the choice of the threshold depends on the ability to balance between False positives and False negatives.

The Receiver Operator Characteristic (ROC) curve is an evaluation metric for binary classification problems.

 It is a probability curve that plots the TPR against FPR at various threshold values The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve.

The higher the AUC, the better the performance of the model at distinguishing between the positive and negative classes.

 

When AUC = 1, then the classifier can perfectly distinguish between all the Positive and the Negative class points correctly. If, however, the AUC had been 0, then the classifier would be predicting all Negatives as Positives and all Positives as Negatives.

When 0.5<AUC<1, there is a high chance that the classifier will be able to distinguish the positive class values from the negative class values. This is so because the classifier can detect more numbers of True positives and True negatives than False negatives and False positives.