Evaluate Models
Dummy Models
Manually assign the target values to the target while ignoring any feature. There are multiple ways: 1. Assign the most frequent 2. Choose the preferred result 3. Stratified prediction
It serves as the baseline to be compared with other normal models.
from sklearn.dummy import DummyClassifier
dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train)
y_dummy_predictions = dummy_majority.predict(X_test)
dummy_majority.score(X_test, y_test)
Confusion Matrix
Investigate the false positive and false negative rate.
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
classification_report(y_test, tree_predicted）
confusion = confusion_matrix(y_test, y_predicted)
Sensitivity/True Positive Rate/Recall: How many positive cases are identified?
TP/(TP+FN)
Precison: How many positive cases are correct?
TP/(TP+FP)
False Positive Rate/Specificity: What fraction of all negative cases are incorrectly identified as positive?
FP/(TN+FP)
A higher Sensitivity will inevitably lead to a lower Precision, vice versa.
F score can combine precision and recall into a single number.
Curves
Precision-Recall Curves:
X axis: Precision
Y axis: Recall
Top right corner is the ideal point (precision = 1, recall = 1)
ROC Curves
X axis: False Positive Rate
Y axis: True Positive Rate
Top left corner is the ideal point so that False positive rate is zero and true positive rate is one.
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
plt.figure()
plt.xlim([0.0, 1.01])
plt.ylim([0.0, 1.01])
plt.plot(precision, recall, label='Precision-Recall Curve')
plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3)
plt.xlabel('Precision', fontsize=16)
plt.ylabel('Recall', fontsize=16)
plt.axes().set_aspect('equal')
###############################################################
###############################################################
from sklearn.metrics import roc_curve, auc
X_train, X_test, y_train, y_test = train_test_split(X, y_binary_imbalanced, random_state=0)
y_score_lr = lr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)
plt.figure()
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])
plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
plt.axes().set_aspect('equal')
plt.show()
Multi-Class Evaluation
Confusion Matrix (one class case against all the other).
Macro Precision: Each class’s precison will be added together to achive the average, and there is no weight.
Micro Precision: The overall precision of the dataset.
Macro Precision is low. Examine small classes.
Micro Precision is low. Examine large classes.
Adjust the average variable to calculate macro or micro value.
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
precision_score(y_test_mc, svm_predicted_mc, average = 'micro')
precision_score(y_test_mc, svm_predicted_mc, average = 'macro')
GridSearch
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
## cross_val_score
cross_val_score(clf, X, y, cv=5,scoring = 'roc_auc')
## Grid Search
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import roc_auc_score
dataset = load_digits()
X, y = dataset.data, dataset.target == 1
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
clf = SVC(kernel='rbf')
grid_values = {'gamma': [0.001, 0.01, 0.05, 0.1, 1, 10, 100]}
# default metric to optimize over grid parameters: accuracy
grid_clf_acc = GridSearchCV(clf, param_grid = grid_values)
grid_clf_acc.fit(X_train, y_train)
y_decision_fn_scores_acc = grid_clf_acc.decision_function(X_test)
print('Grid best parameter (max. accuracy): ', grid_clf_acc.best_params_)
print('Grid best score (accuracy): ', grid_clf_acc.best_score_)
