Measuring a model · AI Foundations

Why you need it

The intuition

Definition

Accuracy

The share of all predictions that were right. Misleading when one class is rare.

Precision

Of the cases predicted positive, the share that really were: TP / (TP + FP).

Recall

Of the real positive cases, the share the model found: TP / (TP + FN).

Overfitting

Learning the training data so closely, noise included, that the model does worse on new data.

Examples

1
2
3
4
5
6

actual = [0] * 95 + [1] * 5
always_no = [0] * 100
correct = sum(1 for a, p in zip(actual, always_no) if a == p)
print("accuracy:", correct / len(actual))
caught = sum(1 for a, p in zip(actual, always_no) if a == 1 and p == 1)
print("fraud caught:", caught, "of", actual.count(1))

accuracy: 0.95
fraud caught: 0 of 5

1
2
3
4
5
6
7
8
9
10

actual    = [1, 1, 1, 0, 0, 0, 0, 1, 0, 0]
predicted = [1, 0, 1, 1, 0, 0, 0, 1, 0, 1]
pairs = list(zip(actual, predicted))
tp = pairs.count((1, 1))
fp = pairs.count((0, 1))
fn = pairs.count((1, 0))
tn = pairs.count((0, 0))
print("TP", tp, "FP", fp, "FN", fn, "TN", tn)
print("precision", tp / (tp + fp))
print("recall", tp / (tp + fn))

TP 3 FP 2 FN 1 TN 4
precision 0.6
recall 0.75

1
2
3
4
5
6
7
8
9
10
11

train = {(1, 1): "red", (2, 1): "red", (8, 8): "blue", (9, 8): "blue"}
test = {(1, 2): "red", (8, 9): "blue"}

def memoriser(point):
    return train.get(point, "red")

def accuracy(model, data):
    return sum(model(p) == label for p, label in data.items()) / len(data)

print("train", accuracy(memoriser, train))
print("test", accuracy(memoriser, test))

train 1.0
test 0.5

Common mistakes

Quick quiz

1
2
3
4
5

actual    = [1, 0, 1, 1]
predicted = [1, 1, 0, 1]
tp = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1)
fp = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 1)
print(round(tp / (tp + fp), 2))

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

1. PredictCount the boxes

   Type exactly what this prints.

2. ModifyFix recall

   recall uses the wrong box. Fix it to TP / (TP + FN). When there are no real positives at all (TP + FN is 0), return 0.0.

3. WriteA full report

   Write report(actual, predicted) (lists of 0 and 1) that returns a dictionary with "precision", "recall" and "f1", each rounded to 3 decimals.

   • any measure whose bottom part is 0 is 0.0
   • F1 = 2 × P × R / (P + R), from the unrounded P and R

   report([1, 1, 1, 0, 0, 0, 0, 1, 0, 0], [1, 0, 1, 1, 0, 0, 0, 1, 0, 1])
   # {'precision': 0.6, 'recall': 0.75, 'f1': 0.667}

Terms in this lesson