Everyone Can Learn Program
中文 Start learning
Specialty · TestingLesson 4 of 5Free to read

Tables and properties · Software Testing

How do you test many cases without writing many tests?

Course contents

Why you need it

Some functions need dozens of cases: every price band, every date format. Writing a function for each is tiring and hides the pattern. Two techniques scale much better: tables of inputs and expected outputs, and properties that must hold for any input, checked on hundreds of random ones.

The intuition

Table-driven tests: one loop, many rows.

CASES = [(1, 5), (1.01, 9), (5, 9), (6, 11)]
for weight, want in CASES:
    assert shipping(weight) == want, (weight, shipping(weight))

In pytest: @pytest.mark.parametrize("weight, want", CASES), which reports each row as its own test.

Property-based tests: sometimes you can't easily say the exact answer, but you know rules it must follow. For any list, a sort function's output must:

  • have the same length,
  • contain the same items (same counts),
  • be in order.

Generate many random inputs, including awkward ones (empty, duplicates, negatives), and check the rules on each. Use a fixed seed (random.Random(1)) so a failure can be reproduced. The Hypothesis library does this for Python and even shrinks a failing input to the smallest one.

Definition

Table-driven test

One test that loops over rows of (input, expected).

Property-based test

A test that checks rules that must hold for any input, on many generated inputs.

Seed

The starting value of a random generator. The same seed gives the same "random" numbers, so failures can be repeated.

Examples

A table of cases

Try it yourself

Every boundary of the price bands in one list.

import math

def shipping(weight_kg):
    if weight_kg <= 1:
        return 5
    if weight_kg <= 5:
        return 9
    return 9 + 2 * math.ceil(weight_kg - 5)

CASES = [(0.5, 5), (1, 5), (1.01, 9), (5, 9), (5.5, 11), (7, 13)]
failures = 0
for weight, want in CASES:
    got = shipping(weight)
    if got != want:
        failures += 1
        print("FAIL", weight, "want", want, "got", got)
print(len(CASES), "cases,", failures, "failures")
Output
6 cases, 0 failures
Step by step
  1. L10Each row is one test: the boundaries 1 and 5, and just past them.
  2. L12One loop runs them all and reports each failure with its inputs.

Checking properties

Try it yourself

Run a sort on 300 random lists and check the three rules on each.

import random
from collections import Counter

def looks_sorted(fn):
    rng = random.Random(1)
    for _ in range(300):
        xs = [rng.randint(-3, 3) for _ in range(rng.randint(0, 6))]
        out = fn(list(xs))
        if len(out) != len(xs) or Counter(out) != Counter(xs):
            return False
        if any(out[i] > out[i + 1] for i in range(len(out) - 1)):
            return False
    return True

print(looks_sorted(sorted))
print(looks_sorted(lambda xs: sorted(set(xs))))
print(looks_sorted(lambda xs: sorted(xs, reverse=True)))
Output
True
False
False
Step by step
  1. L5A fixed seed: the same lists every run.
  2. L7A small range of values makes duplicates likely, and includes negatives.
  3. L9Same items with the same counts.
  4. L16Drops duplicates: caught by the count rule.

Common mistakes

Random tests you can't repeat

If a random test fails once and you can't get the same input again, you can't debug it. Use a seeded generator.

Re-implementing the code in the test

A test that computes the answer the same way as the code repeats the same bug. Use known values or independent properties.

Only nice random data

Random lists of distinct positive numbers never test duplicates, negatives or empty lists. Shape the generator to include them.

Quick quiz

1. Which is a property of a correct `reverse(xs)` that you can check on any list?

  1. Areverse(reverse(xs)) == xs
  2. Breverse(xs)[0] == 1
  3. Creverse(xs) != xs
Show the answer

Answer: A

Reversing twice gives back the original, for every list. (reverse(xs) == xs is fine for [1, 2, 1].)

2. Why seed the random generator in a property test?

  1. ASo a failing input can be reproduced
  2. BTo make the numbers more random
  3. CPython requires it
Show the answer

Answer: A

Same seed, same inputs: you can rerun the failing case and debug it.

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

  1. PredictRun the table

    Type exactly what this prints.

  2. ModifyKeep the order

    dedupe(items) should remove repeats and keep the first occurrence of each item in its original order. The property check dedupe(xs) keeps order fails, because a set doesn't keep order. Fix it.

  3. WriteA property checker

    Write check_sort(fn) that returns True if fn behaves like a correct sort on 200 random lists, and False as soon as one breaks a rule:

    • same length as the input
    • the same items with the same counts
    • in order

    The grader tries it on sorted and on broken sorts that drop duplicates, reverse, change negative numbers or sort only part of the list. Use random.Random(seed), and make lists that include negatives, duplicates, empty lists and lists longer than 6.

Practise this lesson

Open the interactive lesson to edit and run these examples and solve the exercises with instant feedback. Free to start, no sign-up.

Open the interactive lesson Course map

Interactive lessons open in order. If you have not done the earlier ones yet, you start at the course map.

Terms in this lesson

TermChineseDefinition
Table-driven test表格驱动测试Many (input, expected) rows checked in one loop.
Property-based test基于性质的测试Rules checked on many generated inputs.
HypothesisHypothesisA Python library for property-based testing.