Testing hierarchy

Updated Aug 23, 2026

Pick the first level whose evidence matches the question:

  1. Named pytest example: one scenario, regression, exact output, or error contract matters.
  2. Parameterized pytest: a finite truth table, standards corpus, or set of cases whose rows each carry semantic meaning.
  3. Lightweight Hypothesis: one round trip, invariant, oracle, or metamorphic relation should hold across many cheap, repeatable inputs. A growing set of representative parameter rows is the usual signal.
  4. Structured or stateful Hypothesis: valid data has dependent or recursive structure, or failures depend on operation history.
  5. CrossHair: a small pure function needs bounded symbolic exploration of contracts or changed behaviour.

Mutation testing sits beside the hierarchy. Use mutmut when the question is whether the current suite would notice a plausible defect.

Leave the hierarchy for integration, scheduling, load, performance, resource, or native-code failures. Those need real or simulated boundaries, concurrency or stress tools, benchmarks and profilers, or native sanitizers.

Examples remain useful beside generated tests. Keep exact protocol examples and named regressions even when a property searches the wider domain.

Run lightweight Hypothesis properties on every CI run; run targeted CrossHair and mutmut on slower cadences.