Property Testing

Observing actions: result.actions

The framework already proposes three result views: result.manifest, result.graph, and result.ninja. RFC 0012 adds a fourth, available after build_graph, built from the IR with the recipe shell pinned so identical manifests project identically on every host. The view is additive and exposes no internal IR types. Planned under roadmap 10.1.

Two quantifiers join the then vocabulary: for_all_actions evaluates each listed expression once per action, binding action; for_all_targets does the same over the graph view, binding target. A failure reports the binding that falsified it with the substituted actual values. Assertions read env in full; every copy that leaves the runner — failure reports, diagnostics, rendered views, regression files — carries opaque key tokens and a fixed <redacted> marker instead. Redaction happens only on the way out, so it can never turn a FAIL into an ERROR.

Bounded generation: the forall block

A case may declare forall, a mapping from binding names to domain constructors. The runner expands the domains into a finite family and runs the case's steps once per drawn tuple; drawn values are available wherever let bindings are. The case below draws a target name and a flag ordering, builds the graph, and asserts that no action's environment strays beyond the declared variables and no action lists a source twice. Planned under roadmap 10.2.

netsuke_test_version: "1.1"

test_command_env_is_closed_over_declared_vars:
  description: Generated invocations never leak ambient environment.
  forall:
    target_name: { path_fragment: { max_len: 12 } }
    flag_order: { permutations_of: ["-O2", "-g", "-Wall"] }
  steps:
    - given:
        let:
          declared_env: []
          name: "{{ target_name }}"
          flags: "{{ flag_order }}"
    - when: build_graph
    - then:
        for_all_actions:
          - "action.env | keys | reject('in', declared_env) | length == 0"
          - "action.inputs | unique | length == action.inputs | length"

When the Cartesian product of the declared domains fits under the expansion ceiling, every tuple is checked. Above it, the runner samples with a generator seeded with the fixed default seed 0, so the default run is deterministic. The ceiling, the sample count, and a per-case deadline are enforced limits. Internally the runner reuses Proptest's generator and regression-file conventions; no Proptest type appears in the dialect.

Metamorphic mutations

The most valuable properties of a build-script generator compare two runs rather than inspecting one. These need no predicates from the author, so they ship as a closed mutations vocabulary: the runner derives a mutated manifest, runs the plan-mode pipeline on both, and asserts the named relation. The case below asserts that the generated script ignores declaration order and that adding an unrelated target leaves existing actions alone. Planned under roadmap 10.3.

netsuke_test_version: "1.1"

test_generation_is_order_invariant:
  description: Declaration order never changes the generated script.
  subject:
    manifest: fixtures/webapp/Netsukefile
  mutations:
    - permute_declarations
    - insert_irrelevant_target

Between them the four cover determinism, order invariance, locality, and rename isomorphism. Mutations compose with forall: each drawn tuple runs the base and the mutated pipeline. Both run inside the framework's killable child process, so a mutant that diverges or never terminates cannot wedge the runner.

Coverage and replay

$ netsuke test --seed <n>
$ netsuke test --strict-coverage

Everything else on the command line — filters, --tag, --timeout, --json, the exit codes — is the framework's, described under the command in its preview. The RFC does not specify how the seed, tuple, or falsifying binding appear in JSON output, and does not show a rendered property failure, so neither is sketched here.

From the shallow end

Property testing is opt-in by dialect version. Existing netsuke_test_version: "1.0" files parse and run unchanged; only a file that uses forall, mutations, or result.actions declares "1.1". The shallow end is therefore the framework's own quick start: one file beside the manifest, one case, plain netsuke test.

netsuke_test_version: "1.0"

test_hello_target_is_generated:
  steps:
    - when: generate_ninja
      then:
        - result.ok
        - result.graph.has_target("build/hello.txt")

1. 1. A 1.0 test file

   Fixed fixtures and assertions over the manifest, graph, and Ninja views. No properties, no version bump. See the testing framework's shallow end.

2. 2. mutations, with no expressions

   Declare "1.1", point a case at an existing manifest, and list permute_declarations. The relation is built in, so the first property needs no predicate at all — the RFC's example is five lines of intent.

3. 3. forall over small finite sets

   one_of, permutations_of, and small int ranges enumerate exhaustively under the ceiling, so every combination is checked rather than sampled, with for_all_actions assertions over each. Sampled domains such as path_fragment and list_of come after, with seeds and replay.

4. 4. Read the coverage lint

   The advisory report names each rule whose command or environment no property covers. It is the signpost to the deep end: a list of the next properties worth writing.

5. 5. Gate on --strict-coverage

   Once the list is empty, make it stay empty: an uncovered rule fails the run, so a new invocation shape cannot land without a property.

No tutorial exists yet. Roadmap 10.3.4 plans the users' guide testing chapter for the property dialect, after the dialect has been dogfooded on Netsuke's own example manifests.

Confidence

Every guarantee below is designed, not shipped. The phase's own success criterion is that the RFC's worked examples run green deterministically on a machine with no compiler and no network, and that a seeded failure replays identically from its reported seed and tuple inputs.

Accessibility & localization

The RFC itself puts a plain-language summary before each YAML example for screen readers, which this page follows. It is silent on how a falsifying binding or a minimized tuple should read under --accessibility on; roadmap 7.6.3 plans to document how the framework's results and diagnostics render in that mode, and property reports would need the same treatment.