Netsukefile Testing Framework

Test tree and files

Tests live in a netsuke-tests directory beside the Netsukefile. An optional top-level tests block in the manifest reconfigures discovery; it is tool configuration, not build data, so its values are invisible to templates and tests.root accepts no Jinja. Files whose name starts with _ are support files, loaded only when imported. A run that discovers no files, or whose filters select zero cases, fails rather than reporting a silent green.

netsuke_version: "1.2.0"

tests:
  root: netsuke-tests
  include:
    - "**/*.yml"
    - "**/*.yaml"
  exclude:
    - "**/_*.yml"
    - "**/_*.yaml"

netsuke_test_version: "1.0"   # required in every file

imports:                       # optional support files,
  - _fixtures.yml              # confined to the test tree

vars: {}                       # suite-local test variables

macros: []                     # same shape as Netsukefile macros

fixtures: {}                   # fixture definitions

test_compile_target:           # one or more cases
  steps: []

netsuke_test_version is a MAJOR.MINOR string. The dialect denies unknown keys everywhere, so every addition is a minor-version event and an older runner rejects a newer file rather than misreading it; an unknown key is reported with the nearest known key. Case names match test_[A-Za-z0-9_]+ and are reported as <file>::<name>.

Given, when, then

A case is an ordered list of steps. Each step carries at least one of given (fixtures, environment, doubles, clock), when (a pipeline action), and then (assertions). Context from given persists across the later steps of the same case; nothing persists between cases. The smallest useful test needs none of the machinery:

netsuke_test_version: "1.0"

test_hello_target_is_generated:
  steps:
    - when: generate_ninja
      then:
        - result.ok
        - result.graph.has_target("build/hello.txt")

then:
  - result.ok
  - result.graph.targets | length == 3
  - equals:
      actual: result.graph.edge_count
      expected: 4
  - contains:
      value: result.ninja
      needle: "default app"
  - matches:
      value: result.error.message
      regex: "missing.*rule"

Doubles and fixtures

given.let binds a double to a template function name. The three kinds follow the stub, mock, and spy taxonomy: a stub returns a canned value and is journalled but never verified; a mock is verified, so every declared expectation must be met and an unmatched call fails the action immediately; a spy journals every call and passes through to the effective implementation under test configuration. The terse form is deliberately the loose one, so don't-care collaborators cost one line and full expectation machinery is opt-in.

given:
  env:
    set:
      CC: clang
  let:
    glob: mock(args=["src/*.c"], returns=["src/main.c"])
    cc:   stub(returns="clang")
    now:  spy()
    compile_cmd: substitute("stand_in_compile")

Fixtures

Fixtures are lifecycle objects: ordered setup actions, exports, and ordered teardown, with an action vocabulary of tmpdir, mkdir, write, copy, remove, and env. Every path resolves inside a per-case sandbox; a fixture cannot touch the project root or the host filesystem. Teardown runs in reverse order for every fixture whose setup completed, regardless of later failures, and a failing teardown never masks the case result. All fixtures are case-scoped in the first version; an arbitrary-command run action is deferred with execution generally.

fixtures:
  tiny_c_project:
    description: Minimal C project with one source and a Netsukefile.
    setup:
      - tmpdir: project
      - mkdir: "{{ project }}/src"
      - write:
          path: "{{ project }}/src/main.c"
          text: |
            int main(void) { return 0; }
      - write:
          path: "{{ project }}/Netsukefile"
          text: |
            netsuke_version: "1.2.0"
            targets:
              - name: build/main.o
                command: "cc -c src/main.c -o build/main.o"
                sources: src/main.c
    exports:
      root: "{{ project }}"
      manifest: "{{ project }}/Netsukefile"

The command

netsuke test [FILTER...]

Arguments:
  FILTER                   Case selectors: file paths, file::case names, or
                           substring patterns

Options:
  --tests-dir <DIR>        Override tests.root
  --list                   List discovered cases without running them
  --tag <TAG>              Run only cases with this tag (repeatable)
  --skip-tag <TAG>         Exclude cases with this tag (repeatable)
  --fail-fast              Stop after the first failing case
  --timeout <SECS>         Per-case wall-clock budget (default 60)
  --keep                   Preserve sandboxes of failing cases
  --allow-empty            Succeed when zero cases are selected

Worked example

The case below verifies foreach expansion, when filtering, environment-driven command construction, and macro wiring — without a compiler installed, without touching the real filesystem, and identically on every machine. It is the design's own success criterion.

netsuke_version: "1.2.0"

tests:
  root: netsuke-tests

macros:
  - signature: "compile_cmd(src, obj)"
    body: |
      {{ env('CC') }} -c {{ src }} -o {{ obj }}

targets:
  - foreach: glob('src/*.c')
    when: item | basename != 'skip.c'
    name: "build/{{ item | basename | with_suffix('.o') }}"
    command: "{{ compile_cmd(item, 'build/' ~ (item | basename | with_suffix('.o'))) }}"
    sources: "{{ item }}"

defaults:
  - build/main.o

netsuke_test_version: "1.0"

macros:
  - signature: "stand_in_compile(src, obj)"
    body: |
      STUB {{ src }} -> {{ obj }}

test_skips_filtered_sources:
  description: foreach expands sources; when filters skip.c;
    the compile macro can be substituted.
  tags: [manifest, foreach]
  steps:
    - given:
        env:
          set:
            CC: clang
        let:
          glob: mock(args=["src/*.c"],
                     returns=["src/main.c", "src/skip.c"])
          compile_cmd: substitute("stand_in_compile")
      when: generate_ninja
      then:
        - result.ok
        - result.graph.has_target("build/main.o")
        - not result.graph.has_target("build/skip.o")
        - contains(result.ninja, "STUB src/main.c -> build/main.o")
        - mocks.glob.call_count == 1
        - substitutes.compile_cmd.call_count == 1

$ netsuke test

PASS compile.yml::test_generates_object_targets
FAIL compile.yml::test_substituted_macro_receives_source

  then[1]: substitutes.compile_cmd.calls[0].args[0] == "src/main.c"
    substitutes.compile_cmd.calls[0].args[0] = "src/lib.c"

2 cases: 1 passed, 1 failed

Under the bonnet