Netsukefile Testing Framework

Test tree and files

Tests live in a netsuke-tests directory beside the Netsukefile. An optional top-level tests block in the manifest reconfigures discovery; it is tool configuration, not build data, so its values are invisible to templates and tests.root accepts no Jinja. Files whose name starts with _ are support files, loaded only when imported. A run that discovers no files, or whose filters select zero cases, fails rather than reporting a silent green.

netsuke_version: "1.2.0"

tests:
  root: netsuke-tests
  include:
    - "**/*.yml"
    - "**/*.yaml"
  exclude:
    - "**/_*.yml"
    - "**/_*.yaml"

netsuke_test_version: "1.0"   # required in every file

imports:                       # optional support files,
  - _fixtures.yml              # confined to the test tree

vars: {}                       # suite-local test variables

macros: []                     # same shape as Netsukefile macros

fixtures: {}                   # fixture definitions

test_compile_target:           # one or more cases
  steps: []

netsuke_test_version is a MAJOR.MINOR string. The dialect denies unknown keys everywhere, so every addition is a minor-version event and an older runner rejects a newer file rather than misreading it; an unknown key is reported with the nearest known key. Case names match test_[A-Za-z0-9_]+ and are reported as <file>::<name>.

Given, when, then

A case is an ordered list of steps. Each step carries at least one of given (fixtures, environment, doubles, clock), when (a pipeline action), and then (assertions). Context from given persists across the later steps of the same case; nothing persists between cases. The smallest useful test needs none of the machinery:

netsuke_test_version: "1.0"

test_hello_target_is_generated:
  steps:
    - when: generate_ninja
      then:
        - result.ok
        - result.graph.has_target("build/hello.txt")

then:
  - result.ok
  - result.graph.targets | length == 3
  - equals:
      actual: result.graph.edge_count
      expected: 4
  - contains:
      value: result.ninja
      needle: "default app"
  - matches:
      value: result.error.message
      regex: "missing.*rule"

Doubles and fixtures

given.let binds a double to a template function name. The three kinds follow the stub, mock, and spy taxonomy: a stub returns a canned value and is journalled but never verified; a mock is verified, so every declared expectation must be met and an unmatched call fails the action immediately; a spy journals every call and passes through to the effective implementation under test configuration. The terse form is deliberately the loose one, so don't-care collaborators cost one line and full expectation machinery is opt-in.

given:
  env:
    set:
      CC: clang
  let:
    glob: mock(args=["src/*.c"], returns=["src/main.c"])
    cc:   stub(returns="clang")
    now:  spy()
    compile_cmd: substitute("stand_in_compile")

Fixtures

Fixtures are lifecycle objects: ordered setup actions, exports, and ordered teardown, with an action vocabulary of tmpdir, mkdir, write, copy, remove, and env. Every path resolves inside a per-case sandbox; a fixture cannot touch the project root or the host filesystem. Teardown runs in reverse order for every fixture whose setup completed, regardless of later failures, and a failing teardown never masks the case result. All fixtures are case-scoped in the first version; an arbitrary-command run action is deferred with execution generally.

fixtures:
  tiny_c_project:
    description: Minimal C project with one source and a Netsukefile.
    setup:
      - tmpdir: project
      - mkdir: "{{ project }}/src"
      - write:
          path: "{{ project }}/src/main.c"
          text: |
            int main(void) { return 0; }
      - write:
          path: "{{ project }}/Netsukefile"
          text: |
            netsuke_version: "1.2.0"
            targets:
              - name: build/main.o
                command: "cc -c src/main.c -o build/main.o"
                sources: src/main.c
    exports:
      root: "{{ project }}"
      manifest: "{{ project }}/Netsukefile"

The command

netsuke test [FILTER...]

Arguments:
  FILTER                   Case selectors: file paths, file::case names, or
                           substring patterns

Options:
  --tests-dir <DIR>        Override tests.root
  --list                   List discovered cases without running them
  --tag <TAG>              Run only cases with this tag (repeatable)
  --skip-tag <TAG>         Exclude cases with this tag (repeatable)
  --fail-fast              Stop after the first failing case
  --timeout <SECS>         Per-case wall-clock budget (default 60)
  --keep                   Preserve sandboxes of failing cases
  --allow-empty            Succeed when zero cases are selected

Worked example

The case below verifies foreach expansion, when filtering, environment-driven command construction, and macro wiring — without a compiler installed, without touching the real filesystem, and identically on every machine. It is the design's own success criterion.

netsuke_version: "1.2.0"

tests:
  root: netsuke-tests

macros:
  - signature: "compile_cmd(src, obj)"
    body: |
      {{ env('CC') }} -c {{ src }} -o {{ obj }}

targets:
  - foreach: glob('src/*.c')
    when: item | basename != 'skip.c'
    name: "build/{{ item | basename | with_suffix('.o') }}"
    command: "{{ compile_cmd(item, 'build/' ~ (item | basename | with_suffix('.o'))) }}"
    sources: "{{ item }}"

defaults:
  - build/main.o

netsuke_test_version: "1.0"

macros:
  - signature: "stand_in_compile(src, obj)"
    body: |
      STUB {{ src }} -> {{ obj }}

test_skips_filtered_sources:
  description: foreach expands sources; when filters skip.c;
    the compile macro can be substituted.
  tags: [manifest, foreach]
  steps:
    - given:
        env:
          set:
            CC: clang
        let:
          glob: mock(args=["src/*.c"],
                     returns=["src/main.c", "src/skip.c"])
          compile_cmd: substitute("stand_in_compile")
      when: generate_ninja
      then:
        - result.ok
        - result.graph.has_target("build/main.o")
        - not result.graph.has_target("build/skip.o")
        - contains(result.ninja, "STUB src/main.c -> build/main.o")
        - mocks.glob.call_count == 1
        - substitutes.compile_cmd.call_count == 1

$ netsuke test

PASS compile.yml::test_generates_object_targets
FAIL compile.yml::test_substituted_macro_receives_source

  then[1]: substitutes.compile_cmd.calls[0].args[0] == "src/main.c"
    substitutes.compile_cmd.calls[0].args[0] = "src/lib.c"

2 cases: 1 passed, 1 failed

From the shallow end

The worked example above is the design’s finish line, not its starting point. The quick start in the UX design needs nothing beyond a Netsukefile and one test file in the default netsuke-tests directory: no tests block, no fixtures, no doubles, and no given. The test is the hello.yml case shown under given, when, then.

netsuke_version: "1.2.0"

targets:
  - name: build/hello.txt
    command: echo hello > build/hello.txt

$ netsuke test

PASS hello.yml::test_hello_target_is_generated

1 case: 1 passed

That case needs no compiler, no network, and no real files, because generate_ninja runs against the in-memory pipeline. From there the path deeper is a ladder, and each rung is opt-in:

1. 1Assertions only. Boolean expressions over result, with then after a pipeline action.
2. 2A loose stub. stub(returns=…) in given.let is the one-line default for a collaborator the test does not care about.
3. 3A spy or a verified mock. spy() records calls to the effective implementation; mock(…) adds expectations that must be met.
4. 4The structured mocks block. The full-fidelity form every shorthand desugars to, with matchers and call configurations.
5. 5Fixtures and macro substitution. Sandboxed files and stand-in macros, as in the worked example.

The design calls this strictness ladder deliberate: over-mocking is the documented failure mode of expressive mock frameworks, so the terse form stays the loosest. Test files declare macros exactly as a Netsukefile does, so authors carry one mental model between manifests and tests, and a mistyped key such as fixture: is rejected with the nearest known key.

Accessibility and localization

netsuke test is designed to follow the established display policy flags — --color, --emoji, --progress, and --accessibility — rather than invent its own, so the accessible output modes apply to test reports as they do to builds. Every human-facing string is to be localized through Fluent.

Under the bonnet

How the design earns trust