How Netsuke Works - Netsuke Build System

A build-system compiler

Ninja describes itself as an assembler for build systems: deliberately constrained, no string manipulation, no conditionals, built to run builds as fast as possible. That constraint creates a job opening for a higher-level generator. Netsuke fills it.

The manifest describes the what and why of a build — the project's structure, its logical rules, its configurable parameters. Netsuke compiles that description into a low-level execution plan that Ninja understands. Netsuke manages build logic; Ninja manages execution. Every other design decision follows from that separation.

Netsuke: the compiler

Parses the manifest, expands templates, validates the graph, and writes build.ninja. All the thinking happens here.

Ninja: the assembler

Executes a pre-computed static graph with maximum parallelism. No runtime queries, no surprises. All the running happens here.

The six-stage pipeline

A build transforms the Netsukefile into artefacts through six stages. YAML validity comes first, dynamic logic resolves in the middle, and execution comes last.

1

Manifest ingestion

Locate the project manifest and read it into memory as a raw string. Nothing is interpreted yet.

2

Initial YAML parsing

The raw string is parsed with serde_saphyr into an untyped value. The manifest must be valid YAML before any templating takes place — template errors never masquerade as syntax errors.

3

Template expansion

Netsuke walks the parsed value, evaluating Jinja macros, variables, and the foreach and when keys on targets and actions. Control flow lives only in these explicit keys; structural {% ... %} blocks may not reshape mappings or sequences. Variable lookups respect the precedence globals < entry.vars < per-iteration locals.

4

Deserialization and final rendering

The expanded value is deserialized into strongly typed Rust structs. Jinja expressions render inside string fields only. The structure is now fixed; only the strings were ever negotiable.

5

IR generation and validation

The typed manifest becomes a canonical intermediate representation: a static dependency graph with every path, command, and dependency explicit. Validation happens here — referenced rules must exist, each rule carries exactly one of command or script, every target names exactly one recipe, and circular dependencies and missing inputs are rejected.

6

Ninja synthesis and execution

A code generator traverses the validated IR and writes build.ninja, translating nodes and edges into Ninja rule and build statements. Netsuke then invokes ninja as a subprocess and hands over dependency checking and command execution.

Deterministic by construction. Given the same Netsukefile and environment variables, the generated build.ninja is byte-for-byte identical — so long as the manifest avoids impure helpers such as fetch() and shell(), which observe the host at evaluation time, or explicitly declares their outputs as inputs. Under that discipline, reproducible builds fall out for free, and the output is safe to cache or commit.

The static graph mandate

Ninja's incremental-build speed comes from operating on a pre-computed graph and refusing costly runtime operations — no glob expansion, no string manipulation, no conditionals. A friendlier build system still needs dynamic capability: different flags per platform, source discovery, configuration. Something has to give.

Netsuke's answer: all dynamic logic is fully evaluated before Ninja is ever invoked. Templating, iteration, and conditionals belong to the compile phase; the execution phase sees only literal paths and literal commands. The transition point is the intermediate representation — a static snapshot of the build plan after every Jinja decision has been made.

This is why the pipeline has six stages rather than two. The mandate draws a clean boundary between the user-facing logic layer and the machine-facing execution layer, and everything on the user's side of that line stays inspectable: run netsuke generate and read the plan before anything executes.

The intermediate representation

The IR is the build graph with nothing left to decide: fully resolved, validated, and canonical. It is also a deliberate decoupling layer between the YAML front-end and the Ninja back-end — the object code of the Netsuke compiler.

Front-end freedom

The YAML schema can evolve — even radically — and as long as the transformation still produces the same stable IR, the Ninja generator never notices.

Back-end freedom

Supporting another execution backend means writing one new generator from the IR. The parsing, expansion, and validation front-end is untouched.

The IR contains no Ninja-isms. Placeholders such as $in and $out belong to the synthesis stage, not the graph. What the IR stores is the build's meaning; how a backend spells it is the backend's problem.

Handing off to Ninja

Execution is delegated, not abandoned. Netsuke spawns ninja through Rust's std::process::Command, forwarding relevant CLI arguments — a target name, -j for parallelism — and capturing stdout and stderr so progress can be streamed back with Netsuke's own status reporting.

Directory changes via -C are canonicalized and applied to the child process's working directory rather than forwarded as flags.
Interpolated values in generated commands are shell-escaped by default, so a filename with a space is a filename with a space, not two arguments.
Network access from templating (fetch()) is governed by an auditable policy — HTTPS only by default, widened or narrowed by explicit flags.

Read the full security model

When it breaks

Errors are not exceptional events; they are part of the development workflow. Every error Netsuke reports sets out to answer three questions:

What went wrong?

A concise summary of the failure. "YAML parsing failed." "Build configuration is invalid."

Where did it go wrong?

File, line, and column where applicable. "In Netsukefile at line 15, column 3."

Why — and what can be done about it?

The underlying cause and a concrete fix. "Cause: found a tab character, which is not allowed. Hint: use spaces for indentation instead."

Because each pipeline stage validates its own concern, failures surface at the stage that understands them: syntax errors from parsing, unknown-rule errors from IR validation, command failures from execution. The stage that found the problem is the stage best placed to explain it.

Inside the crate

The pipeline maps directly onto the source tree. Each module owns one boundary:

Module	Responsibility
src/cli/	Command-line configuration, parsing, validation, and merge logic.
src/manifest/	Manifest parsing, expansion, rendering, and diagnostics — stages one to four.
src/ir/	Intermediate representation generation, interpolation, graph, and cycle logic — stage five.
src/runner/	Process execution, path handling, runner errors, and command orchestration — stage six.
src/stdlib/	The standard library exposed to manifest rendering: paths, files, collections, time, command discovery, and network access.
src/localization/	Fluent-backed localization for user-facing messages.

For the complete rationale — schema decisions, the templating trade-offs, error-crate selection, and the CLI design — read the full design document in the repository.