Sempai query engine

Updated Jun 22, 2026

The sempai crate provides a Semgrep-compatible query engine backed by Tree-sitter for semantics-aware code pattern matching. It is organized as three workspace crates:

  • sempai_core — canonical data model including language identifiers (Language), source spans (Span, LineCol), match results (Match), capture bindings (CaptureValue, CapturedNode), structured diagnostics (DiagnosticReport, Diagnostic, DiagnosticCode), and engine configuration (EngineConfig).
  • sempai_yaml — Semgrep-compatible YAML parser built on saphyr and serde-saphyr, exposing schema-aligned rule models for legacy and v2 search principals plus parser-time handling for extract, join, and taint rules.
  • sempai — stable facade crate that re-exports all public types from sempai_core and provides the Engine entrypoint.

The Engine struct exposes three methods for query compilation and execution:

  • compile_yaml(yaml) — compiles a YAML rule file into query plans.
  • compile_dsl(rule_id, language, dsl) — compiles a one-liner domain-specific language (DSL) expression.
  • execute(plan, uri, source) — executes a compiled plan against a source snapshot.

compile_yaml(yaml) now performs real YAML parsing plus a mode-aware validation pass. Malformed YAML returns E_SEMPAI_YAML_PARSE, and schema-shape failures such as missing required rule keys return E_SEMPAI_SCHEMA_INVALID, both using the shared structured diagnostic payload with primary_span locations when available.

After parsing succeeds, compile_yaml(yaml) normalizes search rules into canonical query plans:

  • Valid search rules are normalized into the canonical Formula model defined in sempai_core::formula. Both legacy (pattern*) and v2 (match) syntaxes are lowered into a shared representation.
  • Normalized formulas are validated for semantic correctness: E_SEMPAI_INVALID_NOT_IN_OR is emitted when negated terms appear in disjunction branches, and E_SEMPAI_MISSING_POSITIVE_TERM_IN_AND is emitted when conjunctions contain only constraint formulas.
  • For each valid search rule and declared language, a QueryPlan is returned containing the normalized formula and metadata.
  • Valid extract, taint, join, and unknown future mode strings fail deterministically with E_SEMPAI_UNSUPPORTED_MODE.
  • Compatibility-only r2c-internal-project-depends-on rules normalize to a degenerate formula that will never match real code.

Unsupported-mode diagnostics point at the rule's mode field when that span is available. Semantic validation errors include accurate primary_span locations when available from the parser.

Migration notes

Upgrading from v0.1? See the Sempai v0.1→v0.2 migration guide.

compile_dsl(...) and execute(...) still return "not implemented" diagnostics. They will be wired to the DSL parser and Tree-sitter backend as those components are delivered in subsequent roadmap phases.

All error conditions are reported through DiagnosticReport, which carries stable diagnostic codes suitable for programmatic consumption. Stub methods return the NOT_IMPLEMENTED code where implementation is still pending, while the YAML parser now emits real E_SEMPAI_* codes for malformed or invalid rule files. Diagnostics include a code, message, primary_span (or null when unavailable), and supplementary notes. Both parser-path and validator-path diagnostics use the same JSON schema, and snapshot tests lock this contract.