Sempai is where Weaver stops guessing.
String matching is cheap until it hits real code. Then it lies. The Sempai design fixes that by taking Semgrep-shaped queries, lowering them to a normalized formula model, executing them against Tree-sitter syntax trees, and returning precise spans, captures, and focus for actuation.
Input
Three front doors.
Bare structural patterns via --query. A rich expression DSL via --expr. Semgrep-style YAML via --rule. No auto-detection, no guessing.
Scope
Four languages. Five with a flag.
Rust, Python, TypeScript, and Go are core. HCL is optional behind configuration.
Output
Matches that travel.
Every match leaves as a versioned selector record: span, captures, optional focus, and a source digest. Enough to act on, and to refuse when stale.
Discipline
No fake parity.
extract, taint, and join are parsed for compatibility, then reported as unsupported at execution time. Honest software. Rare sight.
Compatibility matters. Theatre does not.
The design deliberately accepts both legacy Semgrep pattern* operators and v2 match formulas. That is the right move. People already have rule corpora. Asking them to throw that away because a new engine wants to feel special would be unserious.
The supported surface is concrete: pattern, pattern-regex, patterns, pattern-either, v2 pattern, regex, all, any, not, inside, anywhere, plus where, as, and fix. The lexical tokens are equally explicit: $X, $_, $...ARGS, ..., and deep ellipsis. The doc even nails down semantic failure cases such as InvalidNotInOr and MissingPositiveTermInAnd. Good. Vagueness is where bugs hide.
Supported Now
- Legacy and v2 query operators on one surface.
- Rule-file parsing and one-liner DSL parsing.
- Search-mode execution for feature extraction.
- Stable Rust facade API with compile and execute phases.
Explicitly Out
- No full Semgrep parity in MVP.
- No autofix application inside Sempai.
- No path-sensitive dataflow analysis.
- No pretending parsed modes are executed modes.
Why Semgrep Syntax
Because the query language should speak code, not ceremony.
Regexps are blind to structure. Raw Tree-sitter queries are married to grammar internals. ast-grep is serious work, but it is still its own ecosystem with its own pattern habits. Semgrep syntax lands in the useful middle: structural, recognizable, portable, and already familiar to people who write code instead of node-kind tax returns.
That matters more in Weaver than it does in a standalone matcher. Sempai is not just finding text. It is selecting code objects for downstream action. The front door has to be expressive enough for structure, readable enough for humans, and stable enough for agents to generate without turning every query into a grammar seminar.
| Surface | What it does well | Positioning |
|---|---|---|
| Semgrep syntax | Matches code-shaped patterns, carries metavariables, supports context operators such as inside, and comes with an existing rule culture. |
It is the main interface because it is the only option here that stays structural without becoming hostile. |
| Regexps | Fast lexical filtering, line-oriented searches, and cheap text predicates. Useful enough that Sempai keeps regex as an atom. |
Regexps do not know what a function, decorator, argument list, or nested scope is. They match text. Sometimes that is enough. Usually it is a trap. |
| ast-grep | Strong structural matching and a credible code-query workflow built around its own patterns and rewrite model. | Good tool. Wrong centre of gravity for this design. It has its own pattern language and ecosystem. What matters here is rule portability: people already have Semgrep corpora, and the design commits to that operator vocabulary. ast-grep cannot ingest those rules without a translation layer that would defeat the purpose. |
| Tree-sitter queries | Precise, low-level capture of grammar nodes. Excellent as an escape hatch when you need exact parser internals. | They are too close to the metal for the main authoring path. Users should not have to memorize node kinds and field names just to say "find decorated classes". That is what the engine is for. |
Why the hierarchy matters
The design gets this exactly right: keep regex as a supported atom, keep raw Tree-sitter queries as an explicit escape hatch, and make Semgrep syntax the primary surface. That is not compromise. That is layering.
Use the high-level language for the common case. Use the sharp tools when you actually need them. Use the right abstraction until it runs out. Then go lower. Not before.
Expression DSL
The DSL provides a compact expression form for --expr and --expr-file. It maps directly to the normalized formula model. A bare positive pattern needs none of it: --query 'fn $NAME($...ARGS)' takes the host-language pattern directly, with no pattern("...") wrapper.
Atoms
pattern("...")regex("...")ts("...")
Operators
- and, or (infix)
- not(...) (prefix)
- inside(...), anywhere(...) (prefix)
Decorators
where { focus("$X") }as "$X"fix "..."
Example
weaver symbols list \
--lang python \
--expr 'pattern("@decorator\nclass $C: ...") and not(inside("class Test$_: ..."))'
Tree-sitter Extension Keys
PlannedPlanned. These YAML extension keys are design-surface only for now. When they land, they will cover the exact grammar-node cases where Semgrep patterns are too coarse:
| Key | Type | Purpose |
|---|---|---|
| ts-query | string | Raw Tree-sitter S-expression query used as an escape hatch when Semgrep patterns are insufficient. |
| ts-language | string | Override language for the Tree-sitter grammar (when distinct from the rule's target language). |
| ts-focus | string | Capture name to use as the focus span for actuation handoff. |
Compile once. Execute many. Bound the ugly parts.
The engine design makes the right architectural split: compilation and execution are separate phases, QueryPlan is cacheable, and Tree-sitter parses are cacheable per file revision. That matters because query engines die from repeated work and accidental explosion, not from lack of cleverness.
Snippet compilation is the hard part. Pattern fragments with $X, ..., and deep ellipsis are not clean host code, so Sempai wraps them, rewrites tokens into parseable placeholders, and lowers the result into pattern IR. That is less glamorous than waving around "semantic search". It is also how you get something real.
Performance Controls
QueryPlancache keyed by language and rule hash.- Tree-sitter parse cache keyed by file revision.
- Candidate-kind pruning from
PatNode::Kind. - Bounded deep matching via
max_deep_search_nodes. - Deterministic match and capture truncation.
Anchors produce matches. Constraints police them.
This is the design choice that makes the rest of the system coherent. inside and anywhere are treated as constraints in conjunction contexts, not as free-standing match producers. Why does that matter? Because otherwise conjunctions become semantic mush and every "positive term" rule becomes a guess dressed as an operator table.
The execution plan splits anchor-producing terms from constraint terms, then evaluates them in order: first anchor matches, then predicates such as Inside, Not, Anywhere, and Where, then compatible existential matches from additional anchors, then projection of span, captures, and focus. That is not just neat. It is debuggable.
The interface is narrow.
That is a feature.
The stable facade crate exposes exactly what it should: compile YAML, compile DSL, execute plans. Not a sprawling bag of internal types. The public surface is weaver symbols list with exactly one of --query, --query-file, --expr, --expr-file, --rule, or --rule-file; - means stdin, and only one flag may own it. A wrong choice returns an error enumerating the alternatives. Enough to be useful. Not enough to rot instantly.
CLI Contract
weaver symbols list \
--lang rust \
--query 'fn $F($...ARGS) { ... }' \
--json
Actuation Handoff
Sempai does not edit code. It hands Weaver an address. Consumers take the stream through the typed --selectors <path|-> flag; focus is the primary target when present, span the fallback, and the actuator re-checks the source digest before touching anything. That keeps query execution and actuation separate, which is exactly where the blast radius should stop.
Stream Rules
- Zero or more selector records, then exactly one completion record. Zero matches is a successful, completion-only stream.
- Canonical ordering: URI, then start byte, then end byte, then selector id. Deterministic every run.
- A producer failure never emits a successful completion record – consumers can trust the terminator.
- No invented confidence scores. Deterministic structural matches report evidence and compatibility provenance instead.
Payload Fields
| Field | Description |
|---|---|
| schema | weaver.selector.v1 for matches; weaver.selector-stream-end.v1 for the terminator. |
| stream / selector_id | Stream identity plus sequence, and a stable identifier for each selector. |
| uri / language | Source file URI and language of the match. |
| span | Byte offsets for the full match. |
| focus | Narrowed span when a where { focus("$X") } decorator is present. null otherwise. |
| captures | Bounded map of metavariable names to { text, kind, span }. |
| source | Digest and workspace revision of the matched file – the stale-refusal precondition travels with the match. |
| query | Query kind and digest, plus capability and provider provenance. |
| fix | Suggested fix text from the rule's fix key, or null. Surfaced as metadata — not applied by Sempai. |
Selector Stream (--json)
{
"schema": "weaver.selector.v1",
"stream": { "id": "st_9d2f", "seq": 0 },
"selector_id": "sel_01",
"uri": "file:///repo/app.py",
"language": "python",
"span": { "start": 12, "end": 42 },
"focus": null,
"captures": {
"C": { "text": "MyClass", "kind": "identifier", "span": { "start": 18, "end": 26 } }
},
"source": { "digest": "sha256:41ab..." },
"query": { "kind": "expr", "digest": "sha256:77c0..." }
}
{
"schema": "weaver.selector-stream-end.v1",
"stream": { "id": "st_9d2f" },
"matches": 1,
"complete": true
}
Formatted for reading. On the wire each record is a single JSONL line: zero or more selectors, then exactly one completion record.
Weaver Integration
Sempai powers weaver symbols list and sits alongside definitions get, references list, and diagnostics list in the read loop. Where those commands ask the language server "what is this symbol?", Sempai asks the syntax tree "where does this pattern appear?". The first executable slice covers Rust, Python, and TypeScript through an explicitly labelled weaver-syntax-compat-v1 compatibility adapter – the label travels in the output, so nobody mistakes the subset for full parity.
What it refuses to fake
- Parsed modes are not silently "best-effort" executed.
fixis surfaced as metadata, not applied by wishful thinking.- Unsupported constraints return structured errors instead of undefined behaviour with nicer branding.
- Diagnostics are first-class: stable
E_SEMPAI_*codes, byte spans, and multiple independent errors from a recovering parser. No query with error-severity diagnostics is executed.
What it actively defends against
- Match explosions via
max_matches_per_rule. - Capture bloat via
max_capture_text_bytes. - Deep-search blowups via bounded traversal and branch caps.
- Repeated parse work via cache ownership in
weaverd.
| Safety Limit | Default | Effect |
|---|---|---|
| max_matches_per_rule | Configured per deployment | Truncates output deterministically once the cap is reached. |
| max_capture_text_bytes | Configured per deployment | Truncates or disables text fields in capture objects. |
| max_deep_search_nodes | Configured per deployment | Bounds deep-ellipsis traversal to prevent combinatorial blowup. |
| Mode | Parse | Execute |
|---|---|---|
| search | Yes | Full support |
| extract | Yes | UnsupportedMode |
| taint | Yes | UnsupportedMode |
| join | Yes | UnsupportedMode |
The crate split is not decorative.
The design proposes a facade crate (sempai) over focused internal crates for core model, YAML, DSL, Tree-sitter backend, and fixtures. That separation keeps the public API stable while letting the parser and backend evolve at a sane pace. One crate to consume. Several crates to argue with internally. Correct.
| Crate | Role |
|---|---|
| sempai | Stable facade and convenience entrypoints. |
| sempai-core | Data model, diagnostics, plan IR, semantic validation. |
| sempai-yaml | Rule-file parsing via saphyr and serde-saphyr. |
| sempai-dsl | One-liner DSL parsing via logos and Chumsky. |
| sempai-ts | Tree-sitter backend, language profiles, matcher. |
| sempai-fixtures | Corpora and helpers for tests. |
Roadmap, Compressed
- 1. Ship core infrastructure: scaffolding, stable API, YAML parsing, DSL parsing.
- 2. Ship Tree-sitter backend: language profiles, snippet parsing, matcher, escape hatch.
- 3. Ship Weaver integration: the
symbols listsurface, selector streams, and the typed--selectorshandoff into the change loop.
Bottom Line
Sempai does not need to be magical. It needs to be dependable, structural, and explicit about what it can and cannot do. Today that honesty cuts both ways: the horizontal foundations are built, and the executable query loop is the work of the current roadmap phase, not a shipped feature.
Primary source
This white paper is based on weaver/docs/rfcs/0003-sempai-query-to-selector.md with ADRs 011 (query input syntax) and 012 (versioned selector streams), and on the language-design sections of weaver/docs/sempai-query-language-design.md – whose older command-surface sections the RFC and ADRs supersede.
Design basis
The page layout and figure treatment follow the material system, palette, grid language, and motif logic established in the design-language page and the reference plates in example_art/.
The three illustrations on this page were generated to clarify the actual mechanics of the design: pipeline flow, snippet compilation, and anchor-plus-constraint evaluation with focus projection.