Estate catalogue (Phase 1.1)

Ghillie now ships a YAML 1.2 catalogue describing programmes, projects, components, repositories, and their relationships. Catalogue files are validated with msgspec and exposed as a JSON Schema for external linters.

• Use lowercase, dash-separated keys for programmes, projects, and components (for example, wildside-engine). Keys must be unique across the estate.
• Components may omit repository when they represent planned work. When present, repositories require owner, name, and default_branch.
• Capture relationships with depends_on, blocked_by, and emits_events_to. Each entry includes the target component and an optional kind (runtime, dev, test, ops) plus a short rationale.
• Configure per-project noise filters (ignore_authors, ignore_labels, ignore_paths, ignore_title_prefixes) and status preferences under noise and status. Set noise.enabled: false to disable all filters for a project, or use noise.toggles.* to disable individual filters without deleting the configured values. Setting summarise_dependency_prs: false signals that dependency update pull requests should be ignored in downstream summaries.
• Record documentation paths at both project level (documentation_paths) and per repository (repository.documentation_paths) so roadmaps and ADRs are discoverable to ingestion and summarization jobs.
• See examples/wildside-catalogue.yaml for a complete multi-repository project with planned components and cross-project dependencies.

The catalogue linter enforces YAML 1.2 semantics (strings like on remain strings) and referential integrity between components.

1. Generate schema and JSON artefacts from a catalogue file:

   python -m ghillie.catalogue.cli examples/wildside-catalogue.yaml \
     --schema-out schemas/catalogue.schema.json \
     --json-out .cache/catalogue.json

2. Validate against the JSON Schema with pajv:

   pajv -s schemas/catalogue.schema.json -d .cache/catalogue.json

3. A non-zero exit code indicates structural errors, such as missing components in dependency lists or duplicate keys.

The catalogue example models Wildside as a multi-repository project:

• leynos/wildside (core API) depends on wildside-engine and df12 shared libraries (ortho-config, pg-embedded-setup-unpriv, rstest-bdd).
• leynos/wildside-engine underpins the core service and is blocked by shared configuration rollout.
• leynos/wildside-mockup models UI experiments and receives events from the core service.
• wildside-ingestion is marked lifecycle: planned to represent work with no repository yet.

Noise controls ignore dependency bots and generated documentation paths, so the ingestion pipeline can focus on meaningful events.

The catalogue importer reconciles the YAML file into relational tables for estates, projects, components, repositories, and component edges. Imports run inside a single transaction: invalid catalogues fail fast and do not leave partial rows behind. Re-running the same commit is idempotent and will prune entries removed from the source catalogue.

Project noise filters, status preferences, and documentation paths are persisted alongside projects and repositories, so ingestion and reporting services can consume them without parsing YAML at runtime.

Operational note: existing deployments must add a JSON documentation_paths column to the repositories table before enabling this feature because Base.metadata.create_all will not alter existing tables in place.

Example: load the example catalogue into a SQLite database using the asynchronous importer:

import asyncio
from pathlib import Path

from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from sqlalchemy import select

from ghillie.catalogue import CatalogueImporter, init_catalogue_storage


async def main() -> None:
    engine = create_async_engine("sqlite+aiosqlite:///catalogue.db")
    await init_catalogue_storage(engine)
    session_factory = async_sessionmaker(engine, expire_on_commit=False)

    importer = CatalogueImporter(session_factory, estate_key="wildside")
    await importer.import_path(Path("examples/wildside-catalogue.yaml"), commit_sha="abc123")


asyncio.run(main())

In deployments that already run Dramatiq workers, use the ghillie.catalogue.importer.import_catalogue_job actor. It accepts the catalogue path, database URL, estate key, optional estate name, and commit SHA so scheduling systems can enqueue work without importing Python modules.