Incremental GitHub ingestion (Phase 1.3.b)

Ghillie polls GitHub per managed repository and appends activity into the Bronze raw_events table. Each repository has per-kind watermarks stored in github_ingestion_offsets, allowing the worker to fetch only new commits, pull requests, issues, and documentation changes since the last successful ingestion run.

The reference implementation uses the GitHub GraphQL API for commits, pull requests, and issues, and reuses commit history filtering for documentation paths (for example, roadmaps and ADR directories).

The GraphQL client expects a GitHub token in GHILLIE_GITHUB_TOKEN. For pilot deployments, use a fine-scoped token or GitHub App installation token with read-only access to the managed repositories. For production deployments, see GitHub Application configuration for guidance on creating a GitHub App with least-privilege permissions.

import asyncio

from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine

from ghillie.github import (
    GitHubGraphQLClient,
    GitHubGraphQLConfig,
    GitHubIngestionWorker,
)
from ghillie.registry import RepositoryRegistryService


async def main() -> None:
    engine = create_async_engine("sqlite+aiosqlite:///ghillie.db")
    session_factory = async_sessionmaker(engine, expire_on_commit=False)

    registry = RepositoryRegistryService(session_factory, session_factory)
    repos = await registry.list_active_repositories()

    client = GitHubGraphQLClient(GitHubGraphQLConfig.from_env())
    # If the catalogue database is separate from Bronze/Silver, pass it via config:
    # config = GitHubIngestionConfig(catalogue_session_factory=catalogue_sf)
    worker = GitHubIngestionWorker(session_factory, client)

    for repo in repos:
        await worker.ingest_repository(repo)

    await client.aclose()


asyncio.run(main())

After ingestion, run RawEventTransformer.process_pending() to hydrate the Silver entity tables (commits, pull_requests, issues, documentation_changes) from the newly-ingested raw events.

The test fixtures now attempt to start a py-pglite Postgres instance by default so behavioural and unit tests exercise real Postgres semantics. If py-pglite cannot start (for example, Node.js is missing), the fixtures automatically fall back to SQLite to keep the suite runnable. To force SQLite explicitly, set GHILLIE_TEST_DB=sqlite before invoking make test. See docs/testing-sqlalchemy-with-pytest-and-py-pglite.md for full guidance.