Report correctness validation (Phase 2.4.a)

Updated Jul 14, 2026

Generated repository reports are validated before persistence. If the status model produces output that fails basic correctness checks, Ghillie retries generation a bounded number of times. If all retries fail, the run is marked for human review rather than silently storing invalid data.

Validation checks

The following heuristics are applied to each generated report:

  • Non-empty summary: The model must produce a non-empty, non-whitespace summary string.
  • Truncation detection: Summaries ending with a trailing ellipsis (... or \u2026) are rejected as likely truncated.
  • Highlight plausibility: The number of highlights must be proportional to the number of events in the evidence bundle. A highlight count exceeding five times the event count is rejected as implausible.

Retry behaviour

When validation fails, the service retries model invocation up to validation_max_attempts times (default: 2). If the model eventually produces a valid result, the report is persisted normally. If all attempts fail, the service persists a ReportReview marker in the Gold layer and raises a validation error.

Human review markers

Operators can query ReportReview rows to identify failed generation runs. Each review marker records:

  • the repository and reporting window,
  • the number of generation attempts,
  • the specific validation issues encountered,
  • a lifecycle state (pending or resolved).

Review markers are uniquely constrained per (repository_id, window_start, window_end) to prevent duplicates from repeated retries in the same window.

API behaviour for validation failures

When the on-demand endpoint encounters a validation failure, it returns HTTP 422 Unprocessable Entity with a JSON body:

{
  "title": "Report validation failed",
  "description": "Generated report failed correctness checks after retries.",
  "review_id": "abc-123",
  "issues": [
    {
      "code": "empty_summary",
      "message": "Summary is empty or contains only whitespace."
    }
  ]
}

The review_id field references the persisted ReportReview row so operators can locate the review marker directly.

Validation configuration

Variable Default Description
GHILLIE_VALIDATION_MAX_ATTEMPTS 2 Maximum model invocation attempts