Quality and review

What the system may decide, and what it must hand over

Six evaluators inspect a draft and return typed findings. None of them approves anything, and the difference is the whole design.

The evaluator catalogue

Five run as structured model calls inside the generation graph. Chrono makes no model call at all.

Evaluators execute in parallel over a draft and aggregate before the graph routes on the result. Each returns typed findings and, where a model was involved, normalized usage metrics for cost accounting. Schemas are versioned alongside the prompts and the orchestration code.

Chrono

Available

Spoken runtime estimation.

Mechanism
Local deterministic heuristic. TEI parsing and spoken-text extraction delegate to tei-rapporteur, then word tokens are counted at 150 words per minute and rounded up to whole seconds.
Returns
Runtime estimate with estimator name, estimator version, input character count, spoken word count, and the words-per-minute setting, so later estimators stay comparable.
ADR 006: Chrono spoken text semantics (opens the Episodic repository)

Anthem

Planned

Brand-guideline compliance.

Mechanism
Structured LLM call linting vocabulary and forbidden topics, and validating tone against the series brand tone descriptors.
Returns
Pass or fail outcomes linked to the canonical episode.
Roadmap task 2.2.4 (opens the Episodic repository)

Caesura

Planned

False endings.

Mechanism
Structured LLM call over the draft, informed by Chrono's runtime estimate.
Returns
Findings identifying the suspected false-ending boundary, where an episode appears to wrap up but continues for another four minutes or more.
Roadmap task 2.2.5 (opens the Episodic repository)

Two of six are implemented. The rest are specified with their contracts and not yet built; the status on each record says which is which.

What a finding looks like

Structured, per-claim, and carrying remediation guidance rather than a verdict. The example is synthetic.

Synthetic Pedante finding Shape only; support levels and severities are defined by the evaluator contract
{
  "claim_id": "c-014",
  "claim_kind": "attributed-quotation",
  "cited_sources": ["src-transcript-01"],
  "support_level": "partial",
  "severity": "high",
  "remediation": "The transcript records this phrasing at 00:14:22 but attributes it to the second speaker. Confirm the attribution or cut the quotation."
}

A finding names what it examined, what it could and could not tie to a source, and what a person might do about it. It does not say the script is wrong, and nothing downstream treats a severity as an instruction.

The decision boundary

Stated as a matrix. Status is per row, because the boundary is delivered in pieces.

What the system decides, and what a person decides
Activity The system A person Status
Weighting and selecting between conflicting sources Computes priority scores from quality, freshness, and reliability, and applies the weighting matrix. Sets the weighting coefficients per series profile, and can inspect every rejected source. Available
Estimating episode runtime Produces a deterministic estimate and records the estimator metadata behind it. Decides whether the estimate is close enough to commission the recording. Available
Scoring a draft for accuracy Returns per-claim findings with severity and remediation guidance. Decides whether a finding is correct and what to do about it. A score is not a verdict. In progress
Spending money on model calls Reserves budget before a billable call, commits on success, and releases on failure. Sets the per-episode budget and the tier of model each task may use. In progress
Approving an episode for an audience Surfaces findings, suspends the graph, and preserves state while the decision is pending. Approves, requests changes, or redirects the run. This decision has no automated path. Planned

Guardrails and budgets

Two mechanisms stop a run: an editorial guardrail and an exhausted budget.

Guardrails persist on series profiles and episode templates, so a generation request carries stable editorial instructions as its system prompt rather than depending on whoever composed the call. Changing the guardrail changes every subsequent run from one place, under the same change history and optimistic locking as the rest of the profile.

Budgets use reservation semantics around each billable call: reserve, then commit on success or release on failure. A run that cannot reserve does not make the call. Cost line items pin the provider pricing snapshot in force at the time, so a later price change does not rewrite what an episode cost.

Malformed structured output from a model Available
The run stops. Nothing partially parsed reaches the canonical episode.
Chapter-marker timing that does not match segment transitions Available
Validation rejects the payload. Starts must be ordered, integer-only, and in PT#H#M#S form.
Optimistic-locking conflict on a profile or template Available
The write is refused so a concurrent edit is never silently overwritten.
Repeated request with a matching idempotency key Available
The stored response replays. The same key with a different canonical body returns 409 Conflict.
Per-episode budget exhausted In progress
The reservation is refused before the billable call is made.

Editorial checkpoints

Designed, not delivered. The honest state of the most important part of the system.

A checkpoint suspends the generation graph and preserves its state while an editor considers the findings. Because the state is durable, a review period can last as long as it needs to without losing the run - the graph resumes rather than restarting.

A thick stack of printed transcript pages under a desk lamp, the top page marked up in pencil with margin notes and struck-through lines.
What an evaluator cannot do. Findings raise a question; a person answers it.

The reviewer actions the design provides for are approve, request changes, and redirect the run with targeted remediation prompts. Drafts that fall below threshold re-enter generation; drafts above it proceed to approval. Which threshold, and whether a team wants automatic re-entry at all, is a series-level decision.