The Fogo Evaluation Harness

• → Why and when. Built once Fogo's AI was running end-to-end across all six surfaces. Prerequisite for onboarding more users with quality protocols in place, and for distilling Fogo into a smaller open-source student model.
• → Architecture. Two layers: deterministic gate, then quality scoring 1–5 across five dimensions. Four LLM-judged, one programmatic.
• → Calibration and fix. Blind-rated 28 prompts. First pass: κ_w = 0.07 with +1.02 bias, judge consistently a full point stricter. Root cause was structural: rubric had anchors at 1 / 3 / 5 only, so the 2 and 4 buckets were left to interpolation. Added them. κ_w 0.07 → 0.565, within-1 61% → 94%.
• → What sitting in the rater's seat caught. Six bugs, two inside the eval itself. The consequential one: the rejudge pipeline wrote new scores into one field but left a parallel field describing the old ones. Silent. The data just quietly disagreed with itself.
• → Where it lands. A calibrated eval harness, then the infrastructure that makes calibration a continuous practice instead of a script. Same data model becomes the training-data schema for the student-model distillation that comes next.

• · Separate mechanical failure from reasoning failure. A leaked label or a 12-second response is a different problem than a flat tone, and giving both the same quality score is dishonest.
• · Score each surface against its own rubric. A great review and a great factual chat have almost nothing in common. Averaging them under one rubric flattens the work that matters.
• · Write anchors at every score on the scale, including the in-between ones. The buckets you skip are where rater drift accumulates.
• · Report bias and noise separately. Headline κ collapses them. The signed-error tells you whether to rewrite anchors or collect more data.
• · Calibration surfaces bugs the rest of your tooling misses. The two eval-internal bugs in this writeup only came out because κ misalignment forced a deep read of the persisted data.

Fogo is a personal-finance app I built solo, and the AI does the substantive work across the product. Six surfaces (transaction classification, dashboard narrative, monthly review, chat, ambient insights, behavioral confirmations) are all driven by an LLM conditioned on a SQL-built financial state vector. The model is doing the substantive work on every surface, which is why the eval has to be real: a miscalibrated judge here teaches the whole system to optimize for the wrong thing.

The harness

Two layers, because mechanical failure and reasoning failure need different treatment. Layer 1 is a deterministic gate: pass/fail checks that catch responses already broken in ways the LLM judge can't reliably see. Layer 2 only runs if Layer 1 passes; it scores the response 1–5 across five quality dimensions. The split exists because asking "how good is the tone of this response?" is meaningless if the response leaked a raw internal label into user-facing text or returned in 18 seconds when the surface budgets 3. Mixing those failure modes under one quality score makes "did the prompt regress?" unanswerable.

The LLM gateway: what makes any of this measurable

An eval is only as good as the chokepoint every model call passes through. Every API call in the system routes through a single gateway function. The gateway routes by surface (rate limits, model selection, prompt assembly), attaches the SQL-built financial state vector and the deterministic emotional state to every trace as first-class fields, and logs to Langfuse. Without that chokepoint, "score every LLM response" is unenforceable. Calls slip out of direct SDK usage, traces lack the conditioning variables, and the eval becomes a sample of whatever happened to be tagged.

async def call_llm(
    prompt, *,
    surface,              # chat_factual | analytical | narrative | review | multi_turn | behavioral
    user_id,
    financial_context=None, # pre-built state vector, attached to trace
    emotional_state=None,   # deterministic 9-state machine output
    model="auto",             # surface drives default routing
):
    # surface → rate limits + model + Langfuse trace name + eval hook

Six surfaces, and why the eval has to differ on each

A single global rubric flattens distinctions that matter most. A great review response and a great factual chat response have almost nothing in common. The review opens with emotional acknowledgment and pacing; the factual chat answers in two lines and stops. Scoring them under one rubric averages those distinctions away. The eval is surface-aware in three places: dimension weights vary per surface, anchor examples are written per surface, and Layer 1 latency thresholds are set per surface.

The calibration session

Cohen's κ measures rater agreement above chance: higher is better, with 0.6+ conventionally called substantial agreement and anything below 0 worse than random. Even so, headline κ alone hides whether disagreement is fixable. Blind scoring on 28 prompts spanning all six surfaces, 101 ratings total. I report three statistics per dimension because κ on a 1–5 ordinal scale collapses two very different failure modes into one number: quadratic-weighted κ (chance-corrected, penalizes large gaps more than small), MAE, and mean signed error (separates systematic bias from random noise). The bias number is the one that tells you whether to rewrite anchors or collect more data.

Anchor revisions and replay

Root cause was structural, not stylistic. The rubric had anchor examples at 1 / 3 / 5 only, leaving the 2 and 4 buckets to interpolation. That's exactly where rater drift accumulates. Reading the judge's per-prompt rationales surfaced the actual disagreement: I was scoring "competent execution" as 5; the judge consistently said variants of "would reach 5 if it surprised, reframed, or opened a new avenue; competent and direct lands at 4." Added explicit 2 and 4 anchors to all four LLM-judged dimensions, then re-rated the same 30 responses under the new rubric. The 2/4 additions force raters and judge to land in the same place on the in-between scores instead of interpolating two different distributions.

The orchestrator: calibration as a system, not a script

The bugs surfaced in calibration didn't survive the rebuild. Two of them lived inside the eval itself: the rejudge pipeline wrote new scores into one field while leaving the parallel rationale field describing the old ones, and the calibration script re-ran the judge against the prompts instead of reading the persisted scores (so "reproducing" a κ number quietly involved new LLM calls every time). Both trace to the same root cause: n=28 was the manual-rating ceiling, and typing scores into a Python literal caps throughput and reproducibility together. The orchestrator pulls the rubric, prompts, ratings, runs, and results into Postgres with structural invariants that make both bug classes impossible at the database layer. Same prompts, same anchors, same response data. The published κ_w=0.565 reproduces exactly out of the new schema (verified by joining v_latest_ratings × eval_results in the orchestrator's compute_calibration_for_run against the imported historical run).

What changed isn't the math. It's that calibration is now a row in a database, not a script you run. Rate one prompt, the rating persists. Edit an anchor, the version lands as a row in the database. Promote a new rubric, the system rejudges in the background and surfaces the κ delta. No CLI, no manual recompute. The next calibration session is a session, not a project.

What this unblocks

Two things change because of the rebuild: one operational, one architectural.

The numbers in this case study are the first-session baseline: accurate, fully reproducible from the new schema. The continuous-calibration claim is now infrastructural, not aspirational. The next data point lands when the RLHF project picks up the rating loop where this one stops.