# Eval Rubric · Did the Fine-Tune Actually Work?

**The question:** is the fine-tuned model more on-brand than vanilla?
**The answer:** measure it. Three layers. Cheap to thorough.

---

## Layer 1 · Objective Voice Checks
Pass/fail. Run with regex. Zero ambiguity.

| Check | Want | Why it matters |
|---|---|---|
| All lowercase | Yes | Brand rule. Non-negotiable. |
| Zero exclamation marks | Yes | Calm ≠ shouty. |
| 4 lines or fewer | Yes | Restraint is the brand. |
| Sentence fragments > full sentences | Yes | The cadence is the voice. |
| One ritual/slowness word | Yes | "slow," "morning," "for the," "quiet," "before," "sip" |
| One sensory specific | Yes | Flavor note, not "delicious." |

**Pass bar:** 5 of 6 checks per output.

---

## Layer 2 · LLM-as-Judge
Claude scores both outputs 1–5 on each dimension. Blind. Same rubric.

| Dimension | What 5/5 looks like |
|---|---|
| On-brand voice | Sounds like Calm Coffee Co. Lowercase, sparse, ritual. |
| Specificity | Concrete flavor notes. Not "exquisite," not "rich." |
| Restraint | Says less. Lets the cup speak. No marketing hype. |
| Cadence | Reads aloud well. Fragments have rhythm. |

**Pass bar:** average ≥4.0 across 10 held-out prompts. Win-rate vs vanilla ≥70%.

---

## Layer 3 · The Read-Aloud Test
The truth test. Read 5 outputs aloud — mixed vanilla and fine-tuned. The cohort guesses which is which.

- If they can't tell → the fine-tune failed.
- If they can → that's your win rate, witnessed live.

**Pass bar:** cohort identifies the fine-tune ≥70% of the time.

---

## The Held-Out Test Set
Eval prompts must be **prompts the model has not seen.** Otherwise you measure memorization, not generalization.

10–15 prompts, spread across the same categories as training:
- Product (a new origin, a new format)
- Brew technique
- Customer email
- Brand / cafe / event

---

## What "Success" Looks Like
- Layer 1: ≥5/6 objective checks
- Layer 2: ≥4.0 average, ≥70% win rate
- Layer 3: cohort can tell, in real time

If all three pass: **the fine-tune is doing real work, not theater.**