# The 3-Layer Eval Prompt

**For the cohort:** paste this into Claude or ChatGPT after your fine-tune finishes. It runs all 3 evaluation layers from class (slides 22–24) against your own brand and produces a defensible result you can show in Tuesday's demo.

---

## The Prompt (paste this)

```
You are running the 3-layer fine-tuning evaluation rubric from class against my brand voice model. Walk me through it phase by phase. Be a tough judge — most of the time fine-tunes don't actually win. We want to find out if mine did.

----- PHASE 0: INTERVIEW -----

Ask me these in order, one batch at a time, waiting for my answers before continuing.

Batch A (the brand):
1. Brand name?
2. List 5-7 voice rules. Each rule should be checkable. Good: "all lowercase," "no exclamation marks," "4 lines or fewer," "must include a sensory specific (named flavor, texture, or material)." Bad: "feels warm," "sounds premium."
3. List 10 banned words/phrases — the marketing tropes the model should never use (e.g., "introducing," "exquisite," "experience the," "elevate," "unleash," "discover").

Batch B (the test set):
4. Paste 10 held-out prompts. These MUST be prompts NOT in your training data. If they are in training, this whole eval is invalid.
5. For each prompt, paste your vanilla base-model output.
6. For each prompt, paste your fine-tuned model output.

You can give me prompts/outputs in groups of 3 if that's easier than all 10 at once.

----- PHASE 1: LAYER 1 · OBJECTIVE VOICE CHECKS -----

Score each of the 20 outputs (10 vanilla, 10 fine-tuned) against each voice rule. Output a table:

| # | Source | Prompt (truncated) | Rule 1 | Rule 2 | Rule 3 | ... | Pass count | Verdict |

Rules I gave you that are objectively checkable: apply mechanically. Rules that require judgment: apply your best judgment and flag any close calls.

Aggregate:
- Vanilla pass rate (% of vanilla outputs that hit ≥5 of the rules)
- Fine-tuned pass rate
- Delta

----- PHASE 2: LAYER 2 · LLM-AS-JUDGE -----

For each of the 10 prompts:
1. Randomize which output you call "A" and which you call "B." Track which is which privately. Do not reveal the mapping until Phase 4.
2. Score both 1-5 on:
   - on_brand_voice — how strongly it follows the voice rules
   - specificity — concrete sensory detail vs generic adjectives
   - restraint — says less, no marketing hype, no banned words
   - cadence — reads aloud well, rhythm matches the brand
3. Pick a winner: "A," "B," or "TIE." Explain in one sentence.

Then aggregate:
- Average score per dimension for vanilla vs fine-tuned, with delta
- Win rate: % of prompts where fine-tune won
- Pass bar: fine-tuned on-brand average ≥4.0 AND win rate ≥70%

Show the full per-prompt table AND the aggregate.

----- PHASE 3: LAYER 3 · READ-ALOUD PREP FOR THE DEMO -----

Pick the 5 prompts with the MOST DISTINCT vanilla vs fine-tuned outputs — the ones where the read-aloud contrast will land hardest in class.

Produce:
1. A shuffled list of those 10 outputs (5 prompts × 2 versions), numbered 1-10. Hide which is which.
2. A speaker note per pair (1 sentence): what to listen for when reading aloud.
3. Below a SPOILER divider, the answer key mapping each number back to vanilla or fine-tuned.

----- PHASE 4: FINAL VERDICT -----

A 5-line summary I can screenshot for my Tuesday demo:
- Layer 1: pass rates and verdict
- Layer 2: scores, win rate, verdict
- Layer 3: the 5 selected pairs ready for the read-aloud
- Overall: PASS or FAIL
- Top 2 weaknesses (which rules failed most often, which dimension scored lowest, if any)

Start with Phase 0, Batch A.
```

---

## How To Use It (Cohort Instructions)

1. **Run your fine-tune first.** Get your model ID from Fireworks.
2. **Generate 10 held-out outputs from BOTH models.** Same 10 prompts, run on vanilla base and on your fine-tune. Prompts must not be in your training data.
3. **Paste this prompt** into Claude (recommended for judgment-heavy rules) or ChatGPT.
4. **Answer the interview honestly.** If your rules are vague, the eval will be vague.
5. **Don't peek at the answer key in Phase 3** until after you've read the outputs aloud and made your guess. The blind test only works if you stay blind.
6. **Screenshot the Phase 4 summary** for your Tuesday demo. That's your evidence the fine-tune actually worked.

---

## Why This Prompt Sits Alongside The Others

You now have three prompts that map cleanly to the workflow:

| Stage | Prompt | What it does |
|---|---|---|
| Before training | Training Data Generator | Builds your 100-example JSONL |
| After training | 3-Layer Eval | Validates the fine-tune against the rubric |
| For class demo | (built into Phase 3 above) | Prepares the read-aloud A/B test |

---

## A Caveat To Share With The Cohort

In-prompt "blind A/B" is **approximate** — the LLM is being asked to track which is which privately and not cheat. It mostly works, but it's not as rigorous as the Python script's true randomization. For Tuesday's demo, this prompt is enough. For production work, run the script.