← all content

Week 2 of 5 · Class 4 of 10 · Thu 5/28

Build a custom LLM

ChatGPT and Claude are generalized. Today we make one specific to you. A live Fireworks AI fine-tune, then your new team picks a project to demo Tuesday 6/2.

View the Class 4 deck on Gamma →

Rewatch video

Embed not loading? Watch on YouTube →

Today's schedule

4:00–4:20Custom LLM framing
4:20–4:45Live fine-tune walkthrough
4:45–5:20New teams + Week 2 project kickoff
5:20–5:30Q&A + close

Live demo: fine-tune Calm Coffee Co.

Watch a custom LLM get built in real time: 100 examples, about $2, about 12 minutes. The whole point is the side-by-side: same prompt, vanilla model and fine-tuned model, read both aloud. The voice transfer is the demo.

~25 min · live on Fireworks AI

Watch the build · on Fireworks

  1. The training data: 100 JSONL examples in Calm Coffee Co. voice. One per line: system + user + assistant.
  2. The dashboard: upload JSONL, pick a base model (Llama 3.1 8B or Qwen 2.5 7B), defaults for hyperparams, click train, get a model ID.
  3. The side-by-side: in the playground, run the same prompt against vanilla and your fine-tuned model. Read both aloud.

How we know fine-tuned is better: 3 layers

  1. Objective voice checks. Pass/fail regex: all lowercase, zero exclamation marks, 4 lines or fewer, sentence fragments, one ritual word, one sensory specific. Pass bar: 5 of 6.
  2. LLM-as-judge. Claude scores both outputs 1–5 on voice, specificity, restraint, cadence. Pass bar: average ≥4.0 across 10 held-out prompts; win rate ≥70%.
  3. Read-aloud test. Read 5 outputs aloud, mixed. The cohort identifies the fine-tune ≥70% of the time.

Week 2 build: a custom LLM as a team

Your new team fine-tunes its own custom LLM and brings a side-by-side vs. vanilla to demo Tuesday June 2. Platform: Fireworks AI or Together AI. Base model: Llama 3.1 8B or Qwen 2.5 7B. Training data: 30 examples minimum, 100 recommended. Budget under $20 per team.

Pick one direction
🌭 Absurd brand. Invent a deliberately ridiculous brand. Luxury hot dogs. Wellness app for raccoons. SaaS for medieval knights.
🤖 Teammate bot. Fine-tune on the writing of ONE team member (LinkedIn, Substack, blog). At the demo, the cohort guesses who each bot was trained on. Needs that teammate's permission.

Scope it with your team · in class

  1. Whose voice has a quirk you can name in one sentence? If you can't name it, the fine-tune has nothing to learn.
  2. Is the voice different enough that vanilla will obviously fail? If Claude already nails it, there's no demo moment.
  3. Pick ONE output category: product copy, Slack messages, subject lines, social posts. Not five.
  4. Where do your 100 examples come from? Claude generates them in-voice, or you scrape something real.
  5. Lock the ONE prompt you'd run live that makes the contrast obvious in 30 seconds.

Pre-class setup needed

Fireworks account · payment method · $5 credit.

Ending criteria: use them in Tuesday's demo

FUN: did your custom LLM make the cohort laugh?
CREATIVITY: what unexpected thing did the fine-tune teach you?
COLLABORATION: how did your team split the work? Gather examples together? Kick off the fine-tune together?

Week 2 teams

1Jeff Lash + Nikoo Beyzaei + Sabrina Abhyankar + Daliso
2Daniela Perilla + Ramin Talaie + Patricia + Michael Krafft
3Jocelyn Baun + Chris Thorne + Dorincy Shen + Kevin Ko
4Elizabeth + John Dufresne + Aviral Gupta + Arpit

Project files

Everything for the live walkthrough and your Week 2 team project: training data, eval rubric, judge script, prompts.

  1. calm-coffee-co-training.jsonl: 100 examples, the dataset from the live demo.
  2. training-data-generator-prompt.md: paste into Claude to build your own 100-example JSONL.
  3. calm-coffee-eval-rubric.md: the 3-layer rubric.
  4. calm-coffee-eval-template.csv: 15 held-out prompts.
  5. three-layer-eval-prompt.md: paste into Claude after your fine-tune finishes.
  6. calm-coffee-judge.py: Python LLM-as-judge with blind A/B randomization.

Resources

LifeArchitect.ai Models Table850+ tracked LLMs with parameters, tokens, training cost. Updated monthly.
Fireworks AI Fine-tuning DocsUpload JSONL, pick base models, run training jobs.
Together AI Fine-tuningAlternative serverless platform for fine-tuning open-source models.
OpenAI Fine-tuning SunsetOpenAI sunset fine-tuning for new users on May 8, 2026 (context on the shift).

Before you log off