Paul Nguyen

Lab note #15: Harness Bake-off framework

·

The ask

Set up a bake-off where Claude, ChatGPT and OpenCode each get identical instructions from the GitHub repo, follow the same tracked change process, and can be compared fairly.

What changed

  • New bake-off protocol in the repo (docs/BAKEOFF.md): rules for contestants, scoring and commands
  • Round 1 task (a Featured Projects page built only from facts already on the site), an identical kickoff prompt for every harness, and a grading sheet
  • A bake-off script that starts and finishes each run, allows only one harness at a time, and measures process automatically: time, snapshot discipline, typed commits, Lab Note, rollbacks, staying in scope and a live entry
  • New Harness Bake-off and Round 1 pages that update themselves as runs start and finish; the Scoreboard links to them
  • ChatGPT added as a recognized harness name

How

SSH and WP-CLI, a Python scoring engine on the host, and GitHub for the instructions every harness reads.

What worked, what didn’t

  • A full dry run in a throwaway round worked end to end and scored correctly (8 of 10 process points, minus the Lab Note it deliberately skipped). It was then rolled back, which shows up honestly on Claude’s scoreboard line.
  • Runs have to be one at a time: the tracking system exports the whole site, so two harnesses working at once would get each other’s changes credited to them. The script enforces this with a lock.
  • Fairness note: this Claude session wrote the task, so the Claude entry should come from a fresh session that sees only the kickoff prompt, like the others.

Change record

Harnessclaude
Date2026-10-09 17:20
Latest snapshot20261009-171840
Undo this sessionops/rollback.sh --git bb322dc

Commits

  • 68418f1 ops: remove bake-off dry-run test round
  • 032e58a rollback: Rollback to 65cb97c (ops: Harness Bake-off framework, Round 1 task and kickoff prompt, Bake-off pages)
  • a4e33c0 bakeoff: claude finishes round-99
  • 0d87e10 content: test entry page
  • 96f3b72 bakeoff: claude starts round-99
  • 65cb97c ops: Harness Bake-off framework, Round 1 task and kickoff prompt, Bake-off pages
  • 4293454 scoreboard: refresh after Lab note #14