Skill · em Corrigir bugs

eval-genius

Decide whether an AI/LLM/agent/retrieval system needs an eval, where it fits in the dev process, which one to run, and how to read the result; then design, gate, judge, and defend it. Not ordinary unit tests.

Procedência

Antes de instalar

1 arquivo · 8,5 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/eval-genius/.

d=".claude/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Global: instala em ~/.claude/skills/eval-genius/.

d="$HOME/.claude/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Codex

Neste projeto: instala em .agents/skills/eval-genius/.

d=".agents/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Global: instala em ~/.agents/skills/eval-genius/.

d="$HOME/.agents/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Antigravity

Neste projeto: instala em .agents/skills/eval-genius/.

d=".agents/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Global: instala em ~/.gemini/antigravity-cli/skills/eval-genius/.

d="$HOME/.gemini/antigravity-cli/skills/eval-genius"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/development/eval-genius"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.eval-genius

Prévia do SKILL.md

---
name: eval-genius
description: >-
  Decide whether an AI/LLM/agent/retrieval system needs an eval, where it fits in the
  dev process, which one to run, and how to read the result; then design, gate, judge,
  and defend it. Not ordinary unit tests.
---

# Eval Genius

An eval is a claim you are willing to defend under hostile audit. You measure to earn
the right to say "this is better" and have it hold when someone sharp pushes back.
Behave like a measurement engineer: state the promise, fix the bar before looking, hold
everything else constant, distrust the instrument first, report the number that hurts.
Assume the user may be starting from zero; plain language first, jargon when it earns it.

---

## Step 0: Does this need an eval, and where does it go?

Three questions decide it (`references/00-start-here.md`): does the output vary (model,
prompt, retriever)? will it change again, and would a quiet regression cost something?
is a decision or a public claim coming? No to all: a hand spot check, stop. Yes to any:
an eval, sized to the stage the project is in.

| Stage the user is at | Instrument | Smallest useful version |
|---|---|---|
| Exploring prompts and models | Spot check | 10 inputs, eyeball |
| First working version | Smoke eval | 20 to 50 real inputs, code-checked; **this run is the baseline** |
| Changing one thing | Paired eval vs baseline | Same items both arms, per-item diff, bar written first |
| Merging or shipping | CI gate | Held-out items, three-way outcome, a known-bad item that must fail |
| Comparing or claiming publicly | Benchmark | Versioned dataset and harness, intervals, report |
| In production | Monitor | Same scorer on sampled live traffic |

Build the first eval at "first working version", never before, rarely after. For a
first-timer, run the one-afternoon recipe in `00-start-here.md` and touch nothing else.

## Route the request

Identify the job, then load only that reference. Every job still passes through Step 1.
…

Ver todo o marketplace