Skill · em Dados, IA e pesquisa

evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace…

Procedência

Antes de instalar

4 arquivos · 38,4 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/evaluating-code-models/.

d=".claude/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Global: instala em ~/.claude/skills/evaluating-code-models/.

d="$HOME/.claude/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Codex

Neste projeto: instala em .agents/skills/evaluating-code-models/.

d=".agents/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Global: instala em ~/.agents/skills/evaluating-code-models/.

d="$HOME/.agents/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Antigravity

Neste projeto: instala em .agents/skills/evaluating-code-models/.

d=".agents/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Global: instala em ~/.gemini/antigravity-cli/skills/evaluating-code-models/.

d="$HOME/.gemini/antigravity-cli/skills/evaluating-code-models"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/benchmarks.md" "$u/references/benchmarks.md" \
  -o "$d/references/custom-tasks.md" "$u/references/custom-tasks.md" \
  -o "$d/references/issues.md" "$u/references/issues.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.evaluating-code-models

Prévia do SKILL.md

---
name: evaluating-code-models
description: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderb…
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Evaluation, Code Generation, HumanEval, MBPP, MultiPL-E, Pass@k, BigCode, Benchmarking, Code Models]
dependencies: [bigcode-evaluation-harness, transformers>=4.25.1, accelerate>=0.13.2, datasets>=2.6.1]
---

# BigCode Evaluation Harness - Code Model Benchmarking

## Quick Start

BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).

**Installation**:
```bash
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config
```

**Evaluate on HumanEval**:
```bash
accelerate launch main.py \
  --model bigcode/starcoder2-7b \
  --tasks humaneval \
  --max_length_generation 512 \
  --temperature 0.2 \
  --n_samples 20 \
  --batch_size 10 \
  --allow_code_execution \
  --save_generations
```

**View available tasks**:
```bash
python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"
…

Ver todo o marketplace