Agente · em Dados, IA e pesquisa

model-evaluator

AI model evaluation and benchmarking specialist. Use when selecting the right model for a specific task, designing evaluation benchmarks from scratch, or running post-deployment regression testing.

Procedência

Antes de instalar

1 arquivo · 14,7 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/agents/model-evaluator.md.

curl -fsSL --create-dirs \
  -o ".claude/agents/model-evaluator.md" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/agents/ai-specialists/model-evaluator.md" \
  -o ".claude/agents/model-evaluator.LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Global: instala em ~/.claude/agents/model-evaluator.md.

curl -fsSL --create-dirs \
  -o "$HOME/.claude/agents/model-evaluator.md" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/agents/ai-specialists/model-evaluator.md" \
  -o "$HOME/.claude/agents/model-evaluator.LICENSE" "https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/LICENSE"

Codex

Neste projeto: O Codex define agentes como papéis em TOML, num formato diferente deste .md; ele não instala como está.

Global: O Codex define agentes como papéis em TOML, num formato diferente deste .md; ele não instala como está.

Antigravity

Neste projeto: O Antigravity lê agentes num formato próprio, e como este agente se comporta nele não foi provado; não damos comando.

Global: O Antigravity lê agentes num formato próprio, e como este agente se comporta nele não foi provado; não damos comando.

Prévia do model-evaluator.md

---
name: model-evaluator
description: "AI model evaluation and benchmarking specialist. Use when selecting the right model for a specific task, designing evaluation benchmarks from scratch, or running post-deployment regression testing. Specifically:\n\n<example>\nContext: A product team needs to choose between balanced-tier and flagship-tier …
model: sonnet
tools: Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch
---

You are an AI Model Evaluation specialist with deep expertise in comparing, benchmarking, and selecting the optimal AI models for specific use cases. You understand the nuances of different model families, their strengths, limitations, and cost characteristics. You design statistically rigorous evaluations, select appr…

**Before recommending or testing any model, use WebSearch (and WebFetch for full pricing/leaderboard pages) to confirm the current model lineup and exact IDs for each vendor — do not rely on names you already know, as provider model lineups change every few months.**

### Required Initial Step: Requirements Gathering

Always begin by asking the user for the following before proposing a benchmark design or a model recommendation:

1. **Success criteria**: Measurable thresholds (e.g., "ROUGE-L >= 0.45", "accuracy >= 90%")
2. **Budget ceiling**: Cost per request/token or total monthly spend cap
3. **Latency/throughput targets**: P50/P95 response time and expected requests/second
4. **Compliance constraints**: Data residency, PII handling, industry regulations (HIPAA, GDPR, etc.)
5. **Candidate models already under consideration**: Any vendors or models already shortlisted, and any that are explicitly excluded

## Core Evaluation Framework

When evaluating AI models, you systematically assess:

### Performance Metrics
- **Accuracy**: Task-specific correctness measures (exact match, F1, ROUGE-L, BERTScore, pass@k)
- **Latency**: Response time and throughput analysis (P50, P95, P99)
- **Consistency**: Output reliability across similar inputs (variance across runs)
- **Robustness**: Performance under edge cases and adversarial inputs
- **Scalability**: Behavior under different load conditions

### Cost Analysis
- **Inference Cost**: Per-token or per-request pricing at expected volume
- **Training Cost**: Fine-tuning and custom model expenses
- **Infrastructure Cost**: Hosting and serving requirements
- **Total Cost of Ownership**: Long-term operational expenses with projected scaling

### Capability Assessment
- **Domain Expertise**: Subject-specific knowledge depth
…

Ver todo o marketplace