Skill · em Dados, IA e pesquisa

nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible…

Procedência

Antes de instalar

5 arquivos · 46,6 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/nemo-evaluator-sdk/.

d=".claude/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Global: instala em ~/.claude/skills/nemo-evaluator-sdk/.

d="$HOME/.claude/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Codex

Neste projeto: instala em .agents/skills/nemo-evaluator-sdk/.

d=".agents/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Global: instala em ~/.agents/skills/nemo-evaluator-sdk/.

d="$HOME/.agents/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Antigravity

Neste projeto: instala em .agents/skills/nemo-evaluator-sdk/.

d=".agents/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Global: instala em ~/.gemini/antigravity-cli/skills/nemo-evaluator-sdk/.

d="$HOME/.gemini/antigravity-cli/skills/nemo-evaluator-sdk"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/evaluation-nemo-evaluator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/adapter-system.md" "$u/references/adapter-system.md" \
  -o "$d/references/configuration.md" "$u/references/configuration.md" \
  -o "$d/references/custom-benchmarks.md" "$u/references/custom-benchmarks.md" \
  -o "$d/references/execution-backends.md" "$u/references/execution-backends.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.nemo-evaluator-sdk

Prévia do SKILL.md

---
name: nemo-evaluator-sdk
description: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarkin…
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Evaluation, NeMo, NVIDIA, Benchmarking, MMLU, HumanEval, Multi-Backend, Slurm, Docker, Reproducible, Enterprise]
dependencies: [nemo-evaluator-launcher>=0.1.25, docker]
---

# NeMo Evaluator SDK - Enterprise LLM Benchmarking

## Quick Start

NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).

**Installation**:
```bash
pip install nemo-evaluator-launcher
```

**Set API key and run evaluation**:
```bash
export NGC_API_KEY=nvapi-your-key-here

# Create minimal config
cat > config.yaml << 'EOF'
defaults:
  - execution: local
  - deployment: none
  - _self_

execution:
  output_dir: ./results

target:
  api_endpoint:
    model_id: meta/llama-3.1-8b-instruct
    url: https://integrate.api.nvidia.com/v1/chat/completions
    api_key_name: NGC_API_KEY
…

Ver todo o marketplace