Skill · em Dados, IA e pesquisa

nemo-curator

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets,…

Procedência

Antes de instalar

3 arquivos · 13,5 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/nemo-curator/.

d=".claude/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Global: instala em ~/.claude/skills/nemo-curator/.

d="$HOME/.claude/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Codex

Neste projeto: instala em .agents/skills/nemo-curator/.

d=".agents/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Global: instala em ~/.agents/skills/nemo-curator/.

d="$HOME/.agents/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Antigravity

Neste projeto: instala em .agents/skills/nemo-curator/.

d=".agents/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Global: instala em ~/.gemini/antigravity-cli/skills/nemo-curator/.

d="$HOME/.gemini/antigravity-cli/skills/nemo-curator"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/data-processing-nemo-curator"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/deduplication.md" "$u/references/deduplication.md" \
  -o "$d/references/filtering.md" "$u/references/filtering.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.nemo-curator

Prévia do SKILL.md

---
name: nemo-curator
description: GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning w…
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]
dependencies: [nemo-curator, cudf, dask, rapids]
---

# NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

## When to use NeMo Curator

**Use NeMo Curator when:**
- Preparing LLM training data from web scrapes (Common Crawl)
- Need fast deduplication (16× faster than CPU)
- Curating multi-modal datasets (text, images, video, audio)
- Filtering low-quality or toxic content
- Scaling data processing across GPU cluster

**Performance**:
- **16× faster** fuzzy deduplication (8TB RedPajama v2)
- **40% lower TCO** vs CPU alternatives
- **Near-linear scaling** across GPU nodes

**Use alternatives instead**:
- **datatrove**: CPU-based, open-source data processing
- **dolma**: Allen AI's data toolkit
- **Ray Data**: General ML data processing (no curation focus)

## Quick start

### Installation

```bash
# Text curation (CUDA 12)
uv pip install "nemo-curator[text_cuda12]"
…

Ver todo o marketplace