Skill · em Dados, IA e pesquisa

huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need…

Procedência

Antes de instalar

5 arquivos · 73,8 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/huggingface-tokenizers/.

d=".claude/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Global: instala em ~/.claude/skills/huggingface-tokenizers/.

d="$HOME/.claude/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Codex

Neste projeto: instala em .agents/skills/huggingface-tokenizers/.

d=".agents/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Global: instala em ~/.agents/skills/huggingface-tokenizers/.

d="$HOME/.agents/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Antigravity

Neste projeto: instala em .agents/skills/huggingface-tokenizers/.

d=".agents/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Global: instala em ~/.gemini/antigravity-cli/skills/huggingface-tokenizers/.

d="$HOME/.gemini/antigravity-cli/skills/huggingface-tokenizers"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizers"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/algorithms.md" "$u/references/algorithms.md" \
  -o "$d/references/integration.md" "$u/references/integration.md" \
  -o "$d/references/pipeline.md" "$u/references/pipeline.md" \
  -o "$d/references/training.md" "$u/references/training.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.huggingface-tokenizers

Prévia do SKILL.md

---
name: huggingface-tokenizers
description: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance…
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]
dependencies: [tokenizers, transformers, datasets]
---

# HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

## When to use HuggingFace Tokenizers

**Use HuggingFace Tokenizers when:**
- Need extremely fast tokenization (<20s per GB of text)
- Training custom tokenizers from scratch
- Want alignment tracking (token → original text position)
- Building production NLP pipelines
- Need to tokenize large corpora efficiently

**Performance**:
- **Speed**: <20 seconds to tokenize 1GB on CPU
- **Implementation**: Rust core with Python/Node.js bindings
- **Efficiency**: 10-100× faster than pure Python implementations

**Use alternatives instead**:
- **SentencePiece**: Language-independent, used by T5/ALBERT
- **tiktoken**: OpenAI's BPE tokenizer for GPT models
- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)

## Quick start

### Installation

```bash
# Install tokenizers
pip install tokenizers
…

Ver todo o marketplace