Skill · em Dados, IA e pesquisa

distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Procedência

Antes de instalar

5 arquivos · 27,8 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/distributed-llm-pretraining-torchtitan/.

d=".claude/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Global: instala em ~/.claude/skills/distributed-llm-pretraining-torchtitan/.

d="$HOME/.claude/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Codex

Neste projeto: instala em .agents/skills/distributed-llm-pretraining-torchtitan/.

d=".agents/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Global: instala em ~/.agents/skills/distributed-llm-pretraining-torchtitan/.

d="$HOME/.agents/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Antigravity

Neste projeto: instala em .agents/skills/distributed-llm-pretraining-torchtitan/.

d=".agents/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Global: instala em ~/.gemini/antigravity-cli/skills/distributed-llm-pretraining-torchtitan/.

d="$HOME/.gemini/antigravity-cli/skills/distributed-llm-pretraining-torchtitan"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/model-architecture-torchtitan"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/checkpoint.md" "$u/references/checkpoint.md" \
  -o "$d/references/custom-models.md" "$u/references/custom-models.md" \
  -o "$d/references/float8.md" "$u/references/float8.md" \
  -o "$d/references/fsdp.md" "$u/references/fsdp.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.distributed-llm-pretraining-torchtitan

Prévia do SKILL.md

---
name: distributed-llm-pretraining-torchtitan
description: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Model Architecture, Distributed Training, TorchTitan, FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel, Float8, Llama, Pretraining]
dependencies: [torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0]
---

# TorchTitan - PyTorch Native Distributed LLM Pretraining

## Quick start

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.

**Installation**:
```bash
# From PyPI (stable)
pip install torchtitan

# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt
```

**Download tokenizer**:
```bash
# Get HF token from https://huggingface.co/settings/tokens
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
```

**Start training on 8 GPUs**:
```bash
CONFIG_FILE="./torchtitan/models/llama3/train_configs/llama3_8b.toml" ./run_train.sh
```

## Common workflows
…

Ver todo o marketplace