domains / families / research / nemo-curator

nemo-curator

separate

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

When to use

• Preparing LLM training data from web scrapes (Common Crawl)
• Need fast deduplication (16× faster than CPU)
• Curating multi-modal datasets (text, images, video, audio)
• Filtering low-quality or toxic content
• Scaling data processing across GPU cluster
• **16× faster** fuzzy deduplication (8TB RedPajama v2)
• **40% lower TCO** vs CPU alternatives
• **Near-linear scaling** across GPU nodes

When not to use

• CPU-based data processing
• General ML data processing without curation focus

Skill page

Source