NVIDIA Introduces Nemotron-CC: A Massive Dataset for LLM Pretraining

NVIDIA has introduced the discharge of Nemotron-CC, a groundbreaking 6.3-trillion-token English language dataset designed to advance the pretraining of huge language fashions (LLMs). This dataset, derived from Frequent Crawl, goals to raise the accuracy and effectivity of LLMs by way of modern knowledge curation methods, together with the usage of 1.9 trillion tokens of synthetically generated knowledge, in accordance with NVIDIA.

Enhancing LLM Pretraining

NVIDIA’s initiative addresses a essential want in LLM coaching, the place the standard of pretraining datasets performs a pivotal position. Whereas current fashions like Meta’s Llama collection have been based mostly on datasets comprising as much as 15 trillion tokens, the precise composition of those datasets stays largely undisclosed. Nemotron-CC seeks to fill this hole by offering the broader group with a high-quality dataset able to supporting each brief and lengthy token horizon coaching.

Conventional datasets typically sacrifice as much as 90% of information to enhance benchmark accuracies, limiting their utility for in depth coaching. Nemotron-CC, nevertheless, demonstrates tips on how to remodel Frequent Crawl knowledge right into a superior dataset, surpassing even the Llama 3.1 8B mannequin by way of superior strategies comparable to classifier ensembling and artificial knowledge rephrasing.

Important Outcomes

Nemotron-CC’s efficacy is evidenced by its efficiency in numerous benchmarks. When coaching 8B parameter fashions for one trillion tokens, the high-quality subset Nemotron-CC-HQ outperforms main datasets like DCLM, rising MMLU scores by 5.6 factors. Moreover, the whole 6.3-trillion-token dataset matches DCLM on MMLU whereas providing 4 occasions extra distinctive actual tokens. This permits efficient coaching over lengthy token horizons, with Nemotron-CC-trained fashions surpassing Llama 3.1 8B in a number of metrics, together with a 5-point improve in MMLU and a 3.1-point rise in ARC-Problem scores.

Modern Knowledge Curation Strategies

The event of Nemotron-CC concerned a number of key insights. By ensembling totally different model-based classifiers, NVIDIA was in a position to choose a broader array of high-quality tokens. Moreover, rephrasing methods diminished noise and errors, yielding various and invaluable knowledge variants. The choice to disable conventional heuristic filters additional boosted the dataset’s high quality with out compromising accuracy.

NVIDIA utilized its NeMo Curator instrument to extract and refine knowledge from Frequent Crawl, making use of filters for language, deduplication, and high quality classification. This course of was complemented by artificial knowledge technology, contributing roughly two trillion tokens to the dataset.

Future Prospects

Nemotron-CC is positioned as an important useful resource for pretraining state-of-the-art LLMs over various token horizons. NVIDIA plans to increase its choices by releasing extra specialised datasets, together with these centered on particular domains like arithmetic, to additional improve LLM capabilities.

Picture supply: Shutterstock