A large language model is a type of AI system trained on massive volumes of text to generate human-like language, answer questions, and perform reasoning tasks. LLMs are trained on enormous datasets, much of which comes from public web data assembled into corpora such as C4, RefinedWeb, and Dolma.

Web data is used both for initial model training and for retrieval-augmented generation, where a model pulls in live web content to answer a specific query.

The size of the training dataset isn't the only thing that matters. Data quality and recency matter too.