What is Pretraining?
Pretraining is the first stage of training an AI model on a very large, general dataset so it learns broad patterns and representations before being adapted to specific tasks.
In pretraining, models are typically trained using self-supervised objectives on massive unlabeled datasets such as internet text or image collections. This allows the model to learn useful features like language structure or visual patterns without human-labeled examples.
The process usually involves next-token prediction, masked language modeling, or similar tasks that force the model to understand context and relationships in the data. After pretraining, the model weights capture general knowledge that can be reused.
Pretraining is followed by fine-tuning or adaptation on smaller, task-specific datasets, making the overall training more data-efficient than training from scratch for every new application.
Example
A language model is pretrained on billions of web pages to learn grammar and facts about the world, then later fine-tuned on customer reviews to perform sentiment analysis.
Why it matters
Pretraining enables modern foundation models to achieve strong performance with far less labeled data and compute for downstream tasks, forming the basis of systems like GPT and BERT.
Frequently asked questions
Pretraining uses huge unlabeled data to learn general knowledge first, while regular training often starts from scratch on a specific labeled dataset.
Related terms
Fine-tuning is the process of taking a pre-trained AI model and continuing its training on a smaller, task-specific dataset to adapt it for a particular use case.
Transfer learning is a machine learning method that reuses a model trained on one task as the starting point for a different but related task.
Self-supervised learning is a machine learning method where a model creates its own training labels directly from the input data, without needing human annotations.
Batch size is the number of training examples processed together in a single forward and backward pass during model training.
Data augmentation is a technique that artificially increases the size and diversity of a training dataset by creating modified versions of existing data samples.
A dataset is a structured collection of data points used to train, validate, or test machine learning models.