Pre-Training Corpus Weighting
The composition of the data a model reads during training, which determines which sources carry the most influence on what it believes and how it speaks.
Analogy Mental model
You can guess how a child will talk by looking at what was on the family bookshelf.
In practice What you actually do
Published accounts of training mixtures are revealing. Filtered web crawl dominates by volume, encyclopaedic sources are deliberately sampled many times over, and a significant share of the highest-quality text was originally identified by human curation rather than by an algorithm alone.
Where people get burned Pitfalls
One detail deserves emphasis: the earliest high-quality web corpus for a major model was assembled by taking external links that real people had shared and upvoted in a public forum. That is the mechanism behind community discussion carrying outsized weight, and it is not a mystery.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →