An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings.
Our data are temporal biographies: synthetic people whose employer, university, city and other facts change over the years. Each record states the facts that hold in one year and starts with that year, e.g. "In 2005, …". We pre-train GPT-2 style models at three sizes, 110M, 350M and 682M parameters, on the same records in two orders. CD (chronological, dated) reads them year by year, from 2000 to 2024. SD (shuffled, dated) reads them in random order.
Each year's knowledge has its own loss landscape over the same parameter space. A model that knows a year well sits low in a valley of that year's landscape: this is a specialist of that year. CD tracks the newest year, so it moves into a valley of each year it reads. Every year has many such specialists. Where the specialists of different years overlap, a model knows all those years at once; this is where the generalists lie, and where SD ends up. Figure 1 shows the measured landscape around the trained models, and Figure 2 shows how it changes during training.
The paper’s Figure 3(a) is an illustration; here every point is a trained model we measured.
A trained model is a point in parameter space. We pick three trained models O, A and B and cut the plane through them, θ(a, b) = O + a(A − O) + b(B − O). Every point on this plane is a model, and we test each one.
Left: the loss on one year's knowledge (height) and the accuracy on it (color), over the plane. Right: the same plane seen from above. The specialists of a year are the models that know that year well; generalists lie where the specialists of different years overlap. Press play to move through the years.
Strictly not a landscape: it is the loss on the linear interpolation between CD and SD, as training goes on.
CD and SD start from the same random weights and train on the same records; the timeline plays their training.
Left: mixing the weights of CD and SD gives the models in between (the line across); height is their loss, lower is better. Early on the line is flat: the two models are still connected. Later it rises in the middle: a wall now separates them.
(a) Parameter space: a model is its weights, one point in parameter space, flattened onto a 2D map with PCA. Both models keep moving away from where they started. This map is also the floor of the 3D view.
(b) Function space: a model is its loss on each of the 25 years, one point in a 25-dimensional space, flattened onto a 2D map with PCA. CD goes round in a circle, once per epoch: it learns 2000, then 2001, and so on to 2024, then starts over. SD stays in one place: it knows every year all the time.