Memory Limits Help GPT-2 Learn Grammar From Little Data
Researchers trained GPT-2 from scratch with attention capped at a few recent tokens. On 10 million words it beat the standard model on grammar tests, but the edge shrank at 100 million.
TL;DR
- On 10 million words of training text, limiting GPT-2's attention to a few recent tokens raised grammar accuracy from about 61 to 68 percent [^1].
- The edge shrank to a small margin at 100 million words, so the memory limit helps most when training data is scarce [^1].
Background
In a Transformer, self-attention lets each token see nearly every earlier token in a potentially very large context window [^1]. Human working memory is different: it holds only a few items, fades over time and favors what came first and last [^1]. The BabyLM Challenge released training sets of about 10 million and 100 million words, a scale closer to what children hear; it cites estimates that children are exposed to 2 million to 7 million words a year [^2].
What happened
Pranava Madhyastha and Dagmar Adamcová, of City, University of London, the Alan Turing Institute and Grounded Machines, built four such limits into GPT-2 small, a model with about 124 million parameters [^1]. A fixed window lets each token attend only to the last few tokens, with the size set to 4, 5, 7 or 9 after classic estimates of memory capacity: about four "chunks" (Cowan) and "the magical number seven, plus or minus two" (Miller) [^1]. Two decay variants, exponential and logistic, fade the weight of distant tokens instead of cutting them off, and a fourth variant adds learned weights that favor the first and last tokens of a sequence, effects psychologists call primacy and recency [^1]. Where some earlier work added such biases to models that were already trained, every variant here was trained from scratch on the BabyLM sets, with results averaged over three runs [^1].
Grammar was measured with BLiMP, a test of sentence pairs that often differ by a single word or word ending, where the model has to prefer the grammatical version [^1]. With 10 million words, the standard model scored about 61 percent and the fixed-window models around 68 percent, and every constrained variant except two that tested primacy or recency alone beat the baseline significantly [^1]. With 100 million words, the baseline rose to about 71 percent, the exponential-decay and primacy-recency models were no longer significantly different from it, and the best constrained models kept only a small edge [^1]. A window of five tokens was the most consistent performer [^1]. No variant helped with island effects, the limits on moving words within a sentence, or NPI licensing, the rule that words like "ever" need a negative context; scores there were often barely above chance [^1].
The authors also tested how well each model's surprisal, a measure of how unexpected a word is, predicts human eye-tracking, reading-time and brain-signal data on 1,725 sentences [^1]. At 10 million words, the baseline reached an average score of about 3.2, while the best fixed-window models (sizes 7 and 9) and the models with primacy and recency biases came close to twice that [^1]. At 100 million words, this fit got worse for every model, constrained or not [^1]. Inside the network, the five-token window produced specialized attention heads: on "The trophy would not fit in the brown suitcase because it was too small," two second-layer heads focused on "trophy," "fit" and "suitcase," while the baseline's heads stayed diffuse [^1].
Why it matters
The result cuts against a common design direction. The authors note that a large body of NLP work tries to move away from locality bias, and their own starting assumption was that heavy constraints would hurt; instead, a five-token window did well even on argument structure and binding, which are not purely local [^1]. They conclude that hard cognitive constraints "actively scaffold learning rather than hindering it," and that the findings "push against the contemporary trends towards longer contexts and weaker inductive biases" [^1]. An inductive bias is an assumption built into the architecture that steers what a model learns from limited data.
The authors spell out the limits themselves. They tested only 10 million and 100 million words and make no claim about frontier models trained on trillions of tokens [^1]. They tested only English, and say the benefit may look different in languages that rely on case marking, free word order or head-final structures [^1]. Their model of memory only controls how far back a token can see, while real working memory also involves interference between stored items [^1]. And the weak results on island effects raise the possibility that limits based on linear distance get in the way of rules that depend on a sentence's overall structure [^1].
Practical example
Say Priya runs a small psycholinguistics lab and needs a language model for an eye-tracking study, trained only on a child-sized corpus so the results say something about learning. Priya picks GPT-2 small and the 10-million-word BabyLM set [^1]. Trained the standard way, a model like this lands at about 61 percent on BLiMP, and its surprisal is a relatively weak guide to where readers slow down [^1].
Following the paper, Priya changes one thing: an attention mask that limits each token to a window of the five most recent tokens, with all training settings unchanged [^1]. In the paper's runs, fixed windows lifted grammar accuracy to around 68 percent, and windows of 7 or 9 tokens nearly doubled the fit to human reading data [^1]. For Priya's study, that means surprisal values that track reading effort more closely, without collecting more text.
The limits show up in the details. If Priya's sentences test whether "ever" appears in a negative context, the window won't help: models stayed near chance on that [^1]. If a later grant pays for 100 million words, the standard model largely catches up on grammar and the fit to reading data drops for every model [^1]. And if Priya wants to repeat the study in German, which marks case and allows freer word order, the paper offers no evidence either way [^1].
Related gear
We recommend this book because it builds a GPT-style model and its causal attention mask step by step in code, the mask this study narrows to a few recent tokens.