Large Language Gibbs Resamples LLM Answers for Consistency
Edinburgh researchers turn an LLM's next-token predictions into a Gibbs sampler. Judging 64 math claims together this way lifted Llama-3.1-8B's accuracy to 0.895, against 0.724 for a prior method.
TL;DR
- Edinburgh researchers propose Large Language Gibbs, which resamples one variable at a time with an LLM instead of generating structured output in one pass [^1].
- On a true-or-false check of math answers, it lifted Llama-3.1-8B to 0.895 accuracy, against 0.724 for an earlier consistency method [^1].
Background
Language models generate text autoregressively: each token is drawn from a distribution conditioned on the tokens before it [^2]. When a model must produce several related values at once, such as the fields of a table row or the answers to a set of questions, that left-to-right order can distort the result through effects such as recency bias [^1]. Gibbs sampling, a standard statistical method, instead updates one variable at a time given the current values of all the others [^1].
What happened
Sanghyeok Choi, Henry Gouk and Esmeralda S. Whitammer of the University of Edinburgh's School of Informatics posted the method on arXiv on June 17 [^1]. It first fills in all variables in one autoregressive pass. Then it repeatedly picks one variable, lists the others in the prompt in a random order, and lets the LLM's next-token distribution draw a new value for it [^1]. The shuffling matters: it removes the bias of any fixed order, and the chain settles into a stationary distribution that the authors describe as "a compromise between all local conditionals" [^1]. Two variants propose a random candidate instead and let the model accept or reject it, either by picking the more plausible value or by deciding whether to place a bet on it [^1].
The first test was sampling numbers. Earlier work has repeatedly found that LLMs cannot sample well from simple distributions; the authors tested integers drawn uniformly from 0 to 99 and a standard Gaussian [^1]. With the base Llama-3.1-8B, Gibbs sampling was more faithful than drawing one sample per query and less autocorrelated than generating a whole batch in one completion [^1]. With the instruction-tuned model, the bias from instruction tuning carried over into Gibbs sampling, and the betting variant appeared to accept or reject at random [^1].
The main test was consistent reasoning on TruthfulQA and GSM8K-Verification, where the model labels candidate answers as true or false, four claims per question [^1]. With 64 claims judged together, Gibbs reached 0.736 and 0.895 with Llama-3.1-8B, against 0.702 and 0.724 for ICM, an earlier consistency method run without its original label-based checks, and 0.593 and 0.524 for zero-shot answers [^1]. With OLMo-3-32B, Gibbs scored 0.743 and 0.840, against 0.662 and 0.750 for ICM [^1]. A third application used Gibbs-generated synthetic data as a prior for learning Bayesian network structure from variable descriptions, and generally beat a uniform prior on four datasets [^1].
Why it matters
Many LLM tasks need a set of values that fit together, not one good answer. The paper treats the generation order as a removable source of error rather than a given [^1]. The reasoning results come without labels: the method uses only the model's own conditional beliefs to make its answers agree, and in every setting tested, accuracy rose as more claims were judged together [^1]. The authors call LLM conditionals in MCMC "a practical alternative to one-pass generation" for this kind of inference [^1].
The method has clear limits. The chain is guaranteed to sample the symmetric joint distribution the authors aim for only under an assumption they say "is unlikely to hold exactly", though shuffling still approximates it better than any fixed order [^1]. The method costs more compute than one-pass generation and may need many iterations to converge. The experiments used models of up to 32 billion parameters [^1]. On one H100 GPU, a run over 256 questions took up to 40 minutes with Llama-3.1-8B, and OLMo-3-32B usually took four to five times longer [^1]. For safety-critical uses, the authors warn that results should not be trusted "without proper validation" [^1].
Practical example
Say Ana runs a small tutoring app that checks students' answers to math word problems with an open model. Asking Llama-3.1-8B about each answer on its own is the zero-shot setup, which reached 0.524 accuracy on GSM8K-Verification in the paper [^1].
Ana tries Large Language Gibbs instead. The app groups 16 problems with four candidate answers each, 64 true-or-false labels in all, and fills them in with one pass [^1]. Then it keeps picking one label, shuffles the other 63 labeled claims into the prompt and lets the model relabel it. Seeing its own other verdicts encourages the model to keep them consistent, much as the paper's star example uses the fact that "Polaris" and "North Star" name the same star [^1]. After a burn-in phase, the app collects 25 samples per claim and takes the majority vote, as in the paper's setup [^1].
In the paper, that configuration lifted Llama-3.1-8B to 0.895 [^1]. The price is time: on one H100, a 256-question run at this size took about 40 minutes [^1]. Ana also keeps checking a sample of verdicts against a teacher's grading: the method pushes answers to agree with each other and with the model's own beliefs, which is not the same as being right [^1].
Related gear
We recommend this book because it explains Gibbs sampling and other Monte Carlo methods from first principles, and the paper cites it for the Gibbs result it builds on.