AdamW Lacks a Convergence Proof for Heavy-Tailed Gradient Noise
A short paper asks whether AdamW can match the convergence rates Lion and Muon have under heavy-tailed gradient noise, and shows where the proof gets stuck.
TL;DR
- An open-problem paper asks whether AdamW, the standard LLM optimizer, provably converges as fast as Lion and Muon under heavy-tailed gradient noise [^1].
- The authors prove partial results, including a constructed case where AdamW's memory of rare outliers shrinks its steps while the gradient stays large [^1].
Background
An optimizer decides how a neural network's weights change after each batch of training data. AdamW, Adam with decoupled weight decay, was proposed by Ilya Loshchilov and Frank Hutter of the University of Freiburg and is implemented in TensorFlow and PyTorch [^2]. Convergence proofs say how fast an optimizer reaches a point where the gradient is small. Most proofs for AdamW assume gradient noise with finite variance [^1]. Heavy-tailed noise produces rare, very large values that can break that assumption.
What happened
Five researchers from Nanjing University, Zhejiang University and Fudan University posted the paper on arXiv on June 22 [^1]. It is written as an open problem. The authors call AdamW the de facto optimizer for training large language models, but note that its theory still lives mostly in finite-variance regimes, while empirical evidence indicates that gradient noise in LLM pretraining is typically heavy-tailed [^1]. Sign-based optimizers such as Lion and Muon already have sharp convergence rates under such noise, and AdaGrad can converge under it too. For AdamW, no rigorous convergence theory exists in this regime [^1].
The question is deliberately narrow: under the same assumptions, can AdamW match the rate proven for sign-based methods? Already for the case without weight decay, the authors ask for either a proof that it can, or a constructed instance showing that no such guarantee is possible [^1]. They offer partial results. A positive one shows AdamW reaching the target rate in a weighted measure of progress, but only when its two running averages, the momentum and the average of squared gradients, use the same memory length. Practical AdamW usually gives the second one a far longer memory [^1]. A barrier result shows that without any averaging, AdamW reduces to SignSGD, and some unbiased heavy-tailed noise can then push it the wrong way on average. The authors stress that this is a warning about arbitrary noise, not about practical AdamW [^1].
The second barrier is the core of the problem. In a constructed one-dimensional case, rare large outliers still leave SignSGD drifting in the right direction, but AdamW's squared-gradient average remembers them and shrinks its steps, so that memory in the denominator can hide large gradients [^1]. The authors say this is not a universal lower bound over all AdamW settings. It identifies a concrete long-memory regime where AdamW can look like it is making progress by the weighted measure while the gradient itself stays large [^1].
Why it matters
The answer would help explain something practitioners already see. Lion and Muon have shown consistent gains over AdamW in many practical settings, including large LLM systems such as Kimi-K2, GLM-5 and DeepSeek-V4 [^1]. If AdamW satisfies the same heavy-tailed guarantee, the authors write, the advantage of the sign-based optimizers must come from aspects classical convergence theory cannot capture. If AdamW fails, theory would help explain why Muon may beat it in heavy-tailed LLM training [^1].
The paper also shows why the usual proof route does not carry over. Under heavy-tailed noise the expected squared gradient may be infinite, so proofs that replace AdamW's squared-gradient average with a finite second-moment estimate have nothing to stand on [^1]. AdaGrad's accumulator only grows, and recent work could control it; AdamW must also handle the mismatch between two exponential averages [^1]. None of this shows that real AdamW training runs fail. The results are proofs about constructed cases, so the paper marks a gap in the theory, not a verdict on the optimizer.
Practical example
Say Jun runs pretraining experiments for a small language model at a university lab, using AdamW in the usual setup where the squared-gradient average has a much longer memory than the momentum [^1]. For one weight, most gradients are modest and point the same way. Every so often, though, a batch produces a huge value in the opposite direction.
AdamW squares that spike and keeps it in its long-memory average, while the momentum forgets it much sooner [^1]. The step for that weight is the momentum divided by the square root of that average, so for many steps it shrinks toward zero even though the true gradient there has not shrunk. In the paper's constructed one-dimensional example, this keeps the average gradient over the run at no less than half its starting value, while SignSGD, which only looks at the direction, keeps drifting the right way [^1].
What Jun should not conclude is that AdamW is broken. The construction covers one long-memory regime, not every setting [^1]. If Jun's comparison shows Muon training faster, the paper offers two readings: either theory will eventually show AdamW falling behind under heavy tails, or Muon's edge comes from something classical convergence theory does not capture [^1].
Related gear
We recommend this book because its chapter on optimization explains how adaptive methods such as AdaGrad, RMSProp and Adam work, the background this open problem assumes.