Standard GPU Telemetry Can Expose Hidden AI Training Runs
A classifier fed only nine built-in NVIDIA GPU counters told training from other work with 98.2% accuracy and caught most deliberate disguises, provided the readings can't be forged.
TL;DR
- Researchers built a classifier that spots ML training from nine built-in NVIDIA GPU counters with 98.2% accuracy, without seeing code, weights or data [^1].
- It caught most of 20 evasion strategy families, but detected only 43 to 87 percent of white-box attacks it had not seen before [^1].
Background
Compute governance proposals would register, audit or even prohibit AI training runs above a compute threshold [^1]. US Executive Order 14110, in effect from October 2023 until it was rescinded in January 2025, required developers to report runs above 10^26 operations [^1]. Such rules only work if someone can tell when a GPU is training a model [^1]. NVIDIA's NVML, the C-based API behind the nvidia-smi tool, reports a GPU's utilization, memory, temperature, power and clocks [^2].
What happened
Robi Rahman of the Machine Intelligence Research Institute and Sabiha Tajdari of the University of Virginia posted their study on arXiv on June 17 [^1]. Their classifier reads nine NVML counters once per second: GPU and memory utilization, memory used, power draw, temperature, two clock speeds and PCIe traffic in both directions. These counters exist on every NVIDIA GPU from Pascal onwards, and polling them does not slow the workload [^1]. The authors logged 1,404 runs of 162 workloads, about 117 GPU-hours, on nine GPU models from four generations, from an RTX 4050 laptop chip to the B200 [^1]. A random forest with 400 trees separated training from inference and non-ML work with 98.2% accuracy and a 3.06% false-positive rate. Trained on data from eight GPU models and tested on undisguised runs from the ninth, it reached 100% for eight of the nine models [^1].
The authors then played attacker and defender for five rounds. Each round added evasion strategies aimed at the classifier's weak spots, and the defender retrained on them [^1]. The 20 resulting families range from throttling utilization and mimicking cryptocurrency mining to "training-as-inference", where gradient updates are diluted between many batches of inference [^1]. When each family was held out of training, the classifier averaged 95.2% accuracy on it, or 92.2% counting white-box attacks. All multi-GPU strategies were caught at 99.6% or better, and only 5 of the 20 families fell under 90% [^1].
In the final round, the attacker got the classifier's internals. One tactic fit 4 seconds of training across the boundary of two 5-second windows. Another ran a copy of the classifier live and paused training whenever its estimated probability of training reached 0.30 [^1]. These tactics evaded the previous classifier up to 94% of the time. After retraining, the defender caught them whenever it had seen them, but only 43 to 87 percent of white-box strategies it had not seen [^1].
Why it matters
The authors argue that physics favors the defender. A backward pass takes about twice the computation of a forward pass, so training is three times as energy-intensive as inference for the same tokens; an evader can shift that energy around or train less, but cannot hide it without giving up the weight updates [^1]. Multi-GPU training, which the authors treat as the more important case, adds interconnect traffic patterns that are hard to disguise [^1]. The monitor also never sees content, which addresses a central concern in shared cloud settings, where inspecting proprietary information meets legal, operational and ethical objections [^1]. An earlier privacy-preserving approach, WAVE, reads profiler counters at 1200 to 5300 percent runtime overhead [^1].
The result has a hard precondition. It assumes the counters are honest. Against an operator who controls the whole software stack, the paper says, "an adversary with software privilege simply returns forged values" [^1]. A deployable monitor would need a tamper-resistant read path, a signed telemetry channel and securely booted monitoring software, which the authors frame as the next problem: "Can we build the matching HEM?", meaning hardware-enabled mechanism [^1]. They also list limits: single machines with up to 8 GPUs, evasion written in PyTorch rather than custom CUDA kernels, and only NVIDIA GPUs released since 2020 [^1].
Practical example
Say Tomas runs compliance at a cloud provider that rents out 8-GPU H200 nodes. A rule like the one in the rescinded executive order would require reporting large training runs [^1]. A tenant wants to train a model without that showing up, so the job hides its training steps between many batches of inference, hoping the GPUs look like they are serving a chatbot [^1].
Tomas's monitor polls nine NVML counters per GPU once a second and never touches the tenant's code or data [^1]. The disguise runs into physics: each training step still needs a backward pass, which costs about twice the computation of the forward pass and shows up in power draw [^1]. Spread across eight GPUs, the job also produces the interconnect traffic of multi-GPU training, and in the study every multi-GPU disguise of this kind was caught at 99.6% or better when the classifier had not seen it before [^1].
Tomas also picks the alarm level. The classifier outputs a probability of training, and the authors suggest a compute provider would set a higher threshold than a regulator would, to avoid flagging legitimate customers [^1]. The weak spot is the reverse case: if the operator of the hardware were the one hiding training, nothing in software would stop them from feeding the monitor forged readings [^1].
Related gear
We recommend this book because it explains how processors, GPUs included, spend computation, memory bandwidth and power, including the roofline model the classifier's features build on.