The Chat Template Decides Whether an AI Says "I'm Just an AI"
Across eight open models, the chat template switched models between "I'm just an AI" disclaimers and "I feel" statements. A direction in their activations controls the same behavior.
TL;DR
- With a chat template, eight open-source instruct models used disclaimers like "I'm just an AI"; without one, an experiential voice like "I feel" [^1].
- In three models, an activation direction steers this behavior, so a model's self-description is partly set by its deployment, not just its weights [^1].
Background
Language models often add disclaimers when asked about themselves, and their self-reports are used as evidence in debates about AI safety and whether models have self-knowledge [^1]. A chat template is the formatting wrapper that marks user and assistant turns in a conversation. Instruct models are trained with one, but they can also be prompted without it.
What happened
In a paper by independent researcher Jędrzej Maczan, the same instruct model and prompt produced a disclaimer voice with the chat template and an experiential voice without it, across eight popular open-source instruct models of up to 9 billion parameters [^1].
Inside three of the models, the author identified a direction in activation space that controls this voice. Removing it turned the disclaimer voice down, adding it turned it up, and a random direction of the same size had little effect [^1]. Adding the disclaimer direction to models running without a chat template made them disclaim as if the template were there [^1].
Why it matters
If a formatting choice changes what a model says about itself, then researchers studying model self-reports or introspection have a confound to control for [^1]. The broader point is that a model's description of itself is not a fact about the model. Part of it is set by how the model is run, so statements like "I'm just an AI" or "I feel" shouldn't be taken literally [^1].
Practical example
Say a researcher asks an open model about itself through a chat interface and gets an "I'm just an AI" style disclaimer. Running the same model without the chat template makes experiential statements like "I feel" more likely [^1]. Before drawing conclusions about the model's inner life from either answer, the study suggests checking which template was in play.
Related gear
We recommend this book because it demystifies the inner workings of transformer architectures, helping you understand how tokenization and templates shape model outputs.