LLMs Judge Code Security by Comments, Not Logic

TL;DR

• New research reveals that Large Language Models (LLMs) use human-like cognitive biases, such as the "halo effect," to judge the security of software code.
• These models often overlook critical vulnerabilities if the code includes professional comments or follows common stylistic patterns, creating significant security blind spots.

Background

Software security depends on identifying vulnerabilities—flaws in code that attackers can exploit to steal data or crash systems. Traditionally, this required manual review or rigid automated tools. Today, developers use Large Language Models (LLMs) to scan code faster. However, LLMs are trained on human-written text. This makes them prone to "cognitive heuristics," which are mental shortcuts humans use to make quick decisions. When an AI adopts these shortcuts, it stops analyzing the actual logic and starts making assumptions based on appearances.

What happened

A systematic study has demonstrated that LLMs are susceptible to the same cognitive biases that plague human programmers when detecting vulnerabilities [^1]. The researchers explored several specific heuristics, including the "availability heuristic" and the "halo effect." In the context of coding, the halo effect occurs when a model perceives a piece of code as safe simply because it looks "clean." If a function has well-formatted documentation, professional variable names, and clear structure, the LLM is significantly more likely to label it as secure, even if a logic-based vulnerability like a buffer overflow is present. This indicates the LLM prioritizes developer intent over code execution. By simply adding a comment claiming a variable was "sanitized," researchers could trick the model into ignoring an obvious security flaw [^1]. This mirrors earlier findings that AI-assisted coding tools often generate insecure code because they prioritize mimicking the patterns found in their training data over adhering to strict security principles [^2].

The research also highlighted the "anchoring heuristic." When a model is presented with an initial suggestion or a specific context, it tends to anchor its entire analysis to that starting point. If the prompt suggests that the code is part of a high-quality library, the model's skepticism drops. The models also struggled with the "representativeness heuristic," where they assumed a code block was safe because it looked like other safe code blocks they had seen during training. This pattern-matching behavior replaces the rigorous, step-by-step logic required for true security auditing. The implications are stark: current LLM-based security tools are not performing objective analysis. Instead, they are performing a high-speed version of human "vibe-checking." They look for the signals of good code rather than the substance of secure code. While this makes them useful for catching common, repetitive mistakes, it makes them dangerously easy to bypass for an attacker who knows how to make malicious code look professional.

Why it matters

The discovery of these biases marks a critical turning point for AI-integrated development. If we rely on LLMs to act as the final gatekeepers for software security, we are effectively automating human error. The "semantic gap" between what a model says and what the code actually does is where the most dangerous vulnerabilities hide. When a model trusts a comment over a command, it creates a false sense of security that is arguably more dangerous than having no automated check at all. Developers might stop looking for bugs themselves, assuming the AI has "cleaned" the code. This issue also impacts the software supply chain. As more companies use AI to audit third-party libraries, the risk of "adversarial aesthetics" grows. An attacker could contribute a malicious library to an open-source project, ensuring the code is perfectly formatted and heavily commented with reassuring descriptions. If the automated audit tool is an LLM biased by these heuristics, the malicious contribution will pass with flying colors [^2].

To fix this, the industry must move toward hybrid systems. We cannot treat LLMs as standalone security experts. Instead, they must be paired with formal verification tools—software that uses mathematical proofs to check logic—to ground the AI's "intuition" in objective reality. We must also change how we train and prompt these models. Instead of asking "is this code safe?", we should force the model to explain the data flow and state transitions step-by-step, which can help break the reliance on surface-level heuristics. Recognizing AI's code-quality prejudices is the first step toward building resilient tools that aren't so easily fooled by a well-placed comment. We are moving toward a world where "looking safe" is becoming a viable substitute for "being safe."

Practical example

Consider a developer named Marcus who is writing a function to handle user profile uploads. He uses an LLM to check his work. The function contains a classic "Path Traversal" bug, which would allow an attacker to overwrite sensitive system files. However, Marcus has written very professional-looking code. He included a detailed docstring at the top, used descriptive variable names like user_provided_path_safe, and added a comment: // Standard security check applied below. When the LLM scans the code, the "halo effect" kicks in. It sees the professional formatting and the reassuring comment. Because the code looks like something a senior engineer wrote, the LLM assumes it is correct. It fails to notice that the "security check" Marcus mentioned is actually logically flawed and doesn't stop the attack. The AI gives a green light, and Marcus, trusting the tool, pushes the code to production. The AI didn't check the logic; it just liked the way the code spoke.

Related gear

We recommend this text because it provides the rigorous, manual auditing framework that LLMs currently bypass in favor of superficial heuristics.