LLMs Can Write Code, but Cannot Read Your Mind

For a couple of years I taught C and Operating Systems, and the submissions that took longest to mark were never the ones that were obviously broken. They were the ones that compiled cleanly, ran, produced the right answer, and were still wrong. You could not see it by reading the code. You could only see it if you knew what the code was for.

Large language models (LLMs) such as ChatGPT, Claude, and Copilot are very good at producing exactly that kind of code. They complete a function in your editor in seconds, draft a first implementation, explain a stack trace you have been staring at for an hour. What they cannot do is know what the code is for. An LLM has read more implementations than any of us ever will, and it can still hand you the wrong one, because it does not know your threat model, your failure modes, your data flows, your latency budget, your regulatory boundaries, or the operational quirk everyone on your team has quietly worked around for two years.

When you write code yourself, most of that sits in your head, steering the small decisions as you go. When the code arrives off the shelf, it arrives without any of it.

The Same Function, Written Twice

# Version A - looks fine, absolutely wrong for security
import random, string

ALPHABET = string.ascii_letters + string.digits

def password(n: int = 16) -> str:
    return ''.join(random.choice(ALPHABET) for _ in range(n))

print(password())

# Version B - looks the same, actually correct
import secrets, string

ALPHABET = string.ascii_letters + string.digits

def password(n: int = 16) -> str:
    return ''.join(secrets.choice(ALPHABET) for _ in range(n))

print(password())

One import differs. Version A uses random, Version B uses secrets. Everything you can observe about the two functions from the outside is identical: same signature, same alphabet, same length, output that looks equally scrambled to a human reading it. Both are here if you want to run them: Version A and Version B.

What differs is whether someone watching the output can work out what comes next. random is built for simulation and modelling, where you want numbers that are well distributed and reproducible. Given enough output, its internal state can be reconstructed and the rest of the sequence predicted. Predictability is deliberate: a simulation you cannot replay is not much use. secrets draws instead on the operating system’s cryptographic randomness, which is designed specifically to resist that.

It is worth being honest that none of this is truly random. A computer is a deterministic machine, and we are no better: asked to produce a random sequence by hand, people fall into patterns without noticing. What a cryptographic generator promises is narrower and more useful than randomness. It promises unpredictability that holds up against someone actively trying to break it.

Ask an LLM to “write a function to generate random passwords” and Version A is a perfectly reasonable thing for it to return. It is the more common pattern in public code, and it matches the naïve reading of the word “random”. The model has not made an error in any sense it could detect. It has answered the question you asked, and the question you asked did not contain the only detail that mattered.

The Same Trap in C

// Version A - looks fine, absolutely wrong for security
#include <stdio.h>
#include <stdlib.h>

#define PASSWORD_LENGTH 16
#define CHARSET "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"

int main(void) {
    const char charset[] = CHARSET;
    const int charset_size = sizeof(charset) - 1; // exclude null terminator

    for (int i = 0; i < PASSWORD_LENGTH; i++) {
        int index = rand() % charset_size;
        putchar(charset[index]);
    }

    putchar('\n');

    return 0;
}

// Version B - the obvious fix, and still not secure
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define PASSWORD_LENGTH 16
#define CHARSET "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"

int main(void) {
    const char charset[] = CHARSET;
    const int charset_size = sizeof(charset) - 1; // exclude null terminator

    srand((unsigned) time(NULL));

    for (int i = 0; i < PASSWORD_LENGTH; i++) {
        int index = rand() % charset_size;
        putchar(charset[index]);
    }

    putchar('\n');

    return 0;
}

The difference is one line. Version A never calls srand, and the C standard says that a program calling rand() without seeding first behaves as though it had called srand(1). The same seed produces the same sequence, every time:

$ cc -o password-rand password-rand.c
$ ./password-rand
FZXC0wg0LvaJ6atJ

$ ./password-rand
FZXC0wg0LvaJ6atJ

$ ./password-rand
FZXC0wg0LvaJ6atJ

Version B seeds from the clock, so the output at least changes between runs. This is the more interesting of the two examples, because Version B is what a fix looks like. It is what you would write once you noticed the repetition, and it is what a model will often give you if you point the repetition out. It is also still broken. The seed is a timestamp in seconds, so an attacker who knows roughly when the program ran has only a small range to search, and rand() was never a cryptographic generator in the first place.

So the visible bug gets fixed and the dangerous one survives the repair. That is much harder to catch than the first example, where you only needed to know one fact about a standard library. Here the code has already been corrected once, and having been corrected once is usually enough to stop anyone looking again. Both C versions are here as well: Version A and Version B.

Where the Missing Context Lives

Most of what governs a decision like this was never written down anywhere. It lives in the head of whoever has been working on the system, as an accumulated sense of what this code is, who can reach it, and what happens if it is wrong. A model can hold every published implementation of a password generator and still have no way to know whether the one you asked for guards a hobby project or a login page.

It does not know whether you work at a bank or on something you will abandon next weekend. It has no idea whether a compliance regime applies here, or how long this code is expected to survive. It knows what is in the prompt, and nothing else is available to it.

What the Model Reaches For

Left without that context, the model falls back on frequency. It offers what appeared most often in its training data, which correlates with what is popular and only loosely with what is correct. Where two functions look alike but promise different things, as random and secrets do, it has little to distinguish them beyond how often each one showed up. And the result will usually pass your tests, because your tests check that the password is sixteen characters long, not that it is unpredictable.

The obvious response is to put more context in the prompt, and that genuinely helps, though it does not close the gap. Some of what matters cannot go in a prompt at all: you cannot paste an entire architecture, threat model, and decade of operational history into a context window, and you would not want to, because a good deal of it is precisely the thing you are protecting. Doing it means handing over your codebase, and with it whatever keys and known weaknesses are sitting in there.

Even setting that aside, you would leave something out. That is the nature of implicit knowledge: you do not have a list of it. And the model has no way of telling you what you failed to mention.

Where They Genuinely Help and Where They Do Not

Use the model where the stakes are low and the context is thin. Test skeletons, mundane refactors on code that already has coverage, and explaining an error when you can hand over the relevant code and logs alongside it. In all of these the surrounding context is either unimportant or small enough to actually supply.

Do not use the model for security-critical code, concurrency and lifecycle management or anything carrying a compliance obligation. In those areas wrong output looks exactly like right output, and the cost of failing to notice is high. The model can draft, but you remain accountable for what ships, which means reading it on the assumption that something in it is wrong.

The Mistakes We Already Published

None of this starts with LLMs. Developers have always copied from Q&A sites because it saves time, and upvotes have never been a measure of security. A 2021 paper presented at ACSAC went looking for how far that goes. They analysed nearly two million Stack Overflow posts tagged C, C++, and Android, identified around 12,500 insecure posts between them, and then checked whether those snippets had escaped into real software. They had: the insecure code turned up in the latest releases of 151 of the 2,000 popular C and C++ projects they examined.

The example they open with is the same shape as the two above. A string-trimming function passes a plain char to isspace, which is undefined behaviour the moment that char holds a negative value, something it can do on most platforms for any byte above 127. Someone in the same thread later posted a corrected version that casts to unsigned char, so both versions sit on the same page, and the one that spread is the broken one.

A single mistake gets posted once, spreads into a few hundred repositories because it looks right and works most of the time, and those repositories then become training data. The model does not read the correction further down the thread and weigh it against the original; it sees which of the two appears more often. Filtering known-insecure code out of a training set helps at the margins, but the underlying count is still a count of copies, and the broken version has more of them.

Where the Judgement Stays

An LLM multiplies how fast you can produce an implementation. It does not stand in for deciding which implementation you needed. It has no view on your compliance obligations, your users, or your adversaries, and it will hand you code that is entirely correct in isolation and wrong in the only place it is going to run.

Use them where the context is thin and the consequences are small. Where the context is complex, supply it, test against it, and then read the result yourself. That last part is the one that gets skipped.