Skip to content
← Unredacted

Heuer's Ghost: Cognitive Bias in the Age of the Machine Analyst

7 min read

If you've visited the front page of this site, you'll have seen a line from Richards J. Heuer Jr.: "Intelligence seeks to illuminate the unknown." I've quoted it in talks, in reports and, by now, in more than one article. Heuer spent the better part of his career at the CIA, and his Psychology of Intelligence Analysis (published in 1999, though most of it was written in the decades before) remains the closest thing our profession has to a founding text.

What makes the book unusual is where it locates failure. Heuer had little to say about satellites, sources or budgets. His argument was that the most dangerous weaknesses in intelligence sit inside the analyst's own head: in how we perceive, remember and reason, and in the mental shortcuts we take without noticing. Better collection doesn't fix a flawed mind. It only gives it more to be wrong about.

Twenty-odd years later, our industry is busy wiring large language models into every stage of the intelligence cycle. The sales pitch, explicit or not, is that the machine is the cure for the human: tireless, consistent and free of the biases Heuer catalogued. I build AI-native intelligence tooling for a living, so I'm hardly a sceptic of the technology. But I've come to believe that pitch is wrong in a specific, important way. The machine doesn't remove Heuer's biases. It relocates them, and in their new home they're harder to see.

That's the ghost I want to write about.

Do You Really Need More Information?

Chapter 5 of Heuer's book carries the most unfashionable title in our field: "Do You Really Need More Information?"

Heuer leans on an experiment by the psychologist Paul Slovic, in which experienced horse-race handicappers were given progressively more variables (five, then ten, twenty and forty) to predict the outcomes of races. Their accuracy barely moved as the information grew. Their confidence rose steadily. Past a modest threshold, more data didn't make them better forecasters. It made them more certain of the forecasts they would have made anyway.

Now consider what we've built. A modern analyst can pull a summary of every public report on a threat actor in seconds. The machine reads the forty variables so you don't have to, then hands you back a page of fluent, well-structured prose. If Slovic's handicappers grew overconfident from reading the data themselves, imagine what happens when the data arrives pre-digested in the confident voice of a well-written brief.

Fluency is the trap. Psychologists have long observed that we judge information to be truer when it's easier to process. A clean paragraph feels more reliable than a messy spreadsheet holding the same facts. Language models are, above all else, fluency machines. They produce the feeling of a finished assessment whether or not the reasoning underneath would survive scrutiny.

Perception: Seeing What We Expect to See

Heuer's second chapter asks why we fail to see what is there to be seen. His answer is the mindset: the set of expectations through which every new piece of information is filtered. Mindsets are unavoidable and often useful. They're how an experienced analyst recognises a pattern in minutes that a junior would take a week to find. They're also why that same analyst will rationalise away the one indicator that doesn't fit.

A language model has a mindset too. It's called training data.

Think about what the public record of cyber threat intelligence actually contains. It's written overwhelmingly by vendors, and vendors publish what their telemetry sees and what their customers will pay to read about. That record is heavily weighted towards North America, Western Europe and a handful of well-studied adversaries. Ask a model about the threats facing a mid-sized South African bank and you'll get a confident answer. Much of it will describe the threat landscape of somewhere else, simply because that's where the writing is.

This isn't a flaw a vendor can patch. It's reporting bias, the same thing Heuer warned about when analysts over-relied on whatever reporting happened to be available. The difference is that a human analyst can usually tell you where their blind spots are. A model can't. It has no sense of what it hasn't read.

Anchors Away

Among the judgement heuristics Heuer describes, anchoring is the one I see most often in practice. Whatever estimate we form first becomes the anchor, and subsequent evidence nudges us away from it far less than it should.

When the machine writes the first draft, the machine sets the anchor.

I've watched this happen to good analysts, and I've felt it happen to me. You ask for an initial assessment, it comes back plausible, and your job quietly changes from thinking to editing. You correct a date, soften an adjective, add a source. What you rarely do is throw the whole thing away and reason from the evidence up. Human factors researchers have a name for this: automation bias, the tendency to defer to an automated system's output and to stop looking for information that might contradict it.

Language models add a twist of their own. They tend towards agreement with the person asking. Frame your question as "Is this activity linked to Iran?" and you've done more than ask a question; you've suggested an answer. A model trained to be helpful will often oblige. Heuer spent a career warning analysts against confirmation bias. We've now built an assistant that can, if we're careless, produce confirmation on request.

Fluency Is Not Confidence

Sherman Kent, often called the father of intelligence analysis, wrote his essay "Words of Estimative Probability" in 1964, after discovering that colleagues who had signed off the same estimate understood the phrase "serious possibility" to mean anything from roughly a one-in-five to a four-in-five chance. The words sounded precise. The meaning wasn't shared.

Our profession has spent sixty years fixing that problem. Here in the UK, the Professional Head of Intelligence Assessment's probability yardstick ties phrases like "realistic possibility", "likely" and "almost certain" to defined probability ranges, so that a reader knows exactly how confident an assessor is prepared to be.

A language model will use every one of those phrases fluently. It will tell you something is "highly likely" with no underlying probability behind the words at all. The vocabulary of calibrated judgement, detached from any calibration, is worse than no vocabulary, because it borrows credibility the assessment hasn't earned.

Exorcising the Ghost

None of this is an argument against using machines in intelligence work. I'd be a strange founder if it were. It's an argument for using them the way Heuer would have wanted us to use any tool: deliberately, with a clear view of where our thinking is weakest. A few principles I've come to hold:

Make the machine argue, not answer. Heuer's best-known contribution is the Analysis of Competing Hypotheses (ACH), a method that forces the analyst to weigh evidence against every plausible explanation rather than searching for support for a favourite. This is where models are genuinely powerful. They're tireless at generating alternative hypotheses and at hunting for the evidence that would disprove your leading one. Use the machine as a devil's advocate, not an oracle.

No claim without provenance. Every statement in an assessment should trace back to a source a human can open and read. If it can't, it's an opinion wearing the costume of a finding. This rule alone defuses most of the fluency problem.

Separate the drudgery from the judgement. Reading, translating, deduplicating, extracting indicators and mapping behaviour to ATT&CK are tasks machines do faster and often better than we do. Deciding what it means for this organisation, and how confident we should be, is judgement. Keep it with a person who will be held accountable for it.

Ask open questions. "What explains this activity?" invites analysis. "Is this APT29?" invites agreement. The discipline Heuer demanded of analysts in their own reasoning now extends to how we prompt.

Keep score. Calibration is learnt by checking forecasts against outcomes. Record your estimative judgements, human and machine alike, and revisit them. Over time you'll learn which of your confident phrases you've actually earned.

The Ghost Was Always Us

Heuer wrote that thinking analytically is a skill like carpentry or driving a car: it can be taught, it can be learnt, and it improves with practice. I find that reassuring, because it means the answer to the machine age isn't to out-compute the machine. It's to keep practising the craft that the machine can't do for us.

Heuer never saw a language model. I suspect he'd have recognised the problem immediately, though, because it isn't new. The biases he spent his career documenting haven't been engineered away. They've simply found a faster, more articulate place to live.

The ghost was never in the machine. It was always in us. It just has company now.

Mayan Stegmann is a CREST Registered Threat Intelligence Analyst and the co-founder & CEO of Deltabridge. More about Mayan

More writing