01. The Setup
There is a specific kind of wrong that is far more dangerous than ordinary wrong. It is the kind of wrong that presents itself as right, that produces an answer, a conclusion, a recommendation, with exactly the same composure and apparent certainty as when it is correct. This is not a new problem. It is, in fact, an ancient one. We have just managed to build systems that exhibit it at unprecedented scale.
The term for this is miscalibration: a mismatch between a system's expressed confidence and its actual accuracy. A well-calibrated system that is 70% confident is correct roughly 70% of the time. A miscalibrated system that is 70% confident might be correct 40% of the time. or 90%. The confidence score has become decorative.[01]
We became interested in this problem not through literature review, but through a failure. During early Vanguard development, our automated analysis engine was flagging vulnerabilities with high confidence scores. The confidence scores were wrong. Not wrong in the sense that the engine was uncertain but wrong in the sense that it was producing outputs we had no reliable way to trust or distrust without manually reviewing the thing the engine was supposed to be reviewing for us. We had built a confident oracle that needed a confident interpreter. This is not a system. This is a ceremony.
"The engine produced 847 findings rated HIGH confidence. Manual review of a 10% sample confirmed 61% were accurate. The engine did not know which 39% it had gotten wrong. Neither did we, until we checked."[04]
02. Confidence ≠ Accuracy
It is worth being precise about what we mean by confidence, because the word is doing a lot of work and not all of it is legitimate.
In a statistical or probabilistic context, confidence has a formal definition: it is a measure of how certain a model is about a particular output, derived from the distribution of its training signal or the geometry of its output space. It is, in theory, calibratable. You can compare a model's expressed confidence against empirical outcomes and measure the gap. This is a solved problem in narrow domains.[02]
In the context of modern large-scale reasoning systems, and increasingly, in AI-augmented analytical tools, like the ones we build, confidence is often something else entirely. It is a temperature parameter. A softmax output. A number that lives between 0 and 1 and presents itself as a probability without having been subjected to anything like proper calibration on the specific problem domain you are applying it to.[01]
"Confidence without calibration is not a probability. It is an aesthetic choice. The model has decided to sound sure. This is a performance, not a measurement."
The practical consequence: operators learn, quickly, that they cannot trust high-confidence outputs without verification. They verify everything, which means the automation provides comfort rather than leverage. Or, more dangerously, they trust high-confidence outputs without verification, because that is what the system implies they should do, and they are busy, and the cost of being wrong is not immediately visible.
Both failure modes are common. Both are predictable. Neither is the operator's fault.
03. Why This Matters for Software
Software security analysis has a particular exposure to this problem. The domain has three properties that make miscalibrated confidence especially dangerous:
The consequence of a false negative is high. A missed vulnerability that a confident system said did not exist is not just an error, it is a specific confidence in wrongness that the operator acted on. The absence of a finding is often read as the presence of safety.
The surface area is large. Modern codebases have millions of lines. The only practical reason to use automated analysis is because manual review at scale is not feasible. If the automation requires manual validation of its outputs to be trustworthy, you have not solved the scale problem, you have restated it with extra steps.
The signals look the same. A high-confidence true positive and a high-confidence false positive are indistinguishable in the output layer. There is no visual cue, no flag, no asterisk. The system is equally decisive about both. The only way to tell them apart is to already know the answer, which is precisely what you were hoping the system would tell you.
These three properties together create a failure mode we have taken to calling the oracle trap: the system is trusted as an oracle, it performs as an oracle in training or demonstration conditions, and it fails quietly and confidently in operational conditions where the distribution has shifted or the edge cases are exactly the ones that matter.[06]
04. The Calibration Problem
Calibration is not mysterious. The concept is straightforward: after a model has made predictions, you compare those predictions to outcomes, and you measure the gap between expressed confidence and empirical accuracy. If a model says it is 90% confident in a set of claims, roughly 90% of those claims should be true. If the number is 65%, the model is overconfident. If it is 97%, it is underconfident. Either way, the confidence score needs adjustment.[02]
The problem is that proper calibration requires a labeled ground truth dataset that accurately reflects the deployment distribution. In security analysis, this is hard to obtain for several reasons:
- Ground truth requires confirmed exploitation or confirmed absence of exploitability — both expensive to establish.
- The distribution of real vulnerabilities in production code shifts as codebases, languages, and attack techniques evolve.
- The adversary is adaptive. A calibrated model is a target. As soon as your detection is well-characterized, the evasion techniques follow.
What this means in practice: most security reasoning systems are implicitly calibrated against a training distribution that does not match their deployment distribution. The confidence scores are real relative to the training set and somewhat fictional relative to your codebase, your threat model, your operational context.[05]
This is not a flaw in implementation. It is a flaw in the framing. We have been solving the wrong problem: how to make systems more accurate, when the prior problem is how to make systems honest about how accurate they are.
"An accurate system and an honest system are not the same thing. We need both. We have been building only the first one and calling it sufficient."
05. What Vanguard Revealed
Vanguard is our experimental static analysis and security reasoning engine. It is designed to analyze codebases at depth, mapping control flow, data flow, dependency graphs, and behavioral patterns against adversarial threat models. The analysis is automated. The intent is to reduce the human triage burden.
What we found during development is that the confidence problem was embedded in our architecture before we had named it. We had built a system that was, in some respects, highly accurate, and in other respects, dramatically overconfident in ways we only discovered by deliberately attacking our own output.
The specific discovery: Vanguard's confidence scoring was calibrated well for the class of vulnerabilities it had been designed and tested against. For novel vulnerability patterns, things at the edge of its training distribution, it was producing high-confidence outputs that were wrong at a rate approximately three times higher than its average. The system did not know this. The confidence score did not encode any information about whether the finding was in-distribution or out-of-distribution. A classic vulnerability and a novel edge case looked equally confident.[04]
"We ran 400 deliberately crafted edge-case inputs. Average confidence: 0.81. Accuracy on those inputs: 0.44. The system was operating as if it knew what it was doing in territory it had never seen. This is the confidence problem in a very specific form: capability that does not match the scope of application."
This led to a design question we have not yet fully answered: how do you build a reasoning system that knows what it does not know?
The In-Distribution Signal
One direction we have explored: augmenting Vanguard's output with an explicit distributional distance signal. When a finding is based on patterns that are close to the training distribution, the system reports this. When the finding is based on patterns that are further from the distribution (e.g., more novel, more unusual, more edge-case) the system flags this explicitly, separate from and prior to any confidence score.
This does not solve the calibration problem. It localizes it. The operator now knows: this finding is high confidence and this type of finding has been well-validated. This other finding is also high confidence but it is in territory the system is less familiar with. The confidence scores are the same. The honest annotation is different.[05]
Early results are promising in a specific way: analysts given the distributional distance signal triage faster, not because they skip validation, but because they prioritize it correctly. High-confidence, low-distance findings get lighter review. High-confidence, high-distance findings get heavier review. The system is no longer asking the operator to treat all high-confidence findings equally.
06. Toward Honest Machines
We use the word "honest" deliberately, and we think it is the right word.
A calibrated system is not just an accurate system. It is a system that tells you what it knows and what it does not know, in a form that you can act on. Honesty in this technical sense is an engineering property. It can be designed for. It can be measured. It can be optimized. It is not being pursued with sufficient seriousness in the current generation of AI-augmented analytical tools.[03]
Why not? Partly because confidence calibration is harder to demonstrate than raw accuracy. A demo that shows an accurate system is compelling. A demo that shows a calibrated system requires the audience to care about the difference between "usually right" and "right in ways that are predictable and communicate uncertainty about edge cases." This is a subtler thing to sell.
Partly also because the demand signal is unclear. Operators who have not yet experienced a high-confidence failure have not yet felt the need for calibration. The need tends to become vivid after an event.
We think the architecture of honest machines looks something like this:
Separate confidence from familiarity.
Confidence answers: how likely is this to be correct? Familiarity answers: how much experience do I have with this class of input? Both scores should be reported. Neither should be collapsed into the other.
Make distributional distance visible.
Any finding or output that is based on patterns at the edge of the training distribution should say so. Not as a warning label, but as a first-class output attribute.
Build calibration into the deployment pipeline, not just the training pipeline.
Post-deployment, systems should accumulate feedback about their predictions and update confidence estimates based on observed accuracy in the actual deployment environment. This is standard in some ML applications. It is not standard in security tooling.
Design for explicit human review at the confidence boundary.
The goal of automation is not to eliminate human judgment, it is to direct human judgment to where it is most valuable. A well-designed system makes explicit where it needs human input and where it has high confidence in its own output.
Treat overconfidence as a bug, not a feature.
A system that expresses high confidence in a wrong answer is not impressive, it is broken in a specific and dangerous way. The engineering culture around AI-augmented tools needs to treat this with the same severity it treats other forms of incorrect output.
"The goal is not a system that is always right. That system does not exist. The goal is a system that is wrong in ways you can see coming."
07. Open Questions
We do not have this solved. To be direct about where the work stands:
"At what confidence threshold does automated reasoning become safe to act on without human validation, and is that threshold universal or domain-specific?"
// Status: We believe domain-specific. Vanguard research ongoing.
"Can distributional distance be computed efficiently enough to be a real-time output attribute, or does it require batch post-processing?"
// Status: Early prototypes suggest real-time is feasible for static analysis. Unclear for dynamic contexts.
"How do you train operators to read calibrated uncertainty outputs without reverting to treating confidence scores as binary?"
// Status: This is a UX and training problem, not an ML problem. We are not experts in it. We are thinking about it anyway.
"Is overconfidence an inherent property of large-scale reasoning systems trained on human-generated data, or a correctable architectural flaw?"
// Status: Open. The literature is not settled. We have opinions. We are building experiments to stress-test them.
"When should a system refuse to produce a confidence score rather than produce a miscalibrated one?"
// Status: We think more often than current systems do. Exact criteria: unclear.
"How do you communicate calibrated uncertainty to an operator who is under time pressure and needs an answer?"
// Status: Unresolved. This is possibly the hardest problem on this list.
08. Notes & References
This note draws on work across several fields. References below are not exhaustive, they represent the specific sources that shaped our thinking on this problem.
[01] Guo et al., "On Calibration of Modern Neural Networks," ICML 2017. The paper that brought calibration into mainstream ML discourse. Required reading.
[02] Dawid, A.P., "The Well-Calibrated Bayesian," Journal of the American Statistical Association, 1982. Older, foundational, still precise.
[03] Kadane & Lichtenstein, "A Subjectivist View of Calibration," 1982. Useful for the philosophical framing of confidence as a claim about the world.
[04] Vanguard Experiment Log, Cycles 14–18. Internal. Available to lab collaborators on request.
[05] Ovadia et al., "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift," NeurIPS 2019. Directly relevant to the distributional distance problem.
[06] Perrow, "Normal Accidents," 1984. Not about ML. About why systems fail confidently in ways their designers did not anticipate. More relevant than it should be.
[07] Helios Prime, "The Instrument Is the Argument," Lab Notes, May 2026. Our own framing. Every model here was stress-tested by building something.
* A note on scope: this document describes our thinking as of the time of this writing. The field moves. Our opinions may move with it. We will update this note when they do.