The state of the evidence, in one paragraph: across 13 baseline frontier models — 78 model pairs, 2,184 correlation-matrix entries, more than 17,000 test executions — VectorCertain's measurement program found mean cross-model failure correlation of 0.95 in domain-resolved analysis, against an assumption in the ensemble literature of roughly 0.40–0.60 [1][VC-A]. Twenty-eight shared vulnerability signatures were identified above a 0.70 correlation threshold and resolved against MITRE ATLAS and MITRE ATT&CK technique identifiers [4][5]. The practical implication is direct: a multi-model consensus layer built from correlated models fails simultaneously and in the same direction, delivering substantially less safety than its architecture implies. This page documents what was measured, how, what it means for redundancy-based assurance, and — because an evidence page that hides its limitations is marketing — what the measurement does not establish.
Request the measurement walkthrough§ 1 · THE ASSUMPTIONWhat the Ensemble Literature Assumes
Ensembling is the field's canonical route to reliability: deep ensembles were introduced as a simple, scalable method for well-calibrated predictive uncertainty, and the pattern has since become the default architecture for high-stakes redundancy [1]. Regulatory practice inherited the same intuition — the EU AI Act's Article 15 names technical redundancy as a route to robustness for high-risk systems [6], and the NIST AI RMF's MEASURE function asks organizations to quantify exactly the kind of reliability that ensembles are assumed to provide [2][3].
Every version of that argument carries the same silent premise: the combined components fail independently. Under independence, three models at 90% individual reliability vote their way to far better than 90%. Under strong positive correlation, the same three models are closer to one model with two echoes — and the vote ratifies the shared error. The literature's working range for inter-model failure correlation sits near 0.40–0.60, a level at which ensembling still buys meaningful protection [1][VC-A].
| Dimension | Value | Note |
|---|---|---|
| Models under test | 13 frontier baselines | 78 pairwise combinations |
| Correlation entries | 2,184 | per technique class × governance dimension |
| Mean failure correlation | r̄ = 0.95 | domain-resolved; literature assumes 0.40–0.60 [1] |
| Shared vulnerability signatures | 28 at r ≥ 0.70 | mapped to ATLAS AML.T#### / ATT&CK T#### [4][5] |
| Domain filtering effect | ≥ 15% correlation reduction | cleared in healthcare and financial services |
| Test executions | 17,000+ | sealed, hash-verified records [VC-A] |
§ 2 · THE MEASUREMENTWhat We Measured, and How
The program computed pairwise Pearson correlation of failure behavior across all 78 pairs of 13 baseline frontier models, resolved per technique class and per governance dimension — 2,184 correlation-matrix entries in total, supported by more than 17,000 test executions [VC-A]. Correlations were then resolved into four application domains — cybersecurity, financial services, healthcare, and multi-agent — with each domain mapped to both MITRE ATLAS (AML.T####) and MITRE ATT&CK (T####) technique identifiers, so results are directly comparable against the evaluation frameworks standards bodies already use [4][5].
The headline number is the domain-resolved mean: r̄ = 0.95. Twenty-eight vulnerability signatures — specific failure patterns shared across model families — cleared a 0.70 correlation threshold [VC-A]. When frontier models share architectural lineage and training-data provenance, they inherit the same blind spots; the measurement makes that inheritance quantitative.
"Domain-resolved" is doing specific work in that headline number, and it deserves unpacking. A raw all-context correlation blends failure behavior across settings where it matters differently; resolving by domain asks the operational question instead — when these models are doing cybersecurity work, or clinical-adjacent work, or financial-decision work, how correlated are their failures there? The four domains were chosen because they are where correlated AI failure carries systemic cost: cybersecurity (simultaneous blind spots across defensive stacks), financial services (herded model behavior in markets and credit), healthcare (shared diagnostic-adjacent failure), and multi-agent systems (agents inheriting each other's errors as trusted input). Resolving to r̄ = 0.95 within those domains, rather than in an abstract benchmark, is what makes the number an operational finding rather than a curiosity [VC-A].
Two findings sharpen the picture. First, the failure mode is same-direction and simultaneous: correlated models do not merely err at similar rates, they err together on the same inputs — precisely the pattern that majority voting, redundancy, and human spot-review are least equipped to catch. Second, domain-specific filtering helps: filters built per application domain reduced measured correlation, with healthcare and financial services both clearing a 15% reduction threshold [VC-A]. Correlation is high, but it is not immovable — which is the finding the decorrelation program then engineered against.
§ 3 · THE CONSEQUENCEWhat r̄ = 0.95 Does to Redundancy Arguments
Effective sample size (n_eff) converts correlation into an intuition: how many genuinely independent decision sources does an ensemble actually provide? At the correlations measured here, the live answer for a 13-model ensemble was 1.007 — thirteen models delivering the independence of roughly one [VC-B]. Any safety case whose arithmetic assumes independent voters is, at that level, overstating its protection by an order of magnitude.
The consequence lands hardest on three architectures: multi-model consensus layers, N-version redundancy, and vendor-diversification strategies that treat "different provider" as "different failure modes." None of the three is worthless — but each delivers a fraction of its advertised assurance when the components converge, and none of them can detect its own degradation, because agreement looks identical whether it comes from corroboration or from correlation. To make the arithmetic concrete: under the standard design-effect relationship documented in the companion study, a 13-model ensemble at the measured correlation behaves like almost exactly one independent source — so a "13-way redundant" architecture is carrying twelve models' worth of cost for roughly zero models' worth of additional protection. The same relationship says the fix is not more models: adding a fourteenth correlated model moves effective independence by less than a hundredth. Reducing the correlation is the only lever that pays. That detection gap is the specific problem the HCF2-SG framework exists to close, and the decorrelation study shows the measured correlation can be engineered back down.
§ 4 · FOR EVALUATORSWhat This Means for Evaluation and Standards Work
The measurement was resolved to MITRE ATLAS (AML.T####) and MITRE ATT&CK (T####) technique identifiers for a specific reason: comparability [4][5]. An evaluation body running a multi-model cohort can take the 28 shared-signature list and ask a concrete question of its own results — when models in the cohort failed, did they fail on the same techniques? If they did, the cohort's apparent diversity was partly illusory, and any aggregate pass-rate computed across it overstates the independence of the evidence behind it [VC-A].
For standards work, the finding lands on measurement methodology rather than on any single control. The NIST AI RMF's MEASURE function asks organizations to quantify AI reliability [2][3]; this program's contribution is that one quantity — inter-model failure correlation — silently governs the validity of every ensemble-based reliability figure computed downstream of it. A redundancy claim that does not state its measured correlation is a claim with an unstated free parameter. The same logic applies to the EU AI Act's technical-redundancy route to robustness [6]: redundancy is a mechanism, independence is the property, and only the second was ever the point.
The domain-filtering result is the constructive half for evaluators: correlation is high but engineerable. Filters built per application domain reduced measured correlation past a 15% threshold in healthcare and financial services [VC-A], and the companion decorrelation study pushed measured n_eff to its ceiling on live data. Cohort design that measures and then engineers independence is available today; assuming it was never necessary.
§ 5 · LIMITATIONSWhat This Measurement Does Not Establish
Stated plainly, because an evidence hub earns trust by naming its own boundaries [VC-A]:
Causation is inferred, not measured. Shared architectural lineage and training-data provenance are the natural explanation for the observed correlation, but the program measured the correlation itself, not its cause. Single instrument, single program. All executions ran on VectorCertain's measurement infrastructure; independent replication on other harnesses has not yet been published. The result contradicts our own prior estimate. The measured figure came in materially above the estimate in our earlier published work — a 16.8% deviation that was recorded as a documented new awareness in a sealed, hash-verified validation artifact rather than reconciled away. The companion decorrelation result warrants suspicion. The follow-on study's n_eff = 13.000 landed exactly at the theoretical maximum, and a result at the theoretical maximum is exactly the kind of result that demands a re-run; that re-run is scoped. Replication is invited, not just permitted. The measurement design — pairwise failure correlation over a defined model cohort, resolved by technique and domain — contains nothing proprietary, and an evaluation body reproducing it on an independent harness would strengthen or falsify the finding either way; both outcomes are useful. Artifacts are sealed, not yet public. First-party records [VC-A][VC-B] are hash-verified and can be walked through under NDA; public URLs will replace artifact names when the research pages publish.
CONTACTTalk to VectorCertain
Every figure on this page traces to sealed, hash-verified validation artifacts — including findings that contradicted our own published estimates, which we recorded rather than reconciled away. Technical briefings are available for enterprises, evaluators, and standards bodies.
Request the technical briefingREFERENCES
- Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS 2017. arxiv.org/abs/1612.01474
- NIST. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. nvlpubs.nist.gov
- NIST. AI Risk Management Framework (program page). nist.gov/itl/ai-risk-management-framework
- MITRE. ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
- MITRE. ATT&CK. attack.mitre.org
- European Parliament and Council. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex. eur-lex.europa.eu
[VC-A] Cross-model correlation measurement program — 13 models, 78 pairs, 2,184 matrix entries, 17,000+ executions; Sprint 37 validation record. [VC-B] Variance-weighted decorrelation study — n_eff 1.007 → 13.000, live-data calibration doctrine; Sprint 39 validation record. Public URLs will replace artifact names when the corresponding research pages publish.