Intelligence Reform & Probability: A New Yardstick | The Cipher Brief

OPINION – WASHINGTON – Calls for reform within the US Intelligence Community (IC) are growing louder, presenting an opportunity to refine how analysts communicate probability assessments. With budget and staff reductions impacting the Office of the Director of National Intelligence (DNI),and external voices advocating for change,a fresh look at analytic methodologies is warranted.

• Recent intelligence failures, such as the New Orleans terrorist attack, the collapse of the Assad tyranny in syria, and Hamas’s October 7 atrocity against Israel in 2023, have prompted scrutiny. Senator cotton highlighted these misses during DCIA ratcliffe’s confirmation hearing.
• Experts have proposed roadmaps to rebuild the IC, emphasizing the need for complete reform in how analysis is presented to decision-makers.
• The IC has a history of adapting to change, with significant reforms implemented since 9/11, including the USA PATRIOT Act (2001), the creation of the Office of the Director of National Intelligence (2004), and the issuance of ICD 203 Analytic Standards (2007).

This article focuses on a practical aspect of reform: improving the gauge used to weigh and communicate probability.Intelligence analysts have long been focused on self-reflection and analytic biases. This moment of change presents an opportunity to refine the measurement of, and language around, probability.

The challenge lies in the spectrum of probabilistic language used by analysts. The current standard, as outlined in ICD 203, can be prone to miscalculation, misunderstanding, and analyst apathy. A refreshed standard is needed to better serve US national security.

• Current Standards – from ICD 203 section 2, a, noted above:

Author created using AI prompt.

A new standard should be more usable,more frequently enough used,and more accurate.

• More Usable – A simpler scale with fewer categories of probability is more practical. The current gauge asks analysts to consider probability within 5-percentage point increments, which is often unrealistic when forecasting human behavior. A more general range would provide a more realistic assessment.
• More Often Used – Analysts frequently use words such as “likely” and “unlikely” because that’s more effective than placing specific percentages in their written work. The ICD 203 standards can be restrictive and cumbersome, especially in fast-paced situations where analysts need to share insights quickly.
• More Accurate – Analysis is both an art and a science. Requiring exactness, such as gauging probability to a five percentage points, can lead to less accurate results. The job of the intelligence analyst is to forecast, not predict.Broader probability ranges allow analysts to incorporate nuance and make more informed calls.

Give AI a Chance

AI can assist analysts in assessing probability. The IC is exploring how to adopt AI. Examining the probability of an issue is one good example. If an LLM has access to all the same reporting as an analyst, an easy and productive exercise would be to ask the AI platform the likelihood of X happening, and than analyze that result in comparison to the analysts’ original findings. for example, asking Chat GPT the likelihood of Russia and ukraine reaching a peace deal can provide fodder for analytic debate.

Opinions expressed are those of the author and do not represent the views or opinions of The Cipher Brief.