The proverbial needle in a haystack

AS MEMBERS OF CONGRESS struggle to agree on which surveillance programs to re-authorize before the Patriot Act expires, they might consider the unusual advice of an intelligence analyst at the National Security Agency who warned about the danger of collecting too much data. Imagine, the analyst wrote in a leaked document, that you are standing in a shopping aisle trying to decide between jam, jelly or fruit spread, which size, sugar-free or not, generic or Smucker’s. It can be paralyzing.

“We in the agency are at risk of a similar, collective paralysis in the face of a dizzying array of choices every single day,” the analyst wrote in 2011. “’Analysis paralysis’ isn’t only a cute rhyme. It’s the term for what happens when you spend so much time analyzing a situation that you ultimately stymie any outcome …. It’s what happens in SIGINT [signals intelligence] when we have access to endless possibilities, but we struggle to prioritize, narrow, and exploit the best ones.”

The document is one of about a dozen in which NSA intelligence experts express concerns usually heard from the agency’s critics: that the U.S. government’s “collect it all” strategy can undermine the effort to fight terrorism. The documents, provided to The Intercept by NSA whistleblower Edward Snowden, appear to contradict years of statements from senior officials who have claimed that pervasive surveillance of global communications helps the government identify terrorists before they strike or quickly find them after an attack.

The jam vs. jelly document, titled “Too Many Choices,” started off in a colorful way but ended with a fairly stark warning: “The SIGINT mission is far too vital to unnecessarily expand the haystacks while we search for the needles. Prioritization is key.”

These doubts are infrequently heard from officials inside the NSA. These documents are a window into the private thinking of mid-level officials who are almost never permitted to discuss their concerns in public.

In addition to consulting Nobel-prize winning work, NSA analysts have turned to easier literature, such as Malcolm Gladwell’s best-selling Blink: The Power of Thinking Without Thinking. The author of a 2011 document referenced Blink and stated, “The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.” The author added, “Gladwell has captured one of the biggest challenges facing SID today. Our costs associated with this information overload are not only financial, such as the need to build data warehouses large enough to store the mountain of data that arrives at our doorstep each day, but also include the more intangible costs of too much data to review, process, translate and report.”