November 13, 2013

The NSA’s Witch Hunt

Should the government spy on its citizens? The numbers say no.

From the moment Edward Snowden exposed the National Security Agency’s (NSA) secret data collection program in May, the Obama administration has faced a stream of embarrassing headlines. The humiliation culminated in German Chancellor Angela Merkel's outrage over the tapping of her cellphone—a reaction echoed by American allies in France, Spain, Mexico, and Brazil. Even staunch defenders of the NSA’s activities, such as Senator Dianne Feinstein, have joined the chorus of criticism.

Public policy should be based on more than headlines. The operative question is this: Should governments spy on their own citizens to find and neutralize terrorists? In the case of the NSA, unless its tests are keenly accurate and there are more U.S. terrorists than people living in South Dakota, its spying activities are simply not justified.

  cuba  
  Photo credit: Elvert Barnes

The U.S. government has spied on American citizens before. In 1918, 250,000 American volunteers joined the American Protective League (APL) to root out German spies residing in the United States during World War I. As David Henderson and I said in our book, Making Great Decisions in Business and Life:

These ordinary APL volunteers were very effective at getting information about American citizens and reporting suspicious behavior to the Justice Department. They cleverly found that they could “gain access to any house, on the grounds of checking their gas and electric service.” What was the result of their efforts? They reported well over a million “subversives,” which resulted in some deportations for immigration violations and some number of men caught in the midst of marital affairs, but not a single conviction for espionage. Most of the people held in prison for weeks without any charges filed had simply been critics of the war or had made the mistake of eating sauerkraut or listening to Beethoven.

There probably were German spies in America, but these spies were the proverbial needles in the haystack. Unless the APL’s tests were exquisitely accurate, and very few are, the APL likely would have snagged many more innocent than guilty people.

The American Protective League program was essentially an organized witch-hunt. One of my distant ancestors from seventeenth century Massachusetts, Ann Putnam, Jr., was one of the primary accusers in a real witch-hunt: the infamous Salem Witch Trials of 1692-1693. With those trials, more than 200 people were accused of practicing witchcraft and 20 were unjustly executed. Ann Putnam, Jr. later publicly apologized for her role:

I desire to be humbled before God for that sad and humbling providence that befell my father's family in the year about ninety-two; that I, then being in my childhood, should, by such a providence of God, be made an instrument for the accusing of several people for grievous crimes, whereby their lives was taken away from them, whom, now I have just grounds and good reason to believe they were innocent persons; and that it was a great delusion of Satan that deceived me in that sad time, whereby I justly fear I have been instrumental, with others, though ignorantly and unwittingly, to bring upon myself and this land the guilt of innocent blood; though, what was said or done by me against any person, I can truly and uprightly say, before God and man, I did it not out of any anger, malice, or ill will to any person, for I had no such thing against one of them; but what I did was ignorantly, being deluded by Satan.

History shows that if you look for witches, you will find “witches,” just as the APL looked for and found German “spies.” The Salem Witch Trials and the American Protective League provide lamentable and precautionary tales, but we should rely on something more analytical to answer the question at hand. Luckily, the field of medical diagnostics has largely solved this problem.

Many thousands of people walking the streets today harbor deadly diseases. In fact, there are far more cases of cancer and other serious diseases than there are terrorists. Luckily, some diagnostic tests could detect these cancerous cells early enough, enabling treatment. Why, might you ask, are these people left untested? Shouldn’t everyone be screened for every type of cancer? It’s a complicated problem.

We know that finding those individuals harboring cancer is like finding a needle in a haystack, and our ability to do that depends critically on the proportion of needles to hay. Is there one needle for every 100 pieces of hay or one for every 100 million pieces of hay? It will take a superbly accurate test to justify searching for needles with the 1:100 million ratio.

Every diagnostic test has two key measures of accuracy: the accuracy of detecting a disease that is present in a person (sensitivity) and the accuracy of not detecting a disease that is absent in a person (specificity). Each diagnostic test has a positive predictive value (PPV). Positive predictive value is the proportion of persons who test positive who really have the disease. A PPV value of 10 percent would mean that one out of ten individuals who test positive really have the condition and that nine out of ten don’t.

For a diagnostic test to be considered a good screening tool for the general population, experts say it should have a positive predictive value of 5 to 25 percent. For reference, the prostate cancer PSA test has a PPV of 20 to 50 percent and mammography for women 50 and older has a PPV of 14 percent. With the APL program in World War I and the Salem Witch Trials, the PPV was exactly zero.

The PPV target should be set at the point that balances the cost of the false positives (i.e., dealing with the cost and risk of further evaluating healthy people) and the benefit of the true positives (i.e., identifying real cases of disease early enough). For the NSA case, my calculations suggest a target PPV of one percent.

There are 317 million Americans. Assume that the NSA’s methods have a sensitivity of 80 percent (the ability to detect terrorists, such as the Boston Marathon bombers, when they are loose in society) and a specificity of 80 percent (the ability to tell that an innocent person is not a terrorist). If we use the formula for PPV (see Wikipedia for the exact formula) and target a PPV of one percent, then the NSA would only be justified in spying on us if more than 800,000 Americans—roughly the population of South Dakota—were terrorists.

Clearly, the real number is far below 800,000. I estimate the number to be closer to 100. If we enter 100 in the PPV formula, we generate a PPV of 0.00013 percent, which is far below one percent. With these numbers, the NSA is absolutely not justified in spying on Americans.

Why do I think there are only 100 terrorists in the country? Consider the Transportation Security Administration. In the TSA’s own tests, TSA employees caught 40 percent of the bombs present on passengers and in luggage (for a 40 percent sensitivity). Astute readers will notice that the lousy success rate for the TSA’s screening combined with the lack of airplane hijackings over the past decade can mean only one thing: There just aren’t that many motivated terrorists in our society.

If we consider that there have been two attempted hijackings in the post-9/11—and present TSA—era, then we can assume that two potential hijackers were caught beforehand. In other words, four tried and half were caught, leaving two who successfully entered their planes: the shoe bomber and the underwear bomber. Interestingly, there have been exactly zero successful attempts, primarily due to passenger activism. The TSA is not the only bulwark against in-air terrorist activities; we must include flight crews and motivated passengers. It appears, therefore, that the TSA’s actions and all those security lines we have endured over the years have likely prevented zero or close to zero hijackings.

I can think of a dozen or so straightforward and effective terrorist activities—things that one person or a small group could successfully accomplish—and I’m just a rank amateur. It’s scary to imagine what an expert could come up with. The fact that none of my propitious activities has actually been performed—thank goodness—suggests that there are not that many terrorists among us.

Perhaps the NSA’s tests are more accurate than I have assumed. Imagine, for the sake of argument, that the NSA’s tests have astronomically high accuracy, with sensitivity and specificity of 98 percent. Then the NSA would be justified in spying if more than 65,300—think Portland, Maine or Lynchburg, Virginia—Americans were terrorists.

Don’t forget that the TSA is only at 40 percent. The TSA also has an easier job screening passengers and their bags. The NSA has a more difficult job with snatching electronic snippets out of the ether. If the NSA’s tests are merely as accurate as the TSA’s, the NSA would only be justified in spying if more than 4.7 million Americans—the population of South Carolina—were terrorists.

Consider also that the bad guys, such as spammers, scam artists, hackers, and members of the Chinese government might eventually tap into the data the NSA has meticulously collected and organized. Also, the NSA itself might abuse this sensitive data: we already know that at least a dozen NSA employees have abused secret surveillance programs in the past decade, most often to spy on their significant others. Moreover, the data could be used for political advantage: as the latest IRS harassment of the Tea Party reminds us, the IRS has been repeatedly used for political persecution. Finally, freedom of the press could be at stake: the Justice Department admitted that it secretly seized phone records from The Associated Press. And then there are the manifold problems with international relations and eroding trust in government. Overall, it is hard to see how the NSA’s program of organized spying will not lead to untoward results.

The NSA’s screening of the general population to find terrorists is analogous to what the medical profession does with diagnostic tests to screen for cancer. Using the medical profession’s insights and equations, we have determined that unless the NSA’s tests are keenly accurate and there are more terrorists than South Dakotans, the NSA is simply repeating the mistakes of the Salem Witch Trials and the American Protective League in WWI. If the NSA persists, the likely results are truckloads of wasted money, trainloads of innocent Americans incorrectly flagged as terrorists, and no perceptible reduction to our overall risk from violence. The numbers simply do not support the NSA’s secret spying on Americans.


Charles L. Hooper is president of Objective Insights, a company that consults for pharmaceutical and biotech companies, and a visiting fellow at the Hoover Institution.


Letters to the editor may be sent to definingideas@stanford.edu. Editors reserve the right to reject or publish (and edit) letters.