Features:
Prevention Programs and Scientific Nonsense

By D.M. Gorman

A report from the field

In november 2001, a bizarre incident occurred at a conference sponsored by the Center for Substance Abuse Prevention (csap) in Washington, D.C. As reported by Stanley Kurtz on National Review Online (December 5 and 11, 2001) and Sally Satel on Tech Central Station (December 7, 2001), Christina Hoff Sommers of the American Enterprise Institute was invited to address an assembled audience of csap staff, grantees, and consultants concerning the agency’s intentions to fund “Boy Talk,” a prevention program for young men designed to influence such behaviors as drug use and violence. Sommers is a critic of this type of gender-specific exercise in health education, and suggested to the csap audience that the effectiveness of such programs and the commitment of dollars to them should be informed by scientific evidence regarding their effectiveness. She observed that “Girl Power!” — the program on which “Boy Talk” was based — remained unsupported by empirical evidence despite having been the recipient of federal funds for the past six years. At this point, a csap official informed Sommers that she should end her presentation, apparently because the “Girl Power!” program was off limits. Whether Sommers had been informed of this taboo before she began her talk is not clear, but given the association between the two programs it seems reasonable that the experiences concerning one be used to inform implementation of the other. Anyway, Sommers appears to have soldiered on with her presentation, for within minutes she was apparently instructed to “shut the f--- up, bitch” by a member of the audience, causing much merriment among the assembled group of professionals.

Over the summer, I had a similar, although not as overtly censorial or hostile, experience while presenting at the Tenth Annual Meeting of the Society for Prevention Research in Seattle, Washington. While nobody dropped the F-bomb on me or called me “bitch,” the response to my attempt to examine the scientific base of some widely advocated prevention programs was an ad hominem attack coupled with defensive arguments justifying the violation of basic tenets of evaluation practice in prevention research.

Science and the learned society

My presentation at the Society for Prevention Research conference focused on a list of drug and violence prevention programs deemed by an expert panel convened by the U.S. Department of Education’s Safe, Disciplined, and Drug-Free Schools program to be “exemplary.” According to the criteria used by this panel to define exemplary, there must be at least one evaluation that has demonstrated an effect on some violent or drug-related behavior, and this evidence must come from a “methodologically sound evaluation.” In my presentation, I critiqued the Education Department’s criteria through an examination of three of the programs that were conferred exemplary status — Project alert, the Second Step curriculum, and the Adolescent Training and Learning to Avoid Steroids (atlas) program. I pointed out what I considered to be serious methodological flaws in the evaluations of these programs, especially with regard to the recruitment and retention of participants and/or the use of questionable analytical techniques such as one-tail significance tests and post hoc sub-group analysis.

In addition, I noted that while there were indeed isolated effects on behavioral outcomes to be found in the evaluations of these programs, the preponderance of evidence showed that they had little or no effect on drug use or violence. For example, the evaluation of the violence prevention program Second Step produced only one statistically significant result out of the 20 outcomes that were assessed at final follow-up, while two evaluations of the atlas steroid prevention program failed to demonstrate any effects on steroid use.

The annual conference of the Society for Prevention Research seemed an appropriate venue to discuss such issues. After all, the society describes itself on its website as “an international scientific organization focused upon the advancement of science-based prevention programs and policies through empirical research.” Its journal is entitled Prevention Science. The conference theme was “Effectiveness and Dissemination in Prevention Research.” Presenting a paper that addressed the scientific base of a set of research findings that purported to demonstrate program effectiveness should have been ok, right? Wrong.

My 15-minute presentation was the first of four diverse papers addressing “Issues with the Safe and Drug-Free Schools Program.” During the course of the other presentations, the developer of the atlas program, Linn Goldberg, entered the room, having been summoned by one of his colleagues. He was the first to speak during the question-and-answer session at the end of the formal presentations. He began by saying that he had not been present for my presentation, but had been informed as to its content and had read my abstract in the conference program. He then asked whether I had informed the audience that the atlas program evaluations had demonstrated effects on behaviors such as the use of sports supplements and alcohol and drugs. I said that indeed I had, but that my main problem with the program was that it did not have any effects on its target behavior — in short, it was a steroid prevention program that did not prevent steroid use. Goldberg responded that steroids were a kind of hook that was used to promote the program and get schools interested in it. To this I countered that a school superintendent seeing a prevention program with the word “steroids” in the title might reasonably expect it to stop adolescents from using steroids.

From here, things began to deteriorate. Referring to my abstract in the conference program — which described the Department of Education exemplary criteria as “inadequate” — Goldberg asserted that it was, in fact, I who was inadequate to the task of assessing prevention programs. Other authoritative groups, he noted, had conferred similar honors on his atlas program. Who was I, he demanded — “from Bryan, Texas” — to question the wisdom of those experts who considered his program exemplary? I must admit that I found this rather amusing, as I had noticed (with some concern) earlier in the day that the program simply listed my affiliation as “School of Rural Public Health, Bryan, Texas” — with no mention at all of the Texas a&m University system.

At this point the session chair interceded, and asked if anyone else had a question or comment. In the course of the debate that followed, my use of science as the standard against which to judge evaluations of drug and violence programs was questioned. Specifically, concern was expressed that the methodological standards I applied to these programs were inappropriate to prevention research as this was a “new science.” For example, it was suggested that use of one-tail significance tests was perfectly ok if the evaluator had no expectation that a program could have negative effects. Quite how one could know that an intervention would have no adverse effects without first having evaluated it was not made clear (and, indeed, school-based prevention programs have been shown to be iatrogenic in the past).

My parting shot to the audience involved a simple reminder of the place of critical discussion of research findings in science. I noted that in 10 years of attending prevention conferences that purported to be interested in research, evaluation, and science I had observed almost total resistance to such critical analysis. Indeed, the only program that I had ever heard discussed critically was the drug education program dare; criticism of any other intervention was met with the type of defensive response that I had just experienced.

Why did my examination of data from evaluations of programs deemed by the Department of Education to be exemplary meet not with critical discussion of these data but with a hostility and defensiveness? Why did Christina Hoff Sommers’s suggestion that we use scientific evidence to inform our decisions as to how to allocate a federal agency’s funds meet with derision and censorship? Why is it that at a time when federal agencies such as csap, the National Institute on Drug Abuse, the Department of Education, and the Centers for Disease Control are advocating the use of so-called “science-based” prevention, the last thing anybody wants to do is engage in the type of critical discussion of evidence that is so fundamental to the practice of science? To answer these questions, one must understand what has become of science in the field of health promotion in recent years.

Science, anti-science, pseudoscience

In 1996, the Johns Hopkins University Press published a book entitled The Flight from Science and Reason that brought together an impressive collection of papers from a New York Academy of Sciences-sponsored conference convened to discuss the effects that various forms of “anti-reason” have had on the academy over the past four decades. In one of the papers included in the volume, the philosopher of science Mario Bunge drew an interesting distinction between two types of activity that currently pervade academic circles — anti-science and pseudoscience. The former, he observes, developed in the United States out of the counterculture of the 1960s, and is premised on a total denial of rational thought and empirical research — considering science to be merely an ideology that serves as a tool of male and/or racial and/or class domination. Variants of academic anti-science include existentialism and radical feminism.

In contrast, academic pseudoscience looks, at least superficially, like real science in that it formulates theories and hypotheses, measures and quantifies phenomena, and conducts statistical analyses. However, while it may display some of the “accoutrements of science,” Bunge observes, it lacks its underlying substance. One type of pseudoscience is comprised of theories that apply these principles in an inappropriate manner — for example, developing meaningless and unquantifiable models of human behavior. Another type of pseudoscience is comprised of empirical studies that purport to be scientific but whose practitioners violate the most basic premises of scientific research. Specifically, there is no genuine attempt in this form of pseudoscience to subject hypotheses to empirical tests that can falsify them.

What Christina Hoff Sommers encountered at csap was, I contend, the anti-science movement in health promotion while, at the Society for Prevention Research, I encountered its pseudoscience counterpart.

Anti-science and health promotion

The anti-science movement in health promotion has arisen as part of the humanistic perspective within the discipline that positions itself in direct opposition to a science-based approach that it terms “positivism.” According to this view, the application of the scientific method — that is, an approach to the world founded upon experimentation and hypothesis testing — to social phenomena is both epistemologically and ethically wrong. For example, writing in the journal Health Education Research in 1998, David Buchanan argued that the science-based approach views events from a cause-and-effect perspective which treats people as objects and thus contributes to alienation, powerlessness, and social disintegration. Therefore, the approach, he notes, is not only of little use in understanding human behavior, but is actually a central part of the ideology that serves to justify exploitation and suffering. Ethically, then, there is no place for a science-based approach to health promotion.

From an epistemological standpoint, the problem with the science-based approach to health promotion, according to its critics, is that it ignores the fundamental ontological distinction between natural processes and social processes. The former can be and have been successfully understood by the application of scientific methods and techniques. The latter cannot be and have not been elucidated through the application of science. Advocates of the humanistic approach such as Buchanan contend that the reason for this is that social phenomena derive from the aims, beliefs, expectations, choices, and intentions of individuals. These cannot be comprehended in terms of cause and effect. The best we can hope for is to describe social processes and actions, and in doing so we might better understand and make sense of aims, beliefs, expectations, choices, and intentions. However, while one can go some way toward “making sense” of social processes and actions one can never know what “causes” them to occur. Thus, in health promotion one cannot introduce a program or policy intended to produce a specific outcome, “control” for other factors that might influence that outcome, and assess the effects of one’s intervention. That is, one cannot undertake scientific research.

Not surprisingly, these arguments for rejecting the scientific approach to studying social processes have not gone unchallenged. Bunge, for example, in his recent book Social Science Under Debate (University of Toronto Press, 1998), has demonstrated that it is very difficult to draw a line between the subject matter of the social and natural sciences. Many disciplines — linguistics and psychology, for example — draw heavily from both the social and natural sciences. Likewise, the content area of public health and health promotion — namely disease processes and behavioral problems — simply cannot be understood as either solely biological processes or solely social processes. Bunge also observes that while the behavior of any one individual may be difficult to predict or explain, group-level properties and patterns do exist (for example, in birth rates and voting) and can be predicted and explained. Thus, aims, beliefs, expectations, choices, and intentions are relatively homogeneous at a group-level, and therefore understanding individual idiosyncrasies is both unnecessary and pointless.

Beyond these broad epistemological objections, I would argue that the movement toward a postmodernistic health promotion policy has potentially disastrous consequences in that it entails the abandonment of trying to rationally understand whether our attempts to improve health are effective and worth expending resources on. If the postmodern edict that explanation of real social events or processes is impossible to achieve is followed to its logical conclusion, then all anybody (including a researcher) can ultimately do is present his or her unique subjective understanding of that event or process.

An example, taken from the work of one of the main proponents of the postmodernist approach to evaluation, will serve to illustrate this. The “evaluation” of the Healthy Communities initiative described by Nina Wallerstein in Social Science and Medicine in 1999 took place over a period of four years in New Mexico. Wallerstein asserts that this is simply too short a period over which to expect any change in outcome variables related to the health status of the recipients of the intervention; hence, none are presented. (It is worth noting that many “positivistic” evaluations of interventions, such as the U.S. Department of Housing and Urban Development’s Moving to Opportunities demonstration project, have shown success in assessing health-related outcomes over comparable periods of time.) Rather than assessing program outcomes, what Wallerstein chooses to do is describe her “learnings” (sic) concerning “the power relationships between myself as evaluator and the communities during the implementation of the evaluation.”

It probably comes as no surprise that Wallerstein’s tale is not a happy one. Two main villains loom large throughout. The first of these is the New Mexico state government agency that funded the intervention. The agency seems not to have shared Wallerstein’s emancipatory ideology and by the second year of the project had the temerity to demand more accountability from the community groups in terms of the actual activities that were being undertaken. For Wallerstein this was not a case of the state government acting responsibly with tax dollars, but rather an expression of the “illegitimate abuses of power over others by the funding agency.”

The other villain of the piece is scientific knowledge. Evaluators who succumb to this form of “power/discourse” are engaged in a misguided search for a nonexistent objective reality. Moreover, according to the tenets of postmodernist theory, one of the primary motives in doing so is the “pursuit of career advancement,” which is engaged in “often at the expense of the community.” True to her postmodernist creed, Wallerstein challenges the positivist social scientist’s claim to objectivity and value-free science, claiming instead that science is just one of “a multiplicity of truths” and the researcher just one of many storytellers and interpreters of what happens in the world. Even worse, scientific data represent the “power/knowledge” discourse of positivist researchers, and this is frequently at odds with the “subjugated knowledge” of the community at whose expense they pursue their own interests.

The scientific knowledge power/discourse does in fact emerge from Wallerstein’s learnings as a formidable enemy, capable of taking possession of an evaluator much like the monster in Ridley Scott’s Alien. Indeed, Wallerstein herself succumbed to the insidious influence of science at one point in her project. Here’s how it happened. At the start of the project, the graduate students conducting Wallerstein’s interviews had a list of potential interviewees that had been given to them by the community coalitions. Over the course of the project she added others to the list without first clearing this with the coalition members. The decision to change the list in this way, Wallerstein claims, was justified “as a scientific decision (accepting the scientific knowledge/power discourse).” Nothing better demonstrates Wallerstein’s limited understanding of what science actually is than this statement. Neither of the lists she used had anything whatsoever to do with science. Each has a very specific set of biases — the first those of the community leaders and the second those of the researchers. Now, Wallerstein could have brought some rigor to the task of interviewing community leaders — for example, by generating a comprehensive list of potential interviewees and randomly selecting subjects from it. This would have produced a more representative pool of potential interviewees, and almost certainly more useful data. And it is hard to think of a reason why introducing such rigor to the data collection process would in any way constitute subjugating, exploiting, or abusing anyone.

To their credit, some of the community group members appear to have been unimpressed by Wallerstein’s efforts at emancipation and liberation. Not only did they take umbrage at her ad hoc interviewing strategy, they also objected to questions that she wanted to ask regarding the diversity of the coalitions (specifically those related to sexual orientation), and a number of the assertions made in the report she prepared were flatly rejected. One group, for example, questioned the validity of the information contained in quotations taken from her interviews regarding the underrepresentation of Hispanics. Wallerstein attributed this “resistance” to her initial failure to recognize her own position of privilege and power that derived from her status as an urban, Anglo university professor (her only “equalizing identity” being her gender). However, an equally plausible explanation of the community group’s reaction is that its members realized that there was something fundamentally wrong with a researcher (whatever her race, gender, or place of residence) seemingly picking quotes out of interviews in an unstandardized manner. (Wallerstein does not report using any systematic procedure to extract information from the interviews; presumably this would have involved further succumbing to the scientific knowledge power/discourse.)

It is worth noting in this regard that while denying the existence of objective knowledge elsewhere in her essay, Wallerstein describes her own report as containing “data” that were “hotly denied” by some community groups. Similar lapses into reality appear elsewhere. For example, on page 43 of the paper she states that postmodernist evaluators, rather than accepting epidemiologic data as “objective reality,” would study them to reveal their “political context and local meaning.” However, on the very next page she argues that children and families in New Mexico face some of the “worst health statistics in the nation,” and proceeds to present epidemiologic data pertaining to prenatal care, drunk-driving fatalities, access to care, teen pregnancy, and poverty in order to substantiate this statement.

Will the experiences of the people of New Mexico and particularly their health be in any way affected for the better by Wallerstein’s postmodernist empowerment evaluation? We will of course never know, since assessing such outcomes is of absolutely no interest or concern to the postmodern researcher.

Pseudoscience and health promotion

The key feature of the pseudoscientific movement within health promotion is that it superficially displays all of the trappings of real science — random assignment of subjects to study conditions, collection of quantifiable data, and use of statistical analysis — but makes no attempt at serious hypothesis testing. These tendencies are most prevalent in school-based efforts intended to ameliorate a range of undesirable behaviors among adolescents such as drug use, violence, and having sex. I have written at length about the prevalence of pseudoscience in the evaluation of these programs over the past seven years. Here I will confine myself to one example — the Seattle Social Development Project (ssdp). Like Linn Goldberg’s atlas program, the ssdp has been the recipient of numerous “best of” awards in recent years. It is a Department of Education “promising program,” both a csap and a Department of Health and Human Services “model program,” and is listed in the National Institute on Drug Abuse’s Research-Based Guide of 1997.

The ssdp was developed by David Hawkins and Richard Catalano of the University of Washington in Seattle, and is based on a conceptual model that stresses the importance of children developing strong bonds to the principal institutions of socialization within society, namely the family and school. Its program combines parent education, training teachers in classroom management practices, and skills training for students. The bonding hypothesis is reasonable, and the program an acceptable means whereby this bonding might be achieved. But what do the data show?

The outcomes reported in the evaluations of the ssdp have broadened out over the years. Initial reports from the project focused on its effects on delinquent behavior and aggression. Later accounts focused on its effects on alcohol use, while a paper authored by Hawkins and colleagues in the Archives of Pediatric and Adolescent Medicine in 1999 reported effects on three sets of outcomes at age 18 years — violent and non-violent criminal behavior, alcohol and drug (substance) use, and sexual activity. Specifically, there were four variables assessed for each of the three outcome behaviors used in the study. These pertained to “lifetime” prevalence of the behavior — for example, whether the subjects had ever committed a violent crime, ever been arrested, ever used alcohol, ever used marijuana, ever had or fathered a baby, or had multiple sexual partners. In addition, “heavy use” of alcohol, cigarettes, and marijuana and prevalence of drunk driving over the previous year were reported.

The results of the study show that there were no statistically significant differences between the intervention and control groups on any of the lifetime substance use measures. There was one statistically significant difference on the four lifetime crime measures, and two such differences on the lifetime sexual activity measures. Of the four previous-year measures, the only statistically significant difference found was for heavy alcohol use. In total, then, just four of the sixteen measures used in the study showed a statistically significant difference between participants in the ssdp and control subjects. Moreover, these effects were all limited to those that received a full dose of the program (that is, those who participated from first grade onward). Those who began the program in fifth grade did not differ from the control group on any of the outcome variables. Also, only about half of the full intervention subjects that started the program in first grade were followed up at 18 years—a very high rate of attrition.

All in all, this would seem to be sufficient evidence to falsify the hypothesis that the program has an effect on a range of behaviors — or so one would think. Not according to Hawkins and colleagues, who assert that the intervention had “enduring effects in reducing violent behavior, heavy drinking, and sexual intercourse by age 18 years.” Now while the developers of the ssdp might be excused for still thinking favorably of their creation despite this set of findings, it was on the basis of this evidence that the program was conferred “best of” status by federal agencies such as the Department of Health and Human Services, the National Institute on Drug Abuse, and csap.

Critical thinking vs. political thinking

Not all health promotion evaluations exhibit the weaknesses discussed herein. For example, Harold Holder and his colleagues at the Prevention Research Center in Berkeley recently conducted a methodologically rigorous community trial of an intervention designed to reduce alcohol-related injury. The published accounts from this study in such journals as the Evaluation Review and Journal of the American Medical Association describe in detail the efforts that were put into isolating the effects of the intervention from other influences on alcohol-related injuries, measuring implementation of the various program components, and assessing its effects on important outcome variables such as motor vehicle crashes and assaults. It is worth noting that Holder and colleagues employed community mobilization as a strategy in their intervention, but were able to quantify its implementation without seemingly exploiting or subjugating the communities in their study.

In the area of school health, the recently completed Hutchinson Smoking Prevention Project of Arthur Petersen and associates was also methodologically sophisticated and the authors were willing to report in their 2000 paper in the Journal of the National Cancer Institute that the program had no effect on smoking. It is also worth noting that both the Holder and Petersen evaluations were funded by the federal government, as was the methodologically ambitious Moving to Opportunities demonstration project mentioned earlier. Thus, the news is not all bad.

It remains true, however, that most evaluations in this area have come to resemble exercises in what the philosopher Mark Notturno called, in his book Science and the Open Society (Central European University Press, 2000), “political thinking” — that is, thinking motivated by the need to be accepted by and to defer to apparent authorities. In the case of the anti-science evident in community-based health promotion, the authorities are those that have declared a new postmodernist era in program evaluation. The heretics are those who hang on to their old positivist ways — individuals who at best are misguided or at worse agents of oppression and exploitation. Vanquishing these positivists is what passes for political activism among postmodernist academics. Hence, silencing those who suggest that research evidence be used to determine whether ideologically driven programs such as “Boy Talk” and “Girl Power!” are funded is perfectly acceptable.

In the case of the pseudoscience that dominates school-based prevention, the authorities are the expert panels that are convened by government and private agencies to determine what is “best practice” or “research-based” or “exemplary.” If an independent researcher examines the evidence pertaining to any of the approved programs and contests a panel’s judgment, this leads not to a critical analysis and discussion of the relevant data but to questions as to the adequacy of the individual who raised the questions. What both of these forms of political thinking demand is deference to authority and solidarity, not a search for the truth about efforts to promote health.

It is interesting to note in this respect that both the proponents of postmodernist evaluation and the proponents of “research-based” health promotion invoke the idea of a paradigm shift in describing their activities. As noted above, the former explicitly present their approach to theory, research, and practice as a radical break with what they call “positivism.” For their part, the advocates of the “research-based” approach present this as a fundamental break with previous approaches such as information dissemination, fear arousal, moral appeals, and alternatives. This is significant, for, as Notturno observes, what is most debilitating about paradigms is that they require the uncritical commitment of followers: Allegiance to the community of fellow adherents and solidarity of belief are what matter, not rational and free thought. Thus, censoring critics and ostracizing those who have yet to convert are not aberrations; they are precisely the functions that the paradigm-based community is intended to serve.