The CCNY team's work on evaluation design and evidence-based practice has lead us to believe that there are valid models beyond the experimental methodology. The Malcom Wiener Center for Social Policy at the Harvard Kennedy School of Government posted a working paper titled "A Lot to Lose: A Call to Rethink What Constitutes “Evidence” in Finding Social Interventions That Work" by Katya Fels Smyth and Lisbeth B. Schorr, January 2009. This paper challenges the dogma regarding the use of experimental design as the only real methodology to judge a social program. We look forward to following its reaction in the social policy and social work learning communities. Below are quotes from the article, the abstract, and a link to download your own copy.
Quotes:
"In assessing the success of efforts to improve outcomes for vulnerable populations, experimental methods must not be the sole arbiter of effectiveness."
"Interventions whose program design will not allow experimental evaluation, meanwhile, are deemed unproven—and, to many funders, “unproven” equates with a passing fad or an idea that is unlikely to deliver concrete results."
"While it is reasonable to expect that a laboratory test conducted in Omaha will be replicable in a similarly equipped laboratory in Ottawa or Oslo, it is harder to conceive that an intervention for suspected child abuse will translate exactly from the South Bronx to Sausalito to San Antonio."
Abstract:
A growing emphasis on accountability has led policy makers, funders, practitioners and researchers to demand greater evidence that program models “work” and that public and private dollars invested are generating relevant results that can be directly attributed to the given intervention. The gold standard for making these judgments is presumed to be the experimental–design study. In this paper, the authors suggest that the underlying assumption that everything that “works” can be judged with the same methodology has dramatic negative consequences for the field, for funders, and for those that desperately need high quality programs. The authors describe the characteristics of What It Takes organizations, which their work suggests support lasting change in the lives of highly marginalized and vulnerable people. They describe the ways that experimental methodology is a poor fit for judging the impact of these program models, while they find insufficient use of more appropriate ways of assessing their impact. They identify the risks inherent in the continued privileging of experimental designs over all others, and suggest that the risks are heightened in periods of great economic stress, when the pressure for accountability is increased. The authors suggest a set of starting points for rethinking evaluation to ensure greater accountability without reducing the chances that those who need help the most will have access to programs that support meaningful, lasting change.
"A Lot to Lose: A Call to Rethink What Constitutes “Evidence”in Finding Social Interventions That Work." (LINK TO PDF FILE)
