Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Words and Concepts Matter: Ten Commandments of Social Science Research by Shawn C. Marsh, Ph.D., and Patricia E. Campie, Ph.D. Social scientists and juvenile justice professionals often experience a disconnect when it comes to research. Lack of shared meaning concerning research terms and concepts contributes to this disconnect and can lead to misuse of research to drive policy and practice. Drawing on our experience as researchers and our work with juvenile and family courts across the nation, we propose the following Ten Commandments of Social Science Research to improve communication and understanding and to encourage appropriate use of research within juvenile justice. 1. Thou shalt embrace theory. Theory is critical to good social science research. Theory provides a framework for developing testable hypotheses and making predictions. Research conducted in the absence of a guiding theory lacks structure and contributes little to our fund of knowledge, since mechanisms for explaining findings are absent. Social scientists do not seek to prove theory; rather, they seek to test and refine theory through a disciplined research program. 2. Thou shalt be skeptical. Skepticism is the hallmark of good science. There is no perfect study, and research should always be assessed through a lens of caution coupled with vigorous exploration of alternative explanations for findings. Accordingly, scientific knowledge is best viewed as a cumulative endeavor that develops and refines over time with multiple studies and replications. 3. Thou shalt not confuse anecdote with evidence. Social scientists heed the old saying, “The plural of anecdote is not evidence.” Personal experiences and examples of apparently successful practices are valuable, but they do not meet the rigorous standards of scientific evidence. Human beings are susceptible to biases whereby they gather, retain, and promote information that is consistent with their experiences and worldview. Social scientists seek to limit these tendencies by using a strong theoretical lens, mapping out research methods a priori, and employing multiple perspectives on the phenomenon under investigation. Shawn C. Marsh, Ph.D., is the Director of the Juvenile and Family Law Department of the National Council of Juvenile and Family Court Judges. Patricia E. Campie, Ph.D., is the Director of the National Center for Juvenile Justice of the National Council of Juvenile and Family Court Judges. 4. Thou shalt embrace quantitative and qualitative methods equally. The debate about whether quantitative research or qualitative research is superior is dead; both approaches are equally important. Quantitative and qualitative methods have different advantages and disadvantages and tend to inform one another. Often qualitative continued on page 9 spring 2009 RAPPORT • Words and Concepts Matter: Ten Commandments of Social Science Research continued from page 8 research develops a rich description of the phenomenon that leads to testing of hypotheses through quantitative methods. Multimethod approaches are often ideal for social science research done in juvenile justice settings because they draw from complementary quantitative and qualitative data sources to create a more comprehensive understanding of the process and outcomes observed. 5. Thou shalt use proven with caution. The word “prove” reflects causal language and is rarely used by social scientists. Establishing causation in the social sciences outside of laboratory settings is extremely difficult. Experimental designs that are most likely to generate support for causation require random assignment of subjects to treatment and control groups. This randomization condition is rare in juvenile justice settings where there is limited control over who participates in interventions. In addition, ethical concerns arise when some youth might receive effective interventions while others randomly receive less effective or no interventions. Instead, applied research in juvenile justice settings often relies on comparing processes and outcomes for youth who receive an intervention with a demographically similar group of youth who have not received the intervention. 6. Thou shalt be mindful of the difference between valid and reliable. These two terms are often used interchangeably by consumers of research. Although there are many different types of validity and reliability, validity generally refers to “measuring what you intended to measure.” In contrast, reliability generally refers to “consistency in measurement.” For an instrument to be valid, it also must be reliable. However, you can have a measure that is reliable but not valid (i.e., you consistently measure the wrong thing). Although this might seem to be a nuance of little import, mixing the two terms or using them inappropriately can produce misleading information that adversely affects the quality of policy decision-making. 7. Thou shalt not use significant when one means substantial. In research, “significance” is a statistical term that refers to the probability of an event occurring. For a variety of reasons, it is somewhat rare to find statistically significant differences between youth outcomes based on their participation in local programs. Unless the amount of statistical significance continued on page 10 spring 2009 RAPPORT • Words and Concepts Matter: Ten Commandments of Social Science Research continued from page 9 has been calculated and reported, it is more accurate to use the term “substantial” when intending to emphasize the magnitude of a change or difference between subjects receiving a given intervention. 8. Thou shalt not pluck language from abstracts without reading the article. Abstracts, by definition, provide a concise summary of a study to assist readers in assessing whether the article should be read. Important details or context contained in the full article often are not included in the abstract because of space limitations. Thus, using findings summarized in the abstract to support your position might be inaccurate and could contribute to spreading wildly generalized misinformation to the field. 9. Thou shalt not ‘cherry pick’ studies to make your point. Good research presents all sides of a story. This presentation should include findings from prior research that support your position as well as those that do not. Picking out studies to make your point without presenting conflicting work is unethical and does nothing to promote progress in the field. 10. Thou shalt not use one study to drive policy. In the context of juvenile justice research and policy work, this is likely the most important commandment. Using one study—or even a handful of studies—to drive policy is both inappropriate and potentially harmful. As noted previously, science is cumulative. Policy should be based on a body of robust evidence that is grounded in strong theory, is methodologically sound, and uses a peer-reviewed process for separating the wheat from the chaff. Social science research can be complex and confusing, and many other commandments could be generated. The commandments presented here are only a modest step toward improving communication and shared meaning between social scientists and juvenile justice professionals as a means to encourage the most appropriate and meaningful use of research in applied settings. spring 2009 RAPPORT • 10