Download Ten Commandments of Social Science Research

Words and Concepts Matter: Ten Commandments
of Social Science Research
by Shawn C. Marsh, Ph.D., and Patricia E. Campie, Ph.D.
Social scientists and juvenile justice professionals often experience a
disconnect when it comes to research. Lack of shared meaning concerning
research terms and concepts contributes to this disconnect and can lead to
misuse of research to drive policy and practice. Drawing on our experience as
researchers and our work with juvenile and family courts across the nation,
we propose the following Ten Commandments of Social Science Research to
improve communication and understanding and to encourage appropriate
use of research within juvenile justice.
1. Thou shalt embrace theory.
Theory is critical to good social science research. Theory provides a framework
for developing testable hypotheses and making predictions. Research
conducted in the absence of a guiding theory lacks structure and contributes
little to our fund of knowledge, since mechanisms for explaining findings are
absent. Social scientists do not seek to prove theory; rather, they seek to test
and refine theory through a disciplined research program.
2. Thou shalt be skeptical.
Skepticism is the hallmark of good science. There is no perfect study, and
research should always be assessed through a lens of caution coupled with
vigorous exploration of alternative explanations for findings. Accordingly,
scientific knowledge is best viewed as a cumulative endeavor that develops
and refines over time with multiple studies and replications.
3. Thou shalt not confuse anecdote with evidence.
Social scientists heed the old saying, “The plural of anecdote is not evidence.”
Personal experiences and examples of apparently successful practices are
valuable, but they do not meet the rigorous standards of scientific evidence.
Human beings are susceptible to biases whereby they gather, retain, and
promote information that is consistent with their experiences and worldview.
Social scientists seek to limit these tendencies by using a strong theoretical
lens, mapping out research methods a priori, and employing multiple
perspectives on the phenomenon under investigation.
Shawn C. Marsh,
Ph.D., is the
Director of the
Juvenile and
Family Law
Department of the
National Council
of Juvenile and
Family Court
Judges.
Patricia E. Campie,
Ph.D., is the
Director of the
National Center
for Juvenile Justice
of the National
Council of Juvenile
and Family Court
Judges.
4. Thou shalt embrace quantitative and qualitative
methods equally.
The debate about whether quantitative research or qualitative research is
superior is dead; both approaches are equally important. Quantitative and
qualitative methods have different advantages and disadvantages and tend
to inform one another. Often qualitative
continued on page 9
spring 2009 RAPPORT • Words and Concepts Matter: Ten Commandments
of Social Science Research
continued from page 8
research develops a rich description of the phenomenon that leads to testing
of hypotheses through quantitative methods. Multimethod approaches are
often ideal for social science research done in juvenile justice settings because
they draw from complementary quantitative and qualitative data sources to
create a more comprehensive understanding of the process and outcomes
observed.
5. Thou shalt use proven with caution.
The word “prove” reflects causal language and is rarely used by social
scientists. Establishing causation in the social sciences outside of laboratory
settings is extremely difficult. Experimental designs that are most likely
to generate support for causation require random assignment of subjects
to treatment and control groups. This randomization condition is rare in
juvenile justice settings where there is limited control over who participates
in interventions. In addition, ethical concerns arise when some youth might
receive effective interventions while others randomly receive less effective
or no interventions. Instead, applied research in juvenile justice settings
often relies on comparing processes and outcomes for youth who receive an
intervention with a demographically similar group of youth who have not
received the intervention.
6. Thou shalt be mindful of the difference between valid
and reliable.
These two terms are often used interchangeably by consumers of research.
Although there are many different types of validity and reliability, validity
generally refers to “measuring what you intended to measure.” In
contrast, reliability generally refers to “consistency in measurement.” For
an instrument to be valid, it also must be reliable. However, you can have
a measure that is reliable but not valid (i.e., you consistently measure the
wrong thing). Although this might seem to be a nuance of little import,
mixing the two terms or using them inappropriately can produce misleading
information that adversely affects the quality of policy decision-making.
7. Thou shalt not use significant when one means
substantial.
In research, “significance” is a statistical term that refers to the probability
of an event occurring. For a variety of reasons, it is somewhat rare to find
statistically significant differences between youth outcomes based on their
participation in local programs. Unless the amount of statistical significance
continued on page 10
spring 2009 RAPPORT • Words and Concepts Matter: Ten Commandments
of Social Science Research
continued from page 9
has been calculated and reported, it is more accurate to use the term
“substantial” when intending to emphasize the magnitude of a change or
difference between subjects receiving a given intervention.
8. Thou shalt not pluck language from abstracts without
reading the article.
Abstracts, by definition, provide a concise summary of a study to assist
readers in assessing whether the article should be read. Important details or
context contained in the full article often are not included in the abstract
because of space limitations. Thus, using findings summarized in the abstract
to support your position might be inaccurate and could contribute to
spreading wildly generalized misinformation to the field.
9. Thou shalt not ‘cherry pick’ studies to make
your point.
Good research presents all sides of a story. This presentation should
include findings from prior research that support your position as well
as those that do not. Picking out studies to make your point without
presenting conflicting work is unethical and does nothing to promote
progress in the field.
10. Thou shalt not use one study to drive policy.
In the context of juvenile justice research and policy work, this is likely the
most important commandment. Using one study—or even a handful of
studies—to drive policy is both inappropriate and potentially harmful. As
noted previously, science is cumulative. Policy should be based on a body of
robust evidence that is grounded in strong theory, is methodologically sound,
and uses a peer-reviewed process for separating the wheat from the chaff.
Social science research can be complex and confusing, and many other
commandments could be generated. The commandments presented here are
only a modest step toward improving communication and shared meaning
between social scientists and juvenile justice professionals as a means to
encourage the most appropriate and meaningful use of research in applied
settings.
spring 2009 RAPPORT • 10