Download ECRAN European Communication on Research Awareness Needs

European Communication
on Research Awareness Needs
WP6 – EBM guide
for healthcare journalists to the
critical interpretation of clinical trials
WP6
ECRAN – EBM guide for healthcare journalists
What is quality of scientific information?
Let’s sort out the wheat from the chaff …
The huge amount of research studies is a big challenge for
health care journalists, who have to sort out the wheat of
scientific excellence from the chaff of low quality or interested
driven science. The purpose of this guide is to ensure
excellence in medical journalism by means of:
•applying the core principles of journalistic coverage and
investigation in the field of science and medicine;
•training journalists to properly cover news events, trends, and
issues in all aspects of health care, sorting out the tangle of
data about clinical trials and transferring them to medical
practice in a comprehensive and multifactorial perspective; …
WP6
WP3 task
3.2 –guide
Others
ECRAN
– EBM
for (1)
healthcare journalists
…
• enabling properly trained journalists to better convey the
complex dimensions of health care challenges from an
overall point of view;
• ensuring an independent coverage about health and
research, analyzing health related data with fairness,
accuracy and balance;
in order to:
• increase the capacity of individuals and policymakers to
adopt health care attitudes and policies that can improve
the health of all the citizens;
• improve the ability of citizens to make wise decisions
about behaviors that promote health, treatment options,
and their choice of health care providers.
WP6
WP3 task
3.2 –guide
Others
ECRAN
– EBM
for (2)
healthcare journalists
What is science?
«A system of certain knowledge in a period of time built up,
arranged and ensured by research, teaching and
accumulated data».
(translation from the German definition in Brockhaus 2003:2193)
Which is the goal of science?
The ultimate goal of science must always be to verify or
reject existing (content-specific or methodical) knowledge
and to replace it with new knowledge. Science is a
cumulative process. The touchstone for assessing new
knowledge is the comparison with existing knowledge.
Systematic reviews, that collect and evaluate knowledge of
several scientific studies, have been developed and
implemented for 20 years as methodical tool for assessing
research.
WP6
WP3 task
3.2 –guide
Others
ECRAN
– EBM
for (3)
healthcare journalists
How should good science look like?
Assessing good science according to its neutrality and objectivity
is unrealistic, because of its several infrastructural, financial and
performance-related implications. In the last decade the call of
transparency has been playing a central role in this process, in
order to ensure the integrity of research. Assessment of the
framework conditions for research should be therefore integrative
part of each journalistic investigation, in order to set an indicator
for a preliminary assessment of the reliability of research
outcomes.
How should good journalistic investigation look like?
Basic scientific understanding of scientific studies and their weak
points enables journalists to facilitate orientation and to avoid
lurking pitfalls in this field (e.g. look at the section “Tips for
understanding studies – Common pitfalls” of this guide).
WP6
ECRAN – EBM guide for healthcare journalists
Toolkit for journalists
There are many challenges to achieving accuracy, balance
and completeness in health care journalism. For a reporter,
there are deadlines, editors, and corporate financial
pressures.
Though quality in health care journalism is far more than
this, because it plays a major role in educating consumers!
Consumers need to be well informed to make sound
choices that affect their health and well being.
The following rating criteria are an internationally validated
methodology, that helps conjugate general and medical
journalistic criteria to review stories claiming about specific
treatments, tests, products or procedures (i.e. drugs or
devices, vitamins or nutritional supplement, diagnostic and
screening tests, dietary recommendations, surgical
procedures).
WP6
ECRAN – EBM guide for healthcare journalists
Rating criteria for healthcare journalists
1 Quantification of benefits
(treatment/test/product/
procedure)
4 Further independent
source and declaration of
conflicts of interest
7 Comparing new
approach with existing
alternatives
2 Quantification of harms
of intervention
3 Proper evaluation of
quality of evidence
5 Story solely or largely
based on news release?
6 Establishing true
novelty of approach
8 Establishing availability
of treatment/test/product/
procedure
9 Adequate estimate of
costs of intervention
10 Is there potential for
disease mongering?
WP6
ECRAN – Rating criteria for healthcare journalists
1
Quantification of benefits
(treatment/test/product/procedure)
Stories are expected to describe the effectiveness of a
benefit. Few stories provide helpful numbers to back the
real benefit of a new treatment, test, product or procedure.
Empirical investigations have shown that news reports,
drug ads and some studies coverage tend to report
benefits as the relative reduction instead of the absolute
risk reduction in the frequency of failures.
Another challenging question may be: «How many patients
have to be treated in order to prevent the problem for one
patient?». This is the Number Needed to Treat (NNT): it is
defined as the inverse of the absolute risk reduction and is
the average number of patients who need to be treated to
prevent one additional bad outcome.
WP6
ECRAN – Rating criteria for healthcare journalists
1
What does “significantly increased” mean? How was this
measured? Apart from this, statistical significance may not
be equal to clinical relevance. The real question is: «What
difference does it make in peoples’ lives?» Reports
indicating surrogates and intermediate endpoints, e.g.
changes in blood test scores, may not influence individual
health scores (increased life expectancy or lower mortality
rate) or quality of live.
Moreover, anecdotal storytelling makes engaging reading
but is not scientific data. Glowing personal stories or single
unchallenged exaggerated success stories may not be a
representative example of how a benefit works and require
a critical appraisal for a balanced framing of it.
WP6
ECRAN – Rating criteria for healthcare journalists
1
Criterion will not be fulfilled, in case of …
 Data of possible benefit is not available at all.
 Using only relative, not absolute, risk reduction or benefit
data.
 Relying too heavily on what may be unrepresentative
patient anecdotes about benefits.
 Citing statistics appropriately, but allowing even a single
unchallenged exaggerated quote to throw the story out of
balance.
Question to clear at the end of the day
Is the benefit really as much more or less significant as it is
presented?
WP6
ECRAN – Rating criteria for healthcare journalists
2
Quantification of harms of intervention
Be aware of appearances! When you hear someone saying
that a new approach «appears to be safe», look for
evidence supporting it.
Stories may emphasize or exaggerate potential benefits of
interventions, while minimizing or completely ignoring their
potential harms. For some patients slight side effects may
even have dramatic impact on their lives. An overall
healthcare report should always consider possible – even
the slightest – adverse effects of interventions, ideally
reporting their frequency and severity.
Any healthcare decision implies tradeoffs, all treatments
may have side effects, risks and complications.
WP6
ECRAN – Rating criteria for healthcare journalists
2
Even a screening test may turn out to be harmful to the
individual getting it: in fact, it may produce a false positive
result, which causes anxiety, and will probably lead to more
testing (i.e. biopsy) and possibly treatment (i.e. surgical
intervention) that comes with its own harms. Moreover,
enthusiasm for early tests in people without symptoms is
often not supported by evidence.
PSA testing, for example, often turns out to cause more
harms than benefits, including false positive results and
overdiagnosis, both of which could lead to unnecessary
treatments associated with significant side effects.
WP6
ECRAN – Rating criteria for healthcare journalists
2
Criterion will not be fulfilled, in case of …
Failing to mention potential harms.
Failing to quantify potential harms.
Failing to describe the severity of potential harms.
Failing to account for “minor” side effects that could have
a significant impact on a patient’s life.
Relying too heavily on a patient anecdote about safety.
Relying too heavily on a researcher’s comment that an
approach appears to be safe – without supporting data.
Question to clear at the end of the day
Which are the risks and the side effects of this intervention
or product?
WP6
ECRAN – Rating criteria for healthcare journalists
3
Proper evaluation of quality of evidence
Healthcare coverage should critically evaluate the evidence
and not merely take published or presented research as
gospel. Quality of evidence has to be classified according
to evidence hierarchy:
WP6
ECRAN – Rating criteria for healthcare journalists
3
Science works cyclically with ebbs and flows: knowledge
stream keeps flowing and keeps changing in the course of
time. So, ask yourself:
At which point in the stream did I grasp the evidence?
Not all evidence is bullet-proof, even if it is published in
high impact journals, because not all studies are equal.
Eminence is not evidence and study stories often turn out
to be ongoing stories.
A solid journalistic investigation is expected to scrutinize
limitations of evidence, number of probands, duration of the
study, relevance of outcomes; to assess whether what
appears as the latest story is already existing in larger and
better-designed stories on the same topic or whether the
statements generalize small group data (e.g. a single
medical center with experienced surgeons).
WP6
ECRAN – Rating criteria for healthcare journalists
3
Criterion will not be fulfilled, in case of …
Failing to dig out the limits of the study design (i.e. lack of control group, lack
of blinding, missing drop out rate).
Not cautioning on the interpretation of uncontrolled data.
Conflating causation and association (i.e. limitations of observational studies).
Generalizing small group data (i.e. at one medical center with experienced
surgeons).
Confounding surrogates with real health outcomes (illness or death vs.
markers or scores).
Not mentioning that scientific meeting outcomes lack adequate peer review.
Not pointing out that findings from animal or lab experiments are not directly
applicable to human health.
Presenting anecdotic data as evidence for intervention.
Question to clear at the end of the day
Does this study design justify these allegations?
WP6
ECRAN – Rating criteria for healthcare journalists
4
Further independent source and declaration of
conflicts of interest
Claims about new treatments, tests, products, and
procedures should be based on evidence, not on beliefs or
business hopes. In health care journalism, it is therefore
essential that second opinions be sought from independent
experts who have no vested interest in the approach being
discussed. For this scope two things must occur:
 there must be an independent expert source quoted
(someone not directly connected with the research),
there must be some attempt to let the public know about
potential conflicts of interest.
WP6
ECRAN – Rating criteria for healthcare journalists
4
Hyperbolic healthcare claims made by drug companies,
medical device manufacturers and academic research
centers are to be confronted with this question:
Do they have something to gain?
May be that: a trial is paid for by the
drug manufacturer; researchers are
getting fees from a drug company;
claims are made by a spokesman for
a device manufacturer; doctors are
being early adapters of a new device;
there
are
underlying
inventors’
interests.
WP6
ECRAN – Rating criteria for healthcare journalists
4
A study funded by industry doesn’t necessarily mean that
findings lack integrity, but evidence from recent studies
shows that potential conflicts might lurk around every
corner in health care. So, the next time you go to your
doctor’s office that is cluttered with drug company
promotions, it is reasonable for you to ask him if he/she
had any financial relationship with the makers of drugs,
devices or test advertised for. Moreover, seek independent
perspectives and beware of statements like: «Experts
believe …, doctor think …». Instead
Give them a name, because vagueness is nameless!
WP6
ECRAN – Rating criteria for healthcare journalists
4
Criterion will not be fulfilled, in case of …
Failing to identify the source of the story (news release,
journal article, editorial, scientific meeting presentation,
etc.) and failing to make obvious the extent to which that
source is likely to be conflicted (e.g. «A PR consultant
working for the company said…» or «Dr. Smith, who
received a company grant to perform the study, said…»)
Failing to include the input of a trusted independent
source with expertise on the subject to comment on the
claims made (ideally, more than one such independent
source would be used).
Question to clear at the end of the day
To what extent are the information independent and how
plausible is the expertise?
WP6
ECRAN – Rating criteria for healthcare journalists
5
Is the story solely or largely based on
news release?
It is simply not sound journalistic practice to lift material
directly from a news release without stating that this is the
case. If we know that a news release was sent out and if
we can get a copy of that release, we will review it along
with the story.
Many news organizations facing financial pressures may
be lead to become more open to using news releases as a
way to publish content while cutting costs. Although news
releases can be valid sources of some information,
journalism is being asked to collect independent
information with the consequence that it is unacceptable to
rely on a news release as the sole source of information.
WP6
ECRAN – Rating criteria for healthcare journalists
5
Particularly in health care, there are many vested interests
trying to influence consumer choices. Journalists are
expected to use independent verification relying neither on
news releases nor on company spokesmen!
A team from Dartmouth Medical School and the Veterans
Affairs Outcomes Group published in 2009 an analysis on
the quality of press releases by medical centers concluding
that «Press releases from academic medical centers often
promote research that has uncertain relevance to human
health and do not provide key facts or acknowledge
important limitations.»
WP6
ECRAN – Rating criteria for healthcare journalists
5
Criterion will not be fulfilled, in case of …
There is clear evidence that the exact same wording from
a news release appears in a story.
Lacking evidence of a press release and the worst a story
can be rated is “not applicable”.
Criterion will be fulfilled, in case of …
A story has several independent sources and proves
NOT to rely only on a news release.
Question to clear at the end of the day
Is the news based only on a press release?
WP6
ECRAN – Rating criteria for healthcare journalists
6
Establishing true novelty of approach
Many alledgedly new treatments, tests, products or
procedures are not really novel. The agent may be another
member of a well-established therapeutic class of drug.
Even if it represents a new class, it may offer no more than
the drugs that are widely available. In the press release for
a new drug this essential information may be lost in the
hype and the drug is portrayed as “novel” in order to
increase initial sales. So ask yourself:
Is it novel or simply new?
Journalists’ reporting should be able to make these
distinctions on the real novelty of a product by researching
or asking independent sources, e.g. specifiying that there
are previous studies on this agent for the same indication.
WP6
ECRAN – Rating criteria for healthcare journalists
6
In health care, newer isn’t always better, the first in a new
class of drugs may offer no more than the drugs that are
widely available. Drug companies are very good in
promoting the alledged novelty of a drug in order to
increase initial sales. Moreover, some claims for new drugs
are made for “off-label use”, i.e. for not officially approved
uses and under doctors’ careful monitoring. Off-label uses
might also result in wasting money and harming patients
and the respective costs may not be covered by the state
or private insurance. By focusing on a new idea journalists
should tell how well it works putting it into the context of
existing alternatives.
WP6
ECRAN – Rating criteria for healthcare journalists
6
Mostly it is rare that a study appears out of nowhere and
the “big picture”, where the “novelty” comes from, is often
missing. To find it out journalists should search for the
opinion of independent experts or reliable scientific sources
like.
WP6
ECRAN – Rating criteria for healthcare journalists
6
Criterion is will not be fulfilled, in case of …
Failing
to
accurately
describe
the
new
product/intervention and, as a consequence of this,
misleading the consumer.
Failing to explain the real novelty of the
product/intervention (preferably comparing it to existing
alternatives on the market).
Failing
to
mention
that
the
described
new
product/intervention is, in fact, for off-label uses.
Question to clear at the end of the day
Is the product/intervention a real novelty?
WP6
ECRAN – Rating criteria for healthcare journalists
7
Comparing new approach with existing
alternatives
It is mandatory for a healthcare journalist to put the new
treatment into the context of existing alternatives that have
a longer, more proven track record. Whatever the new
approach, the report should try to put it into the context of
already available interventions, in order to compare the
effectiveness of the various options. News stories on new
treatments, tests, products and procedures should also
take into consideration the value for money ratio.
In case a treatment should have been developed from
scratch, the journalist is expected to explain that there are
no alternative treatments or only supportive or soothing
remedies.
WP6
ECRAN – Rating criteria for healthcare journalists
7
“To wait and see” is also a plausible alternative to the
proposed
treatment/test/product/intervention.
“Active
surveillance”, declining immediate aggressive therapy, is a
viable and rationale option.
It is wise for patients to know their treatment options. They
may also consider to make lifestyle changes rather than
taking a drug, or else they may pursue a drug or nonsurgical option rather than having surgery.
Promotions of screening tests should also consider the
option for individuals to choose NOT to be screened.
WP6
ECRAN – Rating criteria for healthcare journalists
7
Criterion will not be fulfilled, in case of …
Discussing a surgical approach without mentioning nonsurgical alternative approaches.
Discussing a new test without mentioning other tests that
are available, including the option of not being screened in
the case of a screening test.
Failing to discuss the advantages/disadvantages of the
new idea compared with existing approaches.
Failing to discuss how the new treatment, test, product or
procedure fits into the realm of existing alternatives.
Question to clear at the end of the day
Are there any other treatment options besides this
procedure/product?
WP6
ECRAN – Rating criteria for healthcare journalists
8
Establishing availability of
treatment/test/product/procedure
This criterion shall mostly be applied to coverage about a
new drug in clinical trials (or in a trial on the new use of an
established drug). A journalist is expected to tell his/her
readers/public, if the drug/agent is available at the
chemist’s (or may be soon) or if it is really only in an earlyphase trial (e.g. pilot study or stage before approval).
It is important for a journalist to know that there may be a
lot of marketing hype that suggests that something is wellstudied and ready for use.
Questions should also be raised by the fact that something
should have been approved in the U.S. and not or not yet
in Europe.
WP6
ECRAN – Rating criteria for healthcare journalists
8
Some early studies focus on safety and not on how well
something works. Some other focus on early efficacy and
may not be large enough or designed to show safety of a
drug/treatment in the long term.
Predictions about «something to be approved by …» in a
certain period should also be taken with a grain of salt, they
usually come from someone who stands to benefit.
New approaches, devices or operations should be
scrutinized for their availability (no one wants to be patient
number one with a new device/operation).
Finally, have the personnel required to deliver the new
approach properly been trained and are there enough of
them? What is the learning curve for this?
WP6
ECRAN – Rating criteria for healthcare journalists
8
Criterion will not be fulfilled, in case of …
Making not clear that the drug is not available now.
The approval of a drug is treated as a closed issue.
Something «expected to be approved by …» and its availability on
the market sounds more like a crystal ball prediction.
Time predictions about the market availability of the drug are solely
based on declarations sourced to a company spokesperson.
Failing to describe to what extent a treatment is available (in every
hospital or only in specialty clinics).
Promoting a special intervention without providing a sense of how
many trained doctors there are in the field (e.g. a hi-tech surgical
intervention).
Question to clear at the end of the day
Is this item available now or not?
WP6
ECRAN – Rating criteria for healthcare journalists
9
Adequate estimate of costs of intervention
Journalists are expected to appropriately report on
intervention/product costs, though it may be difficult to
estimate costs of an experimental approach early in its
development. Even if the real cost is probably known at the
time of market entry, health reporters could at least cite costs
of existing alternatives. Journalists should be able to get a
reasonable estimate from their source/experts on the topic.
The rule of thumb is:
If it is not too early to talk about
how well something might work,
then it is not too early
to start discussing what it may cost
WP6
ECRAN – Rating criteria for healthcare journalists
9
A large amount of healthcare stories do not appropriately report on the
cost issue. Healthcare takes a large part of the Gross Domestic Product
in developed countries and we all better start thinking about costs of
treatments, tests, products and procedures, we all pay for.
So, it is not enough to say
The cost is much lower than …
but you should better ask
What does this actually cost?
How much lower?
What about the cost-effectiveness, if compared to …?
Journalists should also consider additional expenses for psychologists,
dieticians, physiotherapists involved in the intervention or that the
convalescence and rehabilitation may be more time-consuming.
Moreover, they should investigate, if insurances cover the
intervention/drug or it is affordable for the patients needing them.
WP6
ECRAN – Rating criteria for healthcare journalists
9
Criterion will not be fulfilled, in case of …
Costs of an intervention are not mentioned at all.
Costs are played down without giving a satisfactory explanation
therefore.
Not comparing the costs of a procedure with other available
alternatives (more or less expensive ones).
Not mentioning, if the new procedure/drug will be covered by
public/private health insurance.
Criterion will not apply, in case of …
An experimental procedure in early development with no mention of
the related costs.
An expert claiming that costs are quite unimportant (e.g. speaking of
well known products like aspirin or weight-loss products).
Question to clear at the end of the day
Is it covered by health insurance or must I pay out of my pocket?
WP6
ECRAN – Rating criteria for healthcare journalists
10
Is there potential for disease mongering?
This criterion assesses whether a condition is exaggerated or
oversold. There are different forms of “mongering”:
 turning risk factors into diseases (e.g. low bone mineral
density becomes osteoporosis);
 misrepresentation or inaccurate description of the natural
history and/or severity of a disease (e.g. early-stage low-grade
prostate cancer);
medicalization of minor or transient variations in function (e.g.
temporary erectile dysfunction or female ‘sexual dysfunction’);
 medicalization of normal states (baldness, wrinkles, shyness,
menopause);
exaggeration of the prevalence of a disease/disorder (e.g.
using rating scales to ‘diagnose’ erectile dysfunction).
WP6
ECRAN – Rating criteria for healthcare journalists
10
Some statements are trying to make all of us feel like there’s
something wrong with us – even if there isn’t. That’s what we call
disease-mongering. This is dangerous because it is an
unnecessary emotional and financial burden for society and may
distract us and precious public resources from important
healthcare issues that should be addressed. Also turning a
normal state or variation of health into a medical problem is
disease mongering, like baldness, wrinkles, shyness or
menopause. Pseudodiseases, that lower the threshold for
common diseases but, in fact, only open new markets, may turn
millions of people overnight into patients needed to be treated
with drugs or vitamins against high cholesterol or osteoporosis.
Often, women are targeted because of premenstrual dysphoric
disorders, female sexual dysfunctions, overactive bladders or
menopause; men equally get their share being hammered
because of low testosterone levels, balding etc.
WP6
ECRAN – Rating criteria for healthcare journalists
10
Identifying disease mongering is a matter of judgment.
Sometimes it is obvious. Sometimes there’s a fine line
about whether a statement on irritable bowel disorder,
erectile dysfunction, restless leg syndrome or osteoporosis
(all of which can be serious for some sufferers) is
misrepresenting the condition to the public. Sometimes
there may be “multiple mongering” with many hyperboles
for the same issue.
WP6
ECRAN – Rating criteria for healthcare journalists
10
Criterion will not be fulfilled, in case of …
Spurious statistics are presented. Question prevalence
estimates!
Statistics inflate the seriousness of a condition (e.g.
«millions of people `suffering` in silence with toenail
fungus…»)
 Someone “selling sickness” wants to scare you into
testing or treatment.
Risk factors are treated as diseases to convince us that
we should be concerned about “a score” (i.e. surrogate
markers, e.g. changed blood parameters) instead of
outcomes like illness or death.
Holding up “worst-case patients experiences” as
representative examples of all with this conditions.
WP6
ECRAN – Rating criteria for healthcare journalists
10
Criterion will not apply, in case of …
A
therapy/procedure/test/drug
possibly
being
overestimated. There is no «treatment mongering» as
such. This item should be best scrutinized under criterion
Quantification of benefit or Proper evaluation of quality of
evidence.
Question to clear at the end of the day
Should I really be concerned about this
disease/circumstance?
WP6
ECRAN – General journalistic criteria
11
Choosing the right issue
Is it a current, relevant and original contribution?
Topicality of an issue may be related to the event and the
frequency of medium publication, the importance or the
originality (i.e. curiosity) of its contents. Relevance may occur for
instance, if the issue directly or indirectly regards a large part of
the population, if a therapy determines high costs for the health
system, if a new development has an ethical dimension, if the
topic has political importance or an exemplary nature. If not
relevant, an issue may instead be curios, humorous, surprising
etc. Novelty of an issue may be characterized by latest data
published e.g. today, yesterday, this week in a scientific journal,
or outcomes of recently held conferences.
The rule of thumb is:
If the issue should not be current, then it must be relevant. If not
relevant, then it must be at least curios/original!
WP6
ECRAN – General journalistic criteria
12
Delivery of message
Has the issue been successfully delivered or
does it even have model character for its format?
A successful message must be delivered in an intelligible
language, with a plain structure avoiding long and complicated
sentences. Specialist or technical terms should only be used in
exceptional cases and, anyway, accordingly explained by means
of context or examples.
Formal factors also play an important role: the story must follow
dramaturgical principles (personalisation, narrative elements,
relation to everyday life); form and content must harmonise;
abstract thinking must be explained by pictures, illustrations,
clear specimen texts; good sound clips in a radio report should
invite to listen to medical issues. In case of reports, WH
questions should all be answered.
Rule of thumb: delivery standard is conform with the format
WP6
ECRAN – General journalistic criteria
13
Factual accuracy
Does the story report essential facts?
Factual accuracy means reporting historical facts without
manifest or essential errors, e.g. accurate reporting of study
conclusions or data, correct description of study design (i.e.
number of participants enrolled/allocated/followed up/analysed in
clinical trials). Accuracy is neither synonymous with objective
truth nor completeness, e.g. a report must not comprise each
and every aspect of the study but the described facts must be
true. A single name error is not enough to invalidate this
criterion, but many little mistakes may create the impression of a
real sloppy work!
Apart from purely data errors or hyperboles in the headlines or
teaser beware of surrogate markers, association vs. correlation,
absolute vs. relative risks, NNT, off-label use, phases of clinical
trials, confidence interval etc ...
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Surrogate markers
A surrogate marker is an event or a laboratory value that
researchers hope can serve as a reliable substitute for an
actual disease. A common example is blood cholesterol levels
which are surrogates, or substitutes, for heart disease.
As a matter of fact, surrogate studies are indirect by definition
and a journalist should be skeptical before celebrating the last
medical breakthrough, because what is sound medical dogma
today may fade over time. Surrogate research is an important
research tool, that can open important therapeutic avenues and
stimulate additional research. Those who earn their living and
reputation from medical research may have a different view on
surrogacy, but it is important to understand that exaggerations
may damage patient care by highlighting surrogate outcomes
instead of important results such as death, quality of life or
functional capacity.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Association vs correlation and language use
Look at this wording describing the results of an observational study:
«Frequent fish consumption was associated with a 50% reduction in the
relative risk of dying from a heart attack.»
And look at the editor’s streamline “adaption” detrimental to meaning:
«Women who ate fish five times a week cut their risk of dying later from a heart
attack by half.»
The new wording suggests that the subjects’ fish consumption is responsible
for their dying less frequently from heart attacks, but the original study does not
support a conclusion of cause and effect.
Epidemiologic – or observational – studies examine the association
between what is known in epidemiologic jargon as an exposure (e.g. a food,
environmental influences, lifestyle) and an outcome (often a disease or death).
Because of all the other exposures occurring simultaneously in the complex
lives of free-living humans that can never be completely accounted for, such
studies cannot provide evidence of cause and effect; they can only provide
evidence of some relationship (between exposure and outcome) that a
stronger design could explore further.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
The only study design involving humans that does rise to the level of demonstrating
cause and effect is a randomized trial. In this design, study subjects are assigned an
exposure (or a control condition) at random, irrespective of any other exposures in their
lives, and all such other exposures are then assumed to even out between the treated
group and the control group of subjects (and this can be demonstrated). As a result, the
only difference between the groups is whether they receive the exposure under study or
the control condition. This approach is a true experiment. Any difference in outcome
seen between the control and the experimental group should be due to the one factor or
variable that differs.
A subtle trap occurs in the transition from the cautious, nondirectional, noncausal,
passive scientifical language in reporting the results of observational studies to the
active language used in mass media. Active language is fine in general (who wants to
write like a scientist?), but problems can arise when the use of causal language is not
justified by the study design. In practice, a shift to causal language can occur at any
stage: writing, editing, or headline composing, with similar effects on meaning, that
could lead readers to overestimate the meaning of a given study and possibly even
make life choices that the evidence does not warrant. Sometimes even scientists and
press-release writers slide into causal language in expressing results of observational
studies.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Absolute vs relative risks
Researchers, clinicians or journalists often report only relative differences in
making claims about a new idea, but they should tell the rest of the story!
Absolute differences make the real sense out of such claims.
Look at this example about blindness in diabetic patients in a 5-year-period:
If the risk for blindness is 2 in 100 (2%) in a group of patients treated conventionally
and 1 in 100 (1%) in patients treated with a new drug, the absolute difference is derived
by simply subtracting the two risks: 2% – 1% = 1%.
Expressed as an absolute difference, the new drug reduces the 5-year risk for
blindness by 1%.
The relative difference is the ratio of the two risks. Given the data above, the relative
difference is: 1% ÷ 2% = 50%
Expressed as a relative difference, the new drug reduces the risk for blindness by half.
Data are accurate in both cases, but: if you are a marketing manager for the
new drug, you are likely to only use the relative risk reduction; if you are a
journalist, you would better do your job by citing the “rest of the story” (absolute
risk reduction) often missing in news releases and direct-to-consumer
prescription drug ads.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Number needed to treat (NNT)
The NNT is defined as the inverse of the absolute risk
reduction or the number of patients who need to be treated to
prevent one additional bad outcome, calculated as 1/absolute risk
reduction.
Applied to our hypothetical diabetes blindness drug example:
the risk for blindness in a patient with diabetes over a 5-year
period is 2 in 100 (2%) in a group of patients treated conventionally
and 1 in 100 (1%) in patients treated with a new drug. So the
absolute difference is derived by simply subtracting the two risks:
2% – 1% = 1%.
the number needed to treat would be 1 / 1% (or .01) = 100.
so you would need to treat 100 people with diabetes for 5 years in
order to prevent one case of blindness.
You can see that this is an important way to look at new claims
about new drugs.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Off-label use
Prescription drugs are officially approved by regulatory agencies for specific
indications to treat specific conditions or diseases. Physicians may prescribe
a drug for a use that is not described in the approved labeling if it seems
reasonable or appropriate to them. This is what’s called off-label use.
For example: it is not uncommon for physicians to prescribe low doses of beta
blocker drugs to help people overcome jitters before public speaking. Beta
blockers are not formally approved for this use, but in such circumstances
doctors are advised that they have the responsibility to be well informed about
the product, to base its use on firm scientific rationale and on sound medical
evidence, to inform the patients and get its consent for the use of the drug.
A study in the Archives of Internal Medicine found that about 21% of all
estimated uses for commonly prescribed medications were off-label, and that
15% of all estimated uses lacked scientific evidence of any useful or beneficial
effect. Authors of the study recommended more careful monitoring of off-label
uses that might waste money and harm patients. Another problem of the offlabel prescribing is that its cost may not be reimbursed by public or private
health insurances.
Finally, whilst off-label use is permitted, marketing of a drug for an off-label use
is not.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Phases of clinical trials
Journalists who report on drugs while they are still in clinical
trials need to understand the distinction between phases I, II,
and III of drug trials. It is misleading to report bold or conclusive
statements about how well a drug works when it is only in phase
I trials, since the primary goal of phase I trials is to evaluate how
safe a drug is, not how well it works. Many times journalists
report on early phase drug trials as if all the evidence is in hand.
Advice for healthcare journalists:
Give accurate portrayals of the status of investigational drugs,
devices and procedures, including significant caveats and
explanations of hurdles, unknown and potential problems.
Without such caveats consumers would be better advised to
have doubts about the accuracy and balance of the story.
WP6
ECRAN – Tips for understanding studies – Common pitfalls
Confidence interval
In the interpretation of trial results scientists divide errors
into two classes: a type I error is the mistake of thinking
something is true when it is not (also known as a false positive);
a type II error is thinking something is not true when in fact it is
(a false negative). When testing a specific hypothesis, scientists
run statistical checks to work out how likely it would be for data
which seem to support the idea to have come about simply by
chance. If the likelihood of such a false positive conclusion is
less than 5%, they deem the evidence that the hypothesis is truly
statistically significant. They are thus accepting that one result in
20 will be falsely positive – but one in 20 seems a satisfactory
low rate.
WP6
ECRAN – EBM guide for healthcare journalists
Valuable papers fur further reading
This list has been updated at November 2013
To find further contents, visit ECRAN website www.ecranproject.eu
- Opportunities for research and NIH
http://www.sciencemag.org/content/327/5961/36.summary
- Demystifying trial networks and network meta-analysis
http://www.bmj.com/content/346/bmj.f2914
- Independent drug testing to ensure drug safety and efficacy
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328348
- Editorial policies for clinical trials and the continued changes in medical journalism
http://jama.jamanetwork.com/article.aspx?articleid=1699471
- Informing the uninformed: optimizing the consent message using a fractional
factorial design
http://www.ncbi.nlm.nih.gov/pubmed/23700028
WP6
ECRAN – EBM guide for healthcare journalists
- The Women’s Health Initiative—A victory for women and their health
http://jama.jamanetwork.com/article.aspx?articleID=1745653
- Better reporting of scientific studies: why it matters
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001504
- Quantification of harms in cancer screening trials: literature preview
http://www.bmj.com/content/347/bmj.f5334
- Risks (and benefits) in comparative effectiveness research trials
http://www.nejm.org/doi/full/10.1056/NEJMp1309322
- The right to participate in high-risk research
http://philpapers.org/rec/SHATRT-2
- Who’s afraid of peer review?
http://www.sciencemag.org/content/342/6154/60.full
- Why we can’t trust clinical guidelines
http://www.bmj.com/content/346/bmj.f3830
- Practices and impact of primary outcome adjustment in randomized controlled trials:
meta-epidemiologic study
http://www.bmj.com/content/347/bmj.f4313
WP6
ECRAN – EBM guide for healthcare journalists
- US study criticized for experimentation with premature infants
http://www.bmj.com/content/347/bmj.f4198
- Do clinical trials work?
http://www.nytimes.com/2013/07/14/opinion/sunday/do-clinical-trialswork.html?pagewanted=all&_r=2&
- Understanding clinical trials: appreciating medical heroes
http://doc.mediaplanet.com/all_projects/4921.pdf
- Dispelling the many myths about clinical trials
http://www.newswise.com/articles/dispelling-the-many-myths-about-clinical-trials
WP6