Download So I found A Paper – now what do I do?

So I Found A Paper – now what
do I do?
How to quickly read a paper and critically (ap)praise it
Dr Ben Mitchell
MBBS BSc, FRACGP
Medical Educator GPTQ
Lecturer The University of Queensland
Session Outline
• SHOULD I CARE? (How to read a paper in 5
minutes)
• SHOULD I BELIEVE THE FINDINGS? (Approach to
critically appraising the paper and interpreting
the results)
• SO WHAT? (are the results clinically meaningful)
• Some online resources to help
Online critical appraisal tools
Helpful statistical articles
Who’s confident?
1 = Not confident
2 = Done a little before
3 = Fairly confident
4 = Pretty confident actually
5 = Very confident
Why “how to read a paper?”
• It can be difficult
• Its not like reading a book
• You want to be able to decide quickly if it will
answer your question (and thus help you decide how you will
spend the next 30 mins of your life)
• It gets easier with practice
A full process might involve:
• a one or two sentence summary of the paper.
• a deeper, more extensive outline of the main
points of the paper, including for example
assumptions made, arguments presented,
data analyzed, and conclusions drawn.
• any limitations or extensions you see for the
ideas in the paper.
• your opinion of the paper; primarily, the
quality of the ideas and its potential impact
Step 1: Should I care?
• Don’t read it the whole way through first time
• Make notes as you read the paper
• After the first read-through, try to summarize the
paper in one or two sentences.
What is this about?
What did they actually do?
– Usually just title and abstract
Have a go with your paper
• Question 1:
– Why was the study done,
– What research question were the authors addressing?
Population
Intervention
Comparator
Outcome
(Time)
Don’t worry about results just yet
Don’t worry about critical appraisal just yet
Step 2: SHOULD I BELIEVE THE
FINDINGS? (Critical Appraisal)
• Be suspicious
• Read Critically
• Not all that is claimed may be true
Step 2: SHOULD I BELIEVE THE
FINDINGS?
3 things that contribute to a result?
Results
• Three reasons for getting the results found:
1. Bias
 Critical appraisal helps here
2. Chance
 Stats helps here
3. True result (or placebo) (but is it clinically
meaningful?)
 Experience/clinical knowledge helps here
Results
• Three reasons for getting the results found:
1. Bias
 Critical appraisal helps here
2. Chance
 Stats helps here
3. True result (or placebo) (but is it clinically
meaningful?)
 Experience/clinical knowledge & study
design helps here
If you are deciding whether a paper is worth
reading, you should do so on the design of the
methods section and not:
– the interest of the hypothesis
– the nature or potential impact of the results,
or
– the speculation in the discussion.
• Most papers come in the IMRaD format
Question 2: Correct study design?
•
•
•
•
•
Cohort?
Case Report?
Case control?
RCT?
Systematic review?
Have a go with your paper
• Question 1:
– Why was the study done,
– What research question were the authors addressing?
• Question 2: What type of study was done?
– Is it the correct one for that question?
– First, decide whether the paper describes a primary study,
which reports research first hand, or a secondary one,
which attempts to summarise and draw conclusions from
primary studies.
Don’t worry about results just yet
Use the provided document to help you
Questions to do with drug treatment or other medical
interventions should be addressed by double blind, randomised
controlled trials
Questions about prognosis require longitudinal cohort studies,
and those about causation require either cohort or case control
studies
Case reports, though methodologically weak, can be produced
rapidly and have a place in alerting practitioners to adverse
drug reactions
Was the study design appropriate to the research?
Rapid critical appraisal using an
appraisal tool
• Different study designs call for different steps
to reduce systematic bias
Use the tool provided to help you look for possible sources of
bias
Most of the information you will need will be in the methods
section
• Smith, Jill P Pell
Results
• Three reasons for getting the results found:
1. Bias
 Critical appraisal helps here
2. Chance
 Stats helps here
3. True result (or placebo) (but is it clinically
meaningful?)
 Experience/clinical knowledge helps here
Your turn
• Take a look at the results of your paper
• Write down the main findings and note the
statistics used (p value, Confidence intervals)
Messerli (NEJM 2012)
• What was the research question?
• What study design was it?
Correlation vs causation
A correlation between X and Y does not prove
causation
• but indicates that either X influences Y,
• Y influences X, or
• X and Y are influenced by a common underlying
mechanism.
Consider:
• Sleeping in shoes is associated with morning
headaches
• Bigger foot size correlates with better reading
ability in children
• Sitting at the front of lecture theatres is
associated with better marks in assessment
RCT - Swayden et al (Patient Education
& Counseling 2012)
• What was the research question?
• What study design was it?
RCT - Swayden et al (Patient Education
& Counseling 2012)
• Difference in perception of time spent P = 0.01
What does a p value mean?
What does a p value mean?
• P values are a statement about statistical significance.
• Hypothesis tests (also known as significance tests) and P
values, are used to assess the strength of the evidence
against the null hypothesis that there is no true association
in the population that the sample was drawn.
• A P value, is the probability of getting a difference
between the two groups, at least as large, than that was
observed, if the null hypothesis was true. This value is the
P value.
• Thus, if the probability of getting such a difference is very,
very small, then we can say because this is so unlikely, we
can reject the null hypothesis, and accept our hypothesis.
• Korownyk et al (BMJ 2014)
• What was the research question?
• What study design was it?
• What were the results?
Results from Korownyk et al
• “We could find at least a case study or better evidence
to support 54% (95% confidence interval 47% to 62%)
of the 160 recommendations (80 from each show).
• For recommendations in The Dr Oz Show, evidence
supported 46%, contradicted 15%, and was not found
for 39%.
• For recommendations in The Doctors, evidence
supported 63%, contradicted 14%, and was not found
for 24%.
• Believable or somewhat believable evidence supported
33% of the recommendations on The Dr Oz Show and
53% on The Doctors.”
• Korownyk et al (BMJ 2014)
• What does a confidence interval mean?
• Wide CI vs narrow CI
Results
• Three reasons for getting the results found:
1. Chance
– Stats helps here
2. Bias
– Critical appraisal helps here
3. True result
– But is it clinically meaningful?
– Experience/clinical knowledge helps here
RCT
• Furyk et al (MJA 2009)
• We know the p value.
• What is the clinical significance of the results?
RCT
• Furyk et al (MJA 2009)
• We know the p value.
• What is the clinical significance of the results?
• Statistical difference is not the same as Clinical
Difference
Evidenced Based Practice
Patient
values
Best
Research
evidence
Clinical
experience
Remember
Thankyou
• Testing treatments.org (great book)
• www.jameslindlibrary.org
• https://www.youtube.com/watch?v=z1b2hFzX
srU
• http://www.bmj.com/about-bmj/resourcesreaders/publications/how-read-paper
– Trisha Greenhalgh BMJ Journal articles
• http://www.youtube.com/watch?NR=1&v=XvnUojPCft
k&feature=endscreen
– Short you tube video on introduction to this topic
• http://www.cebm.net/index.aspx?o=1157
– Appraisal tools, statistics calulators and more
• http://www.cebm.net/index.aspx?o=2320
– Richard Lehman’s Journal Review
– Light hearted GP perspective
Arroll et al (BMJ 2014)
• What study type?
• Who were the participants?
Arroll et al (BMJ 2014)
• What study type?
• Who were the participants?
• What were the results?
Arroll et al
• “Using the hazard ratio, on any one day the
gossipy magazines disappeared 14.51 times
(95% confidence interval 6.69 to 33.32) faster
than the non-gossipy ones”
• What does this mean?
Hazard Ratio
• The hazard ratio equals a weighted relative
risk over the entire duration of a study and is
derived from a time-to-event curve or
Kaplan-Meier curve. This curve describes the
status of both patient groups at different time
points after a defined starting point
Hazard Ratio
• Hazard Ratio is a ratio of hazards
HR = Hazard in treatment arm/hazard in control arm
• A Hazard = instantaneous event rate
= probability that an individual at time t has an event at that time
• HR = 1 (event rates are the same in both arms)
• HR = 2 (at any time twice as many patients in treatment group are
having an event proportionally to the comparator group
• HR = 0.5 (half as many….)
• A hazard ratio of 14 means the a magazine in
the gossipy group, is has 14x the odds of
reaching the outcome (disappearance)
compared to a non-gossipy one
Illustrated
For interest..
“Gossipy magazines had a significantly higher
mortality rate than the non-gossipy ones
(P<0.001), whereas age had no significant
impact on survival (P=0.41).”
“Magazines that disappeared were also
significantly cheaper than those that remained.”
Hazard Ratio of 2
A hazard ratio of 2 does means that treatment
will cause the patient to heal faster, but in a very
specific sense.
In the context of hazard ratio, “fast” means that
a treated patient who has not yet healed by a
certain time has twice the chance of being
healed at the next point in time compared to
someone in the control group.
The hazard ratio is equivalent to the odds that an
individual in the group with the higher hazard reaches the
endpoint first.
In a clinical trial examining time to disease resolution, it
represents the odds that a treated patient will resolve
symptoms before a control patient
A hazard ratio of 14 means the a magazine in the
gossipy group, is has 14x the odds of reaching the
outcome (disappearance) compared to a non-gossipy
one