Download Materials and methods

Corso di clinical writing
What to expect today?
Core modules
•
Introduction
•
General principles
•
Specific techniques
•
Title/Abstract drafting
•
Finding out relevant literature, and Introduction
drafting
•
Nuts & bolts of statistics and Methods drafting
•
Practical session 1 – Appraisal of a published article
Ultimate goal of research: appraisal of causation
Methods of inquiry
Statistical inquiry may be…
Descriptive
(to summarize or describe an observation)
or
Inferential
(to use the observations to make estimates or predictions)
Descriptive statistics
100
100
AVERAGE
Inferential statistics
If I become a scaffolder, how likely
I am to eat well every day?
P
values
Confidence
Intervals
Samples and populations
This is a sample
Samples and populations
And this is its
universal population
Samples and populations
This is another sample
Samples and populations
And this might be its
universal population
Samples and populations
But what if THIS is its
universal population?
Samples and populations
Any inference thus
depend on our confidence
in its likelihood
Accuracy and precision
true value
measurement
spread
Accuracy measures the distance from the true value
Precision measures the spead in the measurements
Accuracy and precision
Random and systematic errors
Thus
Precision expresses the extent of
RANDOM ERROR
Accuracy expresses the extent of
SYSTEMATIC ERROR (ie bias)
Bias
Bias is a systematic DEVIATION from
the TRUTH
-in itself it cannot be ever recognized
-there is a need for external gold
standard and/or permanent
surveillance
An incomplete list of bias
· Selection bias
· Information bias
· Confounders
· Observation bias
· Investigator’s bias (enthusiasm bias)
· Patient’s background bias
· Distribution of pathological changes bias
· Selection bias
· Small sample size bias
· Reporting bias
· Referral bias
· Variation bias
· Recall bias
· Statistical bias
· Selection bias
· Confounding
· Intervention bias
· Measurement or information
· Interpretation bias
· Publication bias
· Subject selection/sampling bias
Simplest classification:
1. Selection bias
2. Information bias
Sackett, J Chronic Dis 1979
Validity
Internal validity entails both PRECISION and
ACCURACY (ie does a study provide a truthful
answer to the research question?)
External validity expresses the extent to which
the results can be applied to other contexts and
settings. It corresponds to the distinction
between SAMPLE and POPULATION)
Validity
Rothwell, Lancet 2005
Navigating through variables
Statistical variables
Variables
CATEGORY
nominal
QUANTITY
ordinal
discrete
continuous
counting
measuring
Device diameter
BMI
Blood pressure
Death: yes/no
Race
ordered
categories
SES
ranks
Categorical variables
Exp
Ctrl
Event
a
b
No event
c
d
Absolute risk reduction (ARR) = [ a / ( a + c ) ] - [ b / ( b + d ) ]
Relative risk (RR) = [ a / ( a + c ) ] / [ b / ( b + d ) ]
Relative risk reduction (RRR) = 1 - RR
Odds ratio (OR) = (a/c)/(b/d) = ( a * d ) / ( b * c )
Categorical variables
Absolute risk reduction (ARR)
25% (25/100) - 40% (40/100) = -15%
Relative risk (RR)
Laparoscopic surg
Open surg
Bleeding
25
40
No bleeding
75
60
100
100
25% (25/100) / 40% (40/100) = 0.62
(given an equivalence value of 1)
Relative risk reduction (RRR)
1 – 0.62 = 38%
Total
Odds ratio (OR)
33% (25/75) / 66% (40/60) = 0.5
(given an equivalence value of 1)
Mean (arithmetic)
Characteristics:
-summarises information well
-discards a lot of information
Assumptions:
-data are not skewed
x
x
N
– distorts the mean
– outliers make the mean very different
-Measured on measurement scale
– cannot find mean of a categorical
measure
‘average’ device diameter may be meaningless
Median
What is it?
– The one in the middle
– Place values in order
– Median is central
Definition:
– Equally distant from all other values
Used for:
– Ordinal data
– Skewed data / outliers
– E.g. …………………
Comparing measures of central tendency
Mean is usually best
– If it works
– Useful properties (with standard deviation [SD])
– But…
Group 1
Lesion diameter
(mm)
Mean
Median
17
19
19
17
18
18
18
Group 2
21
21
21
21
4
17.6
21
Comparing measures of central tendency
It also depends on the underlying distribution…
mean=median=mode
Frequency
Symmetric?
Value
Comparing measures of central tendency
It also depends on the underlying distribution…
Mean>Median>Mode
Asymmetric?
30
Mode
Median
Mean
Frequency
25
20
15
10
5
0
0
1
2
3
4
5
6
7
Number of clips implanted per patient
8
9
Measures of dispersion: examples
Range
99% Confidence Interval (CI)
50
– Top to bottom
– Not very useful
40
Interquartile range
75% CI
30
– Used with median
20
– ¼ way to ¾ way
Standard deviation (SD) 10
SD
– Used with mean
0
40
30
20
10
0
– Very useful
Standard deviation
Standard deviation (SD):
– approximates population σ
SD

as N increases
Advantages:
– with mean enables powerful synthesis
mean±1*SD 68% of data
mean±2*SD 95% of data (1.96)
mean±3*SD 99% of data (2.86)
Disadvantages:
– is based on normal assumptions
2
( x x )
N-1
Statistical variables
Variables
CATEGORY
nominal
QUANTITY
ordinal
discrete
continuous
counting
measuring
Device diameter
BMI
Blood pressure
Death: yes/no
Race
ordered
categories
SES
ranks
Comparisons
Variables
PAIRED
OR
REPEATED
MEASURES
eg
Repeated-measures ANOVA
Paired Student t test
UNPAIRED
OR
INDEPENDENT
MEASURES
eg
Unpaired Student t test
Chi square test
Statistical tests
Are data categorical or continuous?
Categorical data: compare
proportions in groups
Two or more groups,
compare by chisquare test
Non-normal data;
use Mann Whitney test
Continuous data: compare
means or medians in groups
How many groups?
Two groups;
normal data,
same spread?
Normal data;
use t test
More than two groups;
normal data?
Normal data;
use ANOVA
Non-normal
data; use
Kruskal Wallis
Testing normality assumptions
Rules of thumb
1. Referring to previous data or analyses
(eg landmark articles, large databases)
2. Inspection tables and graphs
(eg outliers, histograms)
3. Checking rough equality of mean, median,
mode
4. Performing ad hoc statistical tests
•
Shapiro-Wilks test
•
Kolmogodorov-Smirnov test
•
…
Alpha and type I error
Whenever I perform a test, there is thus a
risk of a FALSE POSITIVE result, ie
REJECTING A TRUE null hypothesis
This error is called type I, is measured as
alpha and its unit is the p value
The lower the p value, the lower the risk of
falling into a type I error (ie the HIGHER the
SPECIFICITY of the test)
Alpha and type I error
Type I error is
like a MIRAGE
Because I see something
that does NOT exist
Beta and type II error
Whenever I perform a test, there is also a
risk of a FALSE NEGATIVE result, ie NOT
REJECTING A FALSE null hypothesis
This error is called type II, is measured as
beta and its unit is a probability
The complementary of beta is called power
The lower the beta, the lower the risk of
missing a true difference (ie the HIGHER the
SENSITIVITY of the test)
Beta and type II error
Type II error is
like being BLIND
Because I do NOT see
something that exists
Summary of errors
Experimental study
H0 accepted H0 rejected
H0 true

Type I
error
H0 false
Type II
error

Truth
Inferential statistics
P values tell you whether there is a
DIFFERENCE and its DIRECTION
Confidence intervals tell you what is
the MAGNITUDE (or SIZE) of such difference
Power and sample size
Whenever designing a study or analyzing a dataset, it is
important to estimate the sample size or the power of
the comparison
SAMPLE SIZE
Setting a specific alpha and a specific beta, you
calculate the necessary sample size given the average
inter-group difference and its variation
POWER
Given a specific sample size and alpha, in light of the
calculated average inter-group difference and its
variation, you obtain an estimate of the power (ie 1-beta)
Questions?
Materials and methods
How was the problem studied?
Materials and methods
How was the problem studied?
The answer is in the Methods
Expanded IMRAD algorithm
Introduction
Background
Limitations of current evidence
Study hypothesis
Methods
Design
Patients
Procedures
Follow-up
End-points
Additional analyses
Statistical analysis
Results
Baseline and procedural data
Early outcomes
Mid-to-long term outcomes
Additional analyses
Discussion
Summary of study findings
Current research context
Implications of the present study
Avenues for further research
Limitations of the present study
Conclusions
Structured approach
• Study design
• Patients (selection)
• Procedures
• Follow-up
• Outcomes (ie end-points, definitions)
• Additional analyses (eg IVUS, QCA, CT)
• Statistical analysis
Materials and methods
• Describe what was done to answer the research
question
• Give full details of the methods
• Include a clear statement of study design
“The EXCITE study was a double-blind, randomized,
parallel design … designed to compare the efficacy and
safety of …”
• Include a sentence about IRB approval,
informed consent, or compliance with animal
welfare regulations
“The protocol was approved by the institutional review
board, and all patients gave informed consent …”
Materials and methods
• State the protocol/procedures. Repeat the
question.
“We tested the efficacy of xemilonercept administered
subcutaneously in a dose of 30 mg, given three times
weekly for up to 6 months.”
“There were 2 primary endpoints. The first was eventfree survival at 182 days, with an event defined as…”
• Describe materials/methods or subjects
adequately
• Write in a logical order (usually chronological)
• Describe analytical methods
Materials and methods
• Use subheadings
• Do not include results in Methods
• Include appropriate figures and tables
• Write in past tense
• Use active voice whenever possible
• Place details in parentheses
– BMI decreased 10% (from 32.6 to 29.4,
p=0.027)
Materials and methods
• Use a figure for a complex design
• Cite references for published methods
• Describe others fully
• Discuss learning curve implications
• Enable the reader to a comprehensive
appraisal of selection, performance,
adjudication, and attrition bias
Materials and methods
• Briefly address questions you can
anticipate from the reader, eg
justify/clarify the design of your study or
specific adjustments in your procedural
protocol:
“Bail-out open surgical resection was
envisaged whenever laparoscopic surgical
resection proved…”
Materials and methods
• Treat limitations of this study in a matterof-fact way:
"These studies were performed as part of a
routine clinical assessment, so that no attempt
was made to ensure either fasting of the patient
or performance of the test at a particular time of
day."
Design subsection
• State clearly the design of the study
• Was it retrospective or prospective?
• Was it a registry or controlled study
• Did you randomly allocated patients?
• Did you follow a protocol (may add figure!)?
You can also include here
details of IRB approval
Retrospective controlled study
Prospective clinical study
Study protocol
Nelson et al,
NEJM 2004
Patient subsection
• State clearly how you selected
patients
• Specific inclusion criteria?
• Specific exclusion criteria?
You can include here details
of written informed consent
Patient data
Ganio et al,
BJS 2001
Procedure subsection
• State clearly how you performed the procedure
• Any novel approaches or devices?
• May include lesion appraisal subsection
• Can contain info on follow-up means
• Complete with details on concomitant or postintervention medications
You can include here pictures
detailing what you did/use
Procedural details
Technical details
Miccoli et al, AOHNS 2006
Outcome subsection
• State clearly which outcomes you adjudicated
• Define each of the most pertinent outcomes
thoroughly
• Define the timing of follow-up
• Make sure you use validated definitions or
classifications (if available, otherwise you are
in trouble!)
You can include here quality of life data
Outcomes & definitions
COLOR,
Lancet Oncol 2005
Additional analyses subsection
• Focuses on additional analyses that may be
pertinent to the study
– Nuclear scans
– Pathologic classifications
– Radiographic imaging
– Ultrasound scans
• Quote thoroughly for established methods
• Define explicitly terms and ways to compute
secondary variables
Statistics subsection
• Explain how you handled and reported categorical
and continuous variables
• Explain how you tested for significance at both
univariate and multivariate analysis
• Define tails and threshold p value
• State width of confidence intervals
• Provide sample size computation
• Spell out which software package was used
Quote extensively and be ready to defend
yourself if you use sophisticated analytic tools
Statistics subsection
COLOR,
Lancet Oncol 2005
Categorical variables
• Categorical variables are probably the most
important ones provided by a clinical study, as hard
clinical end-points are always expressed so
•
Specifically, focus on:
•
Choose a few statistics, and used them consistently
•
Provide confidence intervals
•
May also provide number needed
Continuous variables
• Continuous variables are important for the
appraisal of baseline/procedural characteristics
(eg waist-hip ratio), or additional analyses (eg
length of hospital stay)
• Focus on these points:
•
Provide mean and standard deviation
•
Or median (interquartile range) if non-Gaussian
•
May check for normality assumptions
P shows direction of difference
Nelson et al, NEJM 2004
Confidence intervals (CI)
show size of difference
Nelson et al, NEJM 2004
Other examples
Manouras et al, AJS 2008
Questions?
Take home messages
The most important points to remember
when writing the Methods section are:
Take home messages
The most important points to remember
when writing the Methods section are:
1. State exactly what you have done, no
more than that
2. Concentrate on the primary aim of the
study, not on the ancillary goals
3. Ensure reproducibility
And now article appraisal…