Download SAS Programmer's check list-Quick checks to be done before the statistical reports go off the SAS Programmer's table

PharmaSUG2010 - Paper IB11
SAS Programmer’s check list-Quick checks to be done before the
statistical reports go off the SAS Programmer’s table.
Thomas T. Joseph, Independent Consultant
Babruvahan Hottengada, Independent Consultant
ABSTRACT
“The statistical output looks absurd” is not the coolest thing to hear. If the data is clean and if the output looks absurd, then it is
a programming issue because SAS/STAT procedures can do no wrong. Our experiences as programmers have encouraged
us to write down this check list while doing statistical programming. The essence of this paper is to do primary checks to avoid
programming errors while statistical reporting in clinical trials before QC and review by the statistical team. The few procedures
and checks which we discuss below do not require a very sound mathematical or statistical knowledge, but instead can be
done on your finger tips or using a calculator. The later part of the paper discusses the pitfalls that could occur during pooling
of data for safety and efficacy reporting.
INTRODUCTION
Many programmers especially those with non-statistical backgrounds make the mistake of passing on the reports generated
from SAS procedures without looking at the numbers generated in the output. This can result in correct information being
displayed incorrectly, or incorrect information being displayed incorrectly. This paper will highlight common mistakes made
while programming statistical reports and then discusses some quick checks after programming that can be done without even
writing code to check the values.
The focus of this paper is primarily SAS/STAT procedures generating LS means estimates, LS means difference, confidence
intervals/limits, and odds ratio, and as well as checks on datasets prior to pooling. There are several methods used to
generate LS means estimate, LS means difference, confidence intervals, and similarly for odds ratio, but in this paper we will
be considering a single SAS/STAT procedure (GLM) for LS means, LS means difference, confidence intervals. Similarly, we
will be using PROC TTEST to generate degrees of freedom and confidence intervals, and PROC FREQ to generate odds ratio
statistic. The checks that will be discussed in this paper can be applied to other SAS/STAT procedures that generate the very
same statistic.
CHECKS ON LS MEANS ESTIMATE, CONFIDENCE INTERVALS, AND LS MEANS DIFFERENCE
Least Squares Means can be defined as a linear combination (sum) of the estimated effects (means, etc) from a linear model.
These means are based on the model used. LSMEANS statement in SAS procedures are sometimes used when a
covariate(s) appears in the model such as in ANCOVA [1]. Proc GLM, PROC MIXED, PROC GENMOD are some of the SAS
procedures that generate Least Square means. Confidence interval is another common statistic generated by SAS/STAT
procedures. Confidence interval is defined as the interval in which a value of particular parameter falls in. A confidence interval
with 95% means 95 percent of the time the value of the parameter can be found within that confidence interval.
Below is an example of analysis of covariance done using the PROC GLM model where baseline score, treatment arm, and
locations are the independent variables and mscore is the dependent variable. Variables ptid, trtarm, baseline, mscore, and
location contain information on patient identifiers, treatment arms, baseline score, final score, and location respectively. Using
SAS/ODS the results of the analysis is fed into datasets and finally the desired the datasets are merged to produce an output
similar to Figure 1 below.
The following is the data step and code of analysis done using PROC GLM, and output:
data class;
input ptid
datalines;
1001 A 1.5
1002 A 2.6
1003 A 2.2
1004 A 2.4
1005 A 2.5
1006 A 2.1
1007 A 2.0
1001 B 2.1
1002 B 1.8
1003 B 1.9
1004 B 2.8
trtarm $ baseline mscore location $;
3.2
3.2
3.3
3.7
3.8
3.9
4.0
2.5
2.7
2.6
2.2
AZ
AK
KA
KA
KA
KA
AZ
AK
AK
AZ
AZ
;
run;
ods
ods
ods
ods
trace on;
output GLM.LSMEANS.trtarm.mscore.LSMeanDiffCL=diffcl;
output GLM.LSMEANS.trtarm.mscore.LSMeanCL=meancl;
output GLM.LSMEANS.trtarm.mscore.LSMeans=lsmeans;
proc glm data=class;
format trtarm $15.;
class trtarm location;
Model Mscore=trtarm baseline location;
lsmeans trtarm/ pdiff cl;
quit;
ods trace close;
data diffcl;
set diffcl;
if i=1 then trtarm ='A';
run;
data statistics;
merge meancl(in=a keep=trtarm lowercl lsmean uppercl trtarm dependent effect)
diffcl(in=b keep=effect dependent uppercl lowercl difference trtarm);
by effect trtarm;
run;
proc print data=statistics;
run;
The errors related to confidence interval can easily be detected without writing a quality check code. Checking whether LS
means estimate falls within the upper and lower confidence limits and checking whether the difference of LS means equals the
difference could help in identifying the programming error. Figure 1 displays the output of the print procedure while Figure 2
displays the results of the PROC GLM used.
Figure 1
The SAS System
Effect
Dependent
trtarm
LowerCL
LSMean
UpperCL
Difference
trtarm
mscore
A
0.283003
3.522879
1.592382
0.937692
trtarm
mscore
B
2.110482
2.585186
3.059891
The output shown in Figure 1 has an apparent error- LS mean estimate for each trtarm A does not fall within the upper and
lower confidence limits and so the result is erroneous, which has happened as a result of not renaming the variables during
merging to generate the statistics dataset.From Figure 1, Difference of the LS means =0.937692.
Using the following equation, check for LS means difference of effect trtarm is possible:
LS Means Difference for effect trtarm = LS Mean of A – LS Mean of B= 3.5228- 2.5851 = 0.9377. Here in this case, the
difference of means by subtraction equals the values displayed in Figure 1 and 2. Erroneous results for difference of LS
means could also arise from programming errors.
Figure 2
The GLM Procedure
Least Squares Means
H0:LSMean1=LSMean2
trtarm
mscore LSMEAN
Pr > |t|
A
3.52287879
0.0128
B
2.58518648
trtarm
mscore LSMEAN
95% Confidence Limits
A
3.522879
3.162115
3.883643
B
2.585186
2.110482
3.059891
Least Squares Means for Effect trtarm
i
j
Difference Between
Means
1
2
0.937692
95% Confidence Limits for LSMean(i)-LSMean(j)
0.283003
1.592382
So, checks that could be done on LS means, CI, LS means differences results are as follows:
1. The estimate of LS Means should lie between CI of LS means.
2. The difference of the estimate of LS Means is LS Means difference statistic. In Figure 2, the mscore LS Mean
statistics and LS means difference have been highlighted.
3. Keep an eye for the variable names especially when output delivery system is used to generate datasets.
CHECKS FOR DEGREES OF FREEDOM, CONFIDENCE INTERVALS
1 sample t-test is usually done using PROC MEANS, PROC UNIVARIATE, and PROC TTEST. Some of the statistics of
interest generated by these SAS/STAT procedures are degrees of freedom, confidence intervals .Degrees of Freedom is
usually N-1, where N=number of observations. Confidence intervals can be calculated using
, where
N is the sample size, s is the standard deviation, t is the values at alpha confidence and N-1 sample size, and Y is the mean.
It should be noted that with decreasing sample the confidence intervals tends to be wider.
The following dataset and code is used to show the variation of the confidence intervals with respect to the number of
observations or degrees of freedom:
data class;
input ptid cycle value;
datalines ;
1001 1 10.5
1002 1 20.2
1003 1 20.3
1004 1 20.5
1005 1 30
1006 1 30.5
1007 1 40
1008 1 45
1009 1 43
1001 2 20.5
1002 2 30
;
run;
proc ttest data=class;
by cycle;
var value;
run;
Figure 3
The SAS System
The TTEST Procedure
cycle=1
Statistics
Variable
N
Lower CL
Mean
Value
9
19.698
Mean
28.88
9
Upper CL
Mean
Lower
CL
Std Dev
Std Dev
Upper
CL
Std Dev
38.08
8.0765
11.957
22.907
Std
Err
Minimum
Maximum
10.5
45
3.985
7
T-Tests
Variable
D
F
t Value
Pr > |t|
value
8
7.25
<.0001
cycle=2
Statistics
Variable
N
Lower CL
Mean
Value
2
-35.1
Mean
Upper CL
Mean
Lower
CL
Std Dev
Std Dev
Upper CL
Std Dev
Std
Err
Minimum
Maximum
25.25
85.604
2.997
6.7175
214.36
4.75
20.5
30
T-Tests
Variable
D
F
t Value
Pr > |t|
value
1
5.32
0.1184
Degrees of freedom (at cycle 1) = Number of observations-1=9-1 =8 (highlighted in Figure 3). If you notice closely, you will see
that with decreasing degrees of freedom or decreasing number of observations the width of the confidence intervals increases
but this is not always the case, as it is data driven. An easy check for the mean statistic and confidence interval/limits while
generating the final report is to look whether the mean statistic falls within the confidence interval/limits.
CHECKS FOR ODDS RATIO
Odds ratio is another statistic widely used to do comparison between the treatment arms in the pharmaceutical industry. As
the name suggests, odds ratio is the ratio of the odds of an event happening in a particular group to the odds of it happening in
a different group or simply defined as the ratio of odds [2].
Total N
Event
Treatment A| Treatment B
Na | Nb
na | nb
Odds ratio= Odds of an event happening in treatment A/Odds of the same event happening in treatment B
= (na/(Na-na))/(nb/(Nb-nb))
Below is an example where odds ratio output has been generated. Here in this example, the odds ratio of an event happening
in a particular treatment with sorting order variable (whose variable name is s_order) is generated. Also the odds ratio of
another event (whose variable name is secevent) happening is also calculated. The purpose of this example is to show how
the odds ratio can change with changing values of the events.
data class;
input ptid s_order value $ event secevent;
datalines;
1001 1 A 1 1
1002 1 A 1 1
1003 1 A 1 1
1004 1 B 0 2
1005 1 B 0 2
1006 1 A 1 1
1007 1 A 1 1
1001 2 B 0 2
1002 2 B 0 2
1017 2 A 1 1
1088 2 A 1 1
1087 2 A 1 1
;
run;
proc freq data=class;
tables s_order*event/ norow nocol nopercent
run;
alpha=0.05 cmh ;
Figure 4
The FREQ Procedure
Frequency
Table of s_order by event
event
0
1
Total
1
2
5
7
2
2
3
5
4
8
12
s_order
Total
Summary Statistics for s_order by event
Cochran-Mantel-Haenszel Statistics (Based on Table Scores)
Statistic
Alternative Hypothesis
DF
Value
Prob
1
Nonzero Correlation
1
0.1571
0.6918
2
Row Mean Scores Differ
1
0.1571
0.6918
3
General Association
1
0.1571
0.6918
Estimates of the Common Relative Risk (Row1/Row2)
Type of Study
Method
Value
95% Confidence Limits
Case-Control
Mantel-Haenszel
0.6000
0.0530
6.7948
(Odds Ratio)
Logit
0.6000
0.0530
6.7948
Cohort
Mantel-Haenszel
0.7143
0.1458
3.4986
(Col1 Risk)
Logit
0.7143
0.1458
3.4986
Cohort
Mantel-Haenszel
1.1905
0.5061
2.8004
(Col2 Risk)
Logit
1.1905
0.5061
2.8004
Total Sample Size = 12
In Figure 4, the odds ratio generated is 0.6 (highlighted in Figure 4), which is the ratio of the event ‘0’ happening in s_order 1
to event ‘0’ happening in s_order 2. Odds ratio generated in Figure 4 is as follows:
(na/ (Na-na))/ (nb/ (Nb-nb)) = ((na/Na)/ (1-(na/Na)) / ((nb/Nb)/ (1-(nb/Nb)) = ((2/7)/ (5/7)) / ((2/5)/ (3/5)) = 0.6
Since the values of variable secevent are slightly different from variable event in the dataset class, the outputs for odds ratio
have drastically changed. The following code generates the odds ratio of event ‘1’ happening in group (s_order 1) to event ‘1’
happening in (s_order 2) and tells how significantly the values of odds ratio have changed.
proc freq data=class;
tables s_order*secevent/ norow nocol nopercent
alpha=0.05 cmh ;
run;
Figure 5
The FREQ Procedure
Frequency
Table of s_order by secevent
secevent
1
2
Total
1
5
2
7
2
3
2
5
8
4
12
s_order
Total
Summary Statistics for s_order by secevent
Cochran-Mantel-Haenszel Statistics (Based on Table Scores)
Statistic
Alternative Hypothesis
DF
Value
Prob
1
Nonzero Correlation
1
0.1571
0.6918
2
Row Mean Scores Differ
1
0.1571
0.6918
3
General Association
1
0.1571
0.6918
Estimates of the Common Relative Risk (Row1/Row2)
Type of Study
Method
Value
95% Confidence Limits
Case-Control
Mantel-Haenszel
1.6667
0.1472
18.8744
(Odds Ratio)
Logit
1.6667
0.1472
18.8744
Cohort
Mantel-Haenszel
1.1905
0.5061
2.8004
(Col1 Risk)
Logit
1.1905
0.5061
2.8004
Cohort
Mantel-Haenszel
0.7143
0.1458
3.4986
(Col2 Risk)
Logit
0.7143
0.1458
3.4986
Total Sample Size = 12
In Figure 5, the odds ratio generated is 1.6667 (highlighted in Figure 5), which is the ratio of the event ‘1’ happening in s_order
1 to event ‘1’ happening in s_order 2. So, odds ratio from Figure 5 is as follows:
(na/ (Na-na))/ (nb/ (Nb-nb)) = ((na/Na)/ (1-(na/Na)) / ((nb/Nb)/ (1-(nb/Nb)) = ((5/7)/ (2/7)) / ((3/5)/ (2/5)) = 2.5/1.5 =1.6667
The outputs generated in both cases are correct, but ignoring what output is generated and what is to be displayed could
result in passing the report with the wrong information, and finally result in wrong inferences. So, we suggest it is a good idea
to calculate manually and re-check what the values generated and displayed.
CHECKS BEFORE POOLING DATASETS
Special attention has to be given while pooling datasets because of different attributes, different dictionaries used, and
different units of collecting information in different datasets. If proper attention is not paid, the pooled dataset produced can
produce absurd results. Some of the checks we do before pooling datasets are as follows:
1.
2.
3.
Check for consistency of attributes-lengths, formats, and types in datasets. Consistency of attributes could avoid
erroneous results like having records with truncated information because of varying lengths, misrepresented
information because of different formats.
Standardized of data is strongly recommended before pooling the datasets. This process can produce results with
lesser errors.
Different versions of WHO Drug and MedDRA dictionaries can result in varying categorization of preferred text for
adverse events, concomitant medications etc. So it is recommended to use the same or the latest version of the
dictionaries.
CONCLUSION
The checks discussed in this paper are precautions to be kept in mind while programming. These checks can reduce the
chances of producing erroneous results. Only a handful of statistics, checks related to those statistics, and precautions while
pooling of datasets have been discussed in this paper. The topic in itself is very broad in nature and could be extended to
other statistical procedures widely used in the pharmaceutical industry and also beyond the pharmaceutical industry.
REFERENCES
[1] http://www.uiweb.uidaho.edu/ag/statprog/sas/workshops/mixed/LSMEANS.pdf
[2] Proc freq: It’s more than counts, Richard Severino, The Queen’s Medical Center, Honolulu, HI
ACKNOWLEDGEMENTS
I would like to thank Carol Mathews for reviewing this paper and providing excellent comments.
CONTACT INFORMATION
Thomas Joseph
Independent Consultant
New London, CT 06320
Email: [email protected]
Babruvahan Hottengada
Independent Consultant
New London, CT 06320
Email: [email protected]
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in
the USA and other countries. ® indicates USA registration.
Other brand and product names are trademarks of their respective companies.