Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
PharmaSUG2010 - Paper IB11 SAS Programmer’s check list-Quick checks to be done before the statistical reports go off the SAS Programmer’s table. Thomas T. Joseph, Independent Consultant Babruvahan Hottengada, Independent Consultant ABSTRACT “The statistical output looks absurd” is not the coolest thing to hear. If the data is clean and if the output looks absurd, then it is a programming issue because SAS/STAT procedures can do no wrong. Our experiences as programmers have encouraged us to write down this check list while doing statistical programming. The essence of this paper is to do primary checks to avoid programming errors while statistical reporting in clinical trials before QC and review by the statistical team. The few procedures and checks which we discuss below do not require a very sound mathematical or statistical knowledge, but instead can be done on your finger tips or using a calculator. The later part of the paper discusses the pitfalls that could occur during pooling of data for safety and efficacy reporting. INTRODUCTION Many programmers especially those with non-statistical backgrounds make the mistake of passing on the reports generated from SAS procedures without looking at the numbers generated in the output. This can result in correct information being displayed incorrectly, or incorrect information being displayed incorrectly. This paper will highlight common mistakes made while programming statistical reports and then discusses some quick checks after programming that can be done without even writing code to check the values. The focus of this paper is primarily SAS/STAT procedures generating LS means estimates, LS means difference, confidence intervals/limits, and odds ratio, and as well as checks on datasets prior to pooling. There are several methods used to generate LS means estimate, LS means difference, confidence intervals, and similarly for odds ratio, but in this paper we will be considering a single SAS/STAT procedure (GLM) for LS means, LS means difference, confidence intervals. Similarly, we will be using PROC TTEST to generate degrees of freedom and confidence intervals, and PROC FREQ to generate odds ratio statistic. The checks that will be discussed in this paper can be applied to other SAS/STAT procedures that generate the very same statistic. CHECKS ON LS MEANS ESTIMATE, CONFIDENCE INTERVALS, AND LS MEANS DIFFERENCE Least Squares Means can be defined as a linear combination (sum) of the estimated effects (means, etc) from a linear model. These means are based on the model used. LSMEANS statement in SAS procedures are sometimes used when a covariate(s) appears in the model such as in ANCOVA [1]. Proc GLM, PROC MIXED, PROC GENMOD are some of the SAS procedures that generate Least Square means. Confidence interval is another common statistic generated by SAS/STAT procedures. Confidence interval is defined as the interval in which a value of particular parameter falls in. A confidence interval with 95% means 95 percent of the time the value of the parameter can be found within that confidence interval. Below is an example of analysis of covariance done using the PROC GLM model where baseline score, treatment arm, and locations are the independent variables and mscore is the dependent variable. Variables ptid, trtarm, baseline, mscore, and location contain information on patient identifiers, treatment arms, baseline score, final score, and location respectively. Using SAS/ODS the results of the analysis is fed into datasets and finally the desired the datasets are merged to produce an output similar to Figure 1 below. The following is the data step and code of analysis done using PROC GLM, and output: data class; input ptid datalines; 1001 A 1.5 1002 A 2.6 1003 A 2.2 1004 A 2.4 1005 A 2.5 1006 A 2.1 1007 A 2.0 1001 B 2.1 1002 B 1.8 1003 B 1.9 1004 B 2.8 trtarm $ baseline mscore location $; 3.2 3.2 3.3 3.7 3.8 3.9 4.0 2.5 2.7 2.6 2.2 AZ AK KA KA KA KA AZ AK AK AZ AZ ; run; ods ods ods ods trace on; output GLM.LSMEANS.trtarm.mscore.LSMeanDiffCL=diffcl; output GLM.LSMEANS.trtarm.mscore.LSMeanCL=meancl; output GLM.LSMEANS.trtarm.mscore.LSMeans=lsmeans; proc glm data=class; format trtarm $15.; class trtarm location; Model Mscore=trtarm baseline location; lsmeans trtarm/ pdiff cl; quit; ods trace close; data diffcl; set diffcl; if i=1 then trtarm ='A'; run; data statistics; merge meancl(in=a keep=trtarm lowercl lsmean uppercl trtarm dependent effect) diffcl(in=b keep=effect dependent uppercl lowercl difference trtarm); by effect trtarm; run; proc print data=statistics; run; The errors related to confidence interval can easily be detected without writing a quality check code. Checking whether LS means estimate falls within the upper and lower confidence limits and checking whether the difference of LS means equals the difference could help in identifying the programming error. Figure 1 displays the output of the print procedure while Figure 2 displays the results of the PROC GLM used. Figure 1 The SAS System Effect Dependent trtarm LowerCL LSMean UpperCL Difference trtarm mscore A 0.283003 3.522879 1.592382 0.937692 trtarm mscore B 2.110482 2.585186 3.059891 The output shown in Figure 1 has an apparent error- LS mean estimate for each trtarm A does not fall within the upper and lower confidence limits and so the result is erroneous, which has happened as a result of not renaming the variables during merging to generate the statistics dataset.From Figure 1, Difference of the LS means =0.937692. Using the following equation, check for LS means difference of effect trtarm is possible: LS Means Difference for effect trtarm = LS Mean of A – LS Mean of B= 3.5228- 2.5851 = 0.9377. Here in this case, the difference of means by subtraction equals the values displayed in Figure 1 and 2. Erroneous results for difference of LS means could also arise from programming errors. Figure 2 The GLM Procedure Least Squares Means H0:LSMean1=LSMean2 trtarm mscore LSMEAN Pr > |t| A 3.52287879 0.0128 B 2.58518648 trtarm mscore LSMEAN 95% Confidence Limits A 3.522879 3.162115 3.883643 B 2.585186 2.110482 3.059891 Least Squares Means for Effect trtarm i j Difference Between Means 1 2 0.937692 95% Confidence Limits for LSMean(i)-LSMean(j) 0.283003 1.592382 So, checks that could be done on LS means, CI, LS means differences results are as follows: 1. The estimate of LS Means should lie between CI of LS means. 2. The difference of the estimate of LS Means is LS Means difference statistic. In Figure 2, the mscore LS Mean statistics and LS means difference have been highlighted. 3. Keep an eye for the variable names especially when output delivery system is used to generate datasets. CHECKS FOR DEGREES OF FREEDOM, CONFIDENCE INTERVALS 1 sample t-test is usually done using PROC MEANS, PROC UNIVARIATE, and PROC TTEST. Some of the statistics of interest generated by these SAS/STAT procedures are degrees of freedom, confidence intervals .Degrees of Freedom is usually N-1, where N=number of observations. Confidence intervals can be calculated using , where N is the sample size, s is the standard deviation, t is the values at alpha confidence and N-1 sample size, and Y is the mean. It should be noted that with decreasing sample the confidence intervals tends to be wider. The following dataset and code is used to show the variation of the confidence intervals with respect to the number of observations or degrees of freedom: data class; input ptid cycle value; datalines ; 1001 1 10.5 1002 1 20.2 1003 1 20.3 1004 1 20.5 1005 1 30 1006 1 30.5 1007 1 40 1008 1 45 1009 1 43 1001 2 20.5 1002 2 30 ; run; proc ttest data=class; by cycle; var value; run; Figure 3 The SAS System The TTEST Procedure cycle=1 Statistics Variable N Lower CL Mean Value 9 19.698 Mean 28.88 9 Upper CL Mean Lower CL Std Dev Std Dev Upper CL Std Dev 38.08 8.0765 11.957 22.907 Std Err Minimum Maximum 10.5 45 3.985 7 T-Tests Variable D F t Value Pr > |t| value 8 7.25 <.0001 cycle=2 Statistics Variable N Lower CL Mean Value 2 -35.1 Mean Upper CL Mean Lower CL Std Dev Std Dev Upper CL Std Dev Std Err Minimum Maximum 25.25 85.604 2.997 6.7175 214.36 4.75 20.5 30 T-Tests Variable D F t Value Pr > |t| value 1 5.32 0.1184 Degrees of freedom (at cycle 1) = Number of observations-1=9-1 =8 (highlighted in Figure 3). If you notice closely, you will see that with decreasing degrees of freedom or decreasing number of observations the width of the confidence intervals increases but this is not always the case, as it is data driven. An easy check for the mean statistic and confidence interval/limits while generating the final report is to look whether the mean statistic falls within the confidence interval/limits. CHECKS FOR ODDS RATIO Odds ratio is another statistic widely used to do comparison between the treatment arms in the pharmaceutical industry. As the name suggests, odds ratio is the ratio of the odds of an event happening in a particular group to the odds of it happening in a different group or simply defined as the ratio of odds [2]. Total N Event Treatment A| Treatment B Na | Nb na | nb Odds ratio= Odds of an event happening in treatment A/Odds of the same event happening in treatment B = (na/(Na-na))/(nb/(Nb-nb)) Below is an example where odds ratio output has been generated. Here in this example, the odds ratio of an event happening in a particular treatment with sorting order variable (whose variable name is s_order) is generated. Also the odds ratio of another event (whose variable name is secevent) happening is also calculated. The purpose of this example is to show how the odds ratio can change with changing values of the events. data class; input ptid s_order value $ event secevent; datalines; 1001 1 A 1 1 1002 1 A 1 1 1003 1 A 1 1 1004 1 B 0 2 1005 1 B 0 2 1006 1 A 1 1 1007 1 A 1 1 1001 2 B 0 2 1002 2 B 0 2 1017 2 A 1 1 1088 2 A 1 1 1087 2 A 1 1 ; run; proc freq data=class; tables s_order*event/ norow nocol nopercent run; alpha=0.05 cmh ; Figure 4 The FREQ Procedure Frequency Table of s_order by event event 0 1 Total 1 2 5 7 2 2 3 5 4 8 12 s_order Total Summary Statistics for s_order by event Cochran-Mantel-Haenszel Statistics (Based on Table Scores) Statistic Alternative Hypothesis DF Value Prob 1 Nonzero Correlation 1 0.1571 0.6918 2 Row Mean Scores Differ 1 0.1571 0.6918 3 General Association 1 0.1571 0.6918 Estimates of the Common Relative Risk (Row1/Row2) Type of Study Method Value 95% Confidence Limits Case-Control Mantel-Haenszel 0.6000 0.0530 6.7948 (Odds Ratio) Logit 0.6000 0.0530 6.7948 Cohort Mantel-Haenszel 0.7143 0.1458 3.4986 (Col1 Risk) Logit 0.7143 0.1458 3.4986 Cohort Mantel-Haenszel 1.1905 0.5061 2.8004 (Col2 Risk) Logit 1.1905 0.5061 2.8004 Total Sample Size = 12 In Figure 4, the odds ratio generated is 0.6 (highlighted in Figure 4), which is the ratio of the event ‘0’ happening in s_order 1 to event ‘0’ happening in s_order 2. Odds ratio generated in Figure 4 is as follows: (na/ (Na-na))/ (nb/ (Nb-nb)) = ((na/Na)/ (1-(na/Na)) / ((nb/Nb)/ (1-(nb/Nb)) = ((2/7)/ (5/7)) / ((2/5)/ (3/5)) = 0.6 Since the values of variable secevent are slightly different from variable event in the dataset class, the outputs for odds ratio have drastically changed. The following code generates the odds ratio of event ‘1’ happening in group (s_order 1) to event ‘1’ happening in (s_order 2) and tells how significantly the values of odds ratio have changed. proc freq data=class; tables s_order*secevent/ norow nocol nopercent alpha=0.05 cmh ; run; Figure 5 The FREQ Procedure Frequency Table of s_order by secevent secevent 1 2 Total 1 5 2 7 2 3 2 5 8 4 12 s_order Total Summary Statistics for s_order by secevent Cochran-Mantel-Haenszel Statistics (Based on Table Scores) Statistic Alternative Hypothesis DF Value Prob 1 Nonzero Correlation 1 0.1571 0.6918 2 Row Mean Scores Differ 1 0.1571 0.6918 3 General Association 1 0.1571 0.6918 Estimates of the Common Relative Risk (Row1/Row2) Type of Study Method Value 95% Confidence Limits Case-Control Mantel-Haenszel 1.6667 0.1472 18.8744 (Odds Ratio) Logit 1.6667 0.1472 18.8744 Cohort Mantel-Haenszel 1.1905 0.5061 2.8004 (Col1 Risk) Logit 1.1905 0.5061 2.8004 Cohort Mantel-Haenszel 0.7143 0.1458 3.4986 (Col2 Risk) Logit 0.7143 0.1458 3.4986 Total Sample Size = 12 In Figure 5, the odds ratio generated is 1.6667 (highlighted in Figure 5), which is the ratio of the event ‘1’ happening in s_order 1 to event ‘1’ happening in s_order 2. So, odds ratio from Figure 5 is as follows: (na/ (Na-na))/ (nb/ (Nb-nb)) = ((na/Na)/ (1-(na/Na)) / ((nb/Nb)/ (1-(nb/Nb)) = ((5/7)/ (2/7)) / ((3/5)/ (2/5)) = 2.5/1.5 =1.6667 The outputs generated in both cases are correct, but ignoring what output is generated and what is to be displayed could result in passing the report with the wrong information, and finally result in wrong inferences. So, we suggest it is a good idea to calculate manually and re-check what the values generated and displayed. CHECKS BEFORE POOLING DATASETS Special attention has to be given while pooling datasets because of different attributes, different dictionaries used, and different units of collecting information in different datasets. If proper attention is not paid, the pooled dataset produced can produce absurd results. Some of the checks we do before pooling datasets are as follows: 1. 2. 3. Check for consistency of attributes-lengths, formats, and types in datasets. Consistency of attributes could avoid erroneous results like having records with truncated information because of varying lengths, misrepresented information because of different formats. Standardized of data is strongly recommended before pooling the datasets. This process can produce results with lesser errors. Different versions of WHO Drug and MedDRA dictionaries can result in varying categorization of preferred text for adverse events, concomitant medications etc. So it is recommended to use the same or the latest version of the dictionaries. CONCLUSION The checks discussed in this paper are precautions to be kept in mind while programming. These checks can reduce the chances of producing erroneous results. Only a handful of statistics, checks related to those statistics, and precautions while pooling of datasets have been discussed in this paper. The topic in itself is very broad in nature and could be extended to other statistical procedures widely used in the pharmaceutical industry and also beyond the pharmaceutical industry. REFERENCES [1] http://www.uiweb.uidaho.edu/ag/statprog/sas/workshops/mixed/LSMEANS.pdf [2] Proc freq: It’s more than counts, Richard Severino, The Queen’s Medical Center, Honolulu, HI ACKNOWLEDGEMENTS I would like to thank Carol Mathews for reviewing this paper and providing excellent comments. CONTACT INFORMATION Thomas Joseph Independent Consultant New London, CT 06320 Email: [email protected] Babruvahan Hottengada Independent Consultant New London, CT 06320 Email: [email protected] SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies.