Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Document related concepts

Transcript

Advanced Statistics for Interventional Cardiologists What you will learn • • • • • • • • • • • • • • • Introduction Basics of multivariable statistical modeling Advanced linear regression methods Hands-on session: linear regression Bayesian methods Logistic regression and generalized linear model Resampling methods Meta-analysis Hands-on session: logistic regression and meta-analysis Multifactor analysis of variance Cox proportional hazards analysis Hands-on session: Cox proportional hazard analysis Propensity analysis Most popular statistical packages Conclusions and take home messages 1st day 2nd day What you will learn • Multiple Linear Regression – Basic concepts • • • • • – – – – – – – – Some examples Linear regression model Estimation and testing the regression coefficients Testing and evaluating the regression model Predictions Multiple regression models The model building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative Predictor variables Practical examples Multiple linear regression Example from cardiology How can I predict the impact of balloon dilation pressure on post-procedure minimum lumen diameter (MLD), taking concomitantly into account diabetes status and ACC/AHA lesion type? In other words, how can I predict the impact of a given variable (aka independent) on another continuous variable (aka dependent), taking concomitantly into account other variables ? Multiple linear regression Example from cardiology Time to restenosis (days) (mm) lumen diameter Minimum 400 350 300 250 200 150 100 50 0 0 10 20 30 40 Dilation pressureLesion duringLenght stenting (ATM) 50 60 Multiple linear regression Example from cardiology Briguori et al, Eur Heart J 2002 Multiple linear regression Example from cardiology Briguori et al, Eur Heart J 2002 Multiple linear regression Example from cardiology Mauri et al, Circulation 2005 Multiple linear regression Example from cardiology Mauri et al, Circulation 2005 Multiple linear regression Fitness demo example Aerobic fitness can be evaluated using a special test that measures the oxygen uptake of a person running on a treadmill for a prescribed distance. However, it would be more economical to evaluate fitness with a formula that predicts oxygen uptake using simple measurements such as running time and pulse. The table on the next slide shows the partial listing of the measurements for 31 subjects. Objective Find the regression equation that allows us to make the most reliable predictions of O2 uptake. Multiple Regression Fitness Data Subject • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Donna Gracie Luanne Mimi Chris Allen Nancy Patty Suzanne Teresa Bob Harriett Jane Harold Sammy Buffy Trent Jackie Ralph Jack Annie Kate Carl Don Effie George Iris Mark Steve Vaughn William Sex F F F F M M F F F F M F F M M F M F M M F F M M F M F M M M M Age 42 38 43 50 49 38 49 52 57 51 40 49 44 48 54 52 52 47 43 51 51 45 54 44 48 47 40 57 54 44 45 Weight 68,15 81,87 85,84 70,87 81,42 89,02 76,32 76,32 59,08 77,91 75,07 73,37 73,03 91,63 83,12 73,71 82,78 79,15 81,19 69,63 67,25 66,45 79,38 89,47 61,24 77,45 75,98 73,37 91,63 81,42 87,66 Oxy 59,57 60,06 54,30 54,63 49,16 49,87 48,67 45,44 50,55 46,67 45,31 50,39 50,54 46,77 51,85 45,79 47,47 47,27 49,09 40,84 45,12 44,75 46,08 44,61 47,92 44,81 45,68 39,41 39,20 39,44 37,39 RunTime 8,17 8,63 8,65 8,92 8,95 9,22 9,40 9,63 9,93 10,00 10,07 10,08 10,13 10,25 10,33 10,47 10,50 10,60 10,85 10,95 11,08 11,12 11,17 11,37 11,50 11,63 11,95 12,63 12,88 13,08 14,03 RunPulse 166 170 156 146 180 178 186 164 148 162 185 168 168 162 166 186 170 162 162 168 172 176 156 178 170 176 176 174 168 174 186 RstPulse 40 48 45 48 44 55 56 48 49 48 62 67 45 48 50 59 53 47 64 57 48 51 62 62 52 58 70 58 44 63 56 MaxPulse 172 186 168 155 185 180 188 166 155 168 185 168 168 164 170 188 172 164 170 172 172 176 165 182 176 176 180 176 172 176 192 Multiple linear regression • Simple linear regression is a statistical model to predict the value of one continuous variable Y (dependent, response) from another continuous variable X (independent, predictor, covariate, prognostic factor). • Multiple linear regression is a natural extension of the simple linear regression model – We use it to investigate the effect on the response variable of several predictor variables, simultaneously – It is a hypothetical model of the relationship between several independent variables and a response variable. • Let’s start by reviewing the concepts of the simple linear regression model. Simple linear regression The theoretical model Y = β0 + β1 X + ε Independent variable Distribution of the dependent variable Mean of the distribution of values of the dependent variable Regression line Simple linear regression The estimated model Yestimated = b0 + b1 X Yˆi Yestimated Y Re siduals : ei Yi Yˆi Yi b1 : slope unit b0: intercept X (independent) Linear Regression An estimation problem • Estimate the model parameters β0 and β1 as good as possible. • Find the ‘best-fitting’ line (Y = b0 + b1.X) through the measured coördinates. • How do we find this line? Minimize the sum of squared differences (least squares) between y and yestimated • Parameter Estimators b1 = (n Σ Xi Yi - Σ Xi Σ Yi) / ( n Σ Xi2 – (Σ Xi)2 ) b0 = mean Y – (b1 . mean X) Linear regression Assumptions Least square assumptions Linear relation between X and Y : E(εi) = 0 for all i Homoscedasticity (constant variance): Var(εi) = σ2 for all i Uncorrelated residuals : E(εi εj) = σ2 for all i ≠ j Significance tests assumptions Residuals are normally distributed : εi ≈ N ( 0, σ2 ) for all i Linear Regression Testing parameter significance • Testing significance of the regression parameters allows us to evaluate if there is an effect of the independent variable on the dependent variable. • If the slope β1 is significantly different from zero than we will conclude that the independent variable has a significant effect on the dependent variable. • Is the slope β1 significantly different from 0 ? – No : the value of x will not improve the prediction of y over the ordinary mean, – Yes : knowledge of the x-values will significantly improve the predictions • We can test if the slope is significantly different from 0 in two ways – Using a classical t-test – Construction of a confidence interval Linear Regression Testing parameter significance • The t- test is based on a function of the slope estimate b1 which has the t-distribution when the null hypothesis of ‘zero slope’ is true : tdf = b1/SE(b1) Decision rule: reject H0 if t t / 2,n2 • Testing with confidence intervals : b1 - tn-2,α/2 SEb < β1 < b1 - tn-2,α/2 SEb Decision rule: reject H0 if 0 is not in the confidence region Simple Linear Regression Growth example Examine how weight to height ratio changes as kids grow up. Measurements were taken from 72 children between birth and 70 months. What are your conclusions looking at the scatterplot ? Linear Regression Growth Example Linear Fit ratio = 0,66562 + 0,00528 age Summary of Fit RSquare 0,822535 RSquare Adj 0,819999 Root Mean Square Error 0,051653 Mean of Response 0,855556 Observ ations (or Sum Wgts) 72 Analy sis of Variance Source Model DF Sum of Squares Mean Square F Ratio 1 0,8656172 0,865617 324,4433 Error 70 0,1867605 0,002668 Prob>F C Total 71 1,0523778 <,0001 Parameter Estimates Term Std Error t Ratio Prob>|t| Intercept 0,6656231 Estimate 0,012176 54,67 <,0001 Lower 95% 0,6413397 Upper 95% 0,6899065 age 0,0052759 0,000293 18,01 <,0001 0,0046917 0,0058601 Evaluate the effect of age on ratio using the parameter estimates table? What you will learn • Multiple Linear Regression – Basic concepts • • • • • – – – – – – – – Some examples Linear regression model Estimation and testing the regression coefficients Testing and evaluating the regression model Predictions Multiple regression models The model building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative Predictor variables Practical examples Linear regression Assessing the fit of the model Analysis of Variance summarizes info about the sources of variation in the data by splitting the total sum of squares into two or more components. Y X SSTotal = SSModel + SSError n n n i 1 i 1 2 2 ˆ ˆ (Yi Y ) (Yi Y ) (Yi Yi ) 2 i 1 Linear regression ANOVA table for simple regression Source of Variation Sum of Squares Degrees of Freedom Mean Square F-ratio Model SSM 1 SSM/dfM MSM/MSE Error SSE n-2 SSE/dfE Total SST n-1 SST/dfT P-value In general, dfM equals the number of predictor terms in the model; dfE equals the number of observations minus the number of estimated coefficients in the model; and dfT equals the number of observations minus 1 (if the intercept is included in the model) Linear regression Significance of the model • F-ratio (and its p-value) is used to evaluate significance of the regression model. • MSModel / MSError ≈ F1;n-2 for simple regression • If the observed F-ratio is greater than a critical F-value, than we can conclude that this ratio is significantly greater than 1 and that the regression model explains a significant portion of the variation of the response variable. • Since the simple regression model has only one predictor variable, the F-ratio can also be used to determine if β1 = 0, i.e. if there is a significant effect of the predictor on the response variable (note: squared t-ratio = F-ratio) Linear regression Measure of Fit • A natural measure of ‘goodness of fit’ for the regression model is the coefficient of determination: R2 • R2 expresses the % of variability of the dependent variable explained by the variations in the independent variable • R2 = total variation (SST) – unexplained variation (SSE) total variation (SST) • Properties – R2 varies between 0 and 1 (perfect fit) – The larger R2 is, the more variation of Y is explained by the predictor X – Large values indicate a “strong” relationship between predictor and response variables. – For simple linear regression, R2 is the square of the correlation coefficient. Linear Regression Growth Example Linear Fit ratio = 0,66562 + 0,00528 age Summary of Fit RSquare 0,822535 RSquare Adj 0,819999 Root Mean Square Error 0,051653 Mean of Response 0,855556 Observ ations (or Sum Wgts) 72 Analy sis of Variance Source Model DF Sum of Squares Mean Square F Ratio 1 0,8656172 0,865617 324,4433 Error 70 0,1867605 0,002668 Prob>F C Total 71 1,0523778 <,0001 Parameter Estimates Term Std Error t Ratio Prob>|t| Intercept 0,6656231 Estimate 0,012176 54,67 <,0001 Lower 95% 0,6413397 0,6899065 age 0,0052759 0,000293 18,01 <,0001 0,0046917 0,0058601 Is this regression model significant ? What % of the variation of the response is explained by the model? Upper 95% Linear regression Examine residuals It is always a good idea to look at the residuals from a regression (the difference between the actual values and the predicted values). Residuals should be scattered randomly about a mean of zero. Linear Regression Residual analysis • Residual is the difference between the observed value and the fitted value at a certain level of X ^ ei Yi Y i • Once a model has been fit, the residuals are used to: • Validate the assumptions of the model • Diagnose departures from those assumptions • Identify corrective methods to refine the model Linear Regression Predictions • Two type of predictions of the response Y at new levels of X, the predictor variable, can be made using a validated regression equation. – Estimating the mean response (mean of the distribution of the response Y at new level of X) – Estimating a new observation of Y (individual outcome drawn from the distribution of the response Y at new level of X) • We calculate prediction intervals using the variance of the estimators. The estimation interval for an individual outcome is always larger than the one for a mean response, since the variance of the individual responses is greater than the variance of the mean response. Linear Regression Confidence band for regression line • This band allows us to see the region in which the entire regression line lies. It is useful for determining the appropriateness of a fitted regression function. Linear Regression Demonstration How to do a linear regression analysis with the EXCEL data analysis option ? What you will learn • Multiple Linear Regression – – – – – – – – – Basic concepts Multiple regression models The model building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative Predictor variables Practical examples Multiple linear regression Fitness demo example Aerobic fitness can be evaluated using a special test that measures the oxygen uptake of a person running on a treadmill for a prescribed distance. However, it would be more economical to evaluate fitness with a formula that predicts oxygen uptake using simple measurements such as running time and pulse. The table on the next slide shows the partial listing of the measurements for 31 subjects. Objective Find the regression equation that allows us to make the most reliable predictions of O2 uptake. Multiple Regression Fitness Data Subject • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Donna Gracie Luanne Mimi Chris Allen Nancy Patty Suzanne Teresa Bob Harriett Jane Harold Sammy Buffy Trent Jackie Ralph Jack Annie Kate Carl Don Effie George Iris Mark Steve Vaughn William Sex F F F F M M F F F F M F F M M F M F M M F F M M F M F M M M M Age 42 38 43 50 49 38 49 52 57 51 40 49 44 48 54 52 52 47 43 51 51 45 54 44 48 47 40 57 54 44 45 Weight 68,15 81,87 85,84 70,87 81,42 89,02 76,32 76,32 59,08 77,91 75,07 73,37 73,03 91,63 83,12 73,71 82,78 79,15 81,19 69,63 67,25 66,45 79,38 89,47 61,24 77,45 75,98 73,37 91,63 81,42 87,66 Oxy 59,57 60,06 54,30 54,63 49,16 49,87 48,67 45,44 50,55 46,67 45,31 50,39 50,54 46,77 51,85 45,79 47,47 47,27 49,09 40,84 45,12 44,75 46,08 44,61 47,92 44,81 45,68 39,41 39,20 39,44 37,39 RunTime 8,17 8,63 8,65 8,92 8,95 9,22 9,40 9,63 9,93 10,00 10,07 10,08 10,13 10,25 10,33 10,47 10,50 10,60 10,85 10,95 11,08 11,12 11,17 11,37 11,50 11,63 11,95 12,63 12,88 13,08 14,03 RunPulse 166 170 156 146 180 178 186 164 148 162 185 168 168 162 166 186 170 162 162 168 172 176 156 178 170 176 176 174 168 174 186 RstPulse 40 48 45 48 44 55 56 48 49 48 62 67 45 48 50 59 53 47 64 57 48 51 62 62 52 58 70 58 44 63 56 MaxPulse 172 186 168 155 185 180 188 166 155 168 185 168 168 164 170 188 172 164 170 172 172 176 165 182 176 176 180 176 172 176 192 Multiple linear regression • To investigate the effect on the response variable Y, of several independent X variables, simultaneously. • Even if we are interested in the effect of only one variable, it is wise to include other variables as regressors to reduce the residual variance and improve significance tests of the effects. • Multiple regression models often improve precision of the predictions. Multiple linear regression • The model : yi 0 1 xi1 2 xi 2 ... p xip i • βi represents the change in the response for an incremental change in the ith predictor variable, while all other predictor variables are held constant. βi is referred to as the partial regression coefficient. • Assumptions: residuals (or errors) εi are independent and normally distributed with mean 0 and standard deviation σ Multiple linear regression Estimated additive model with two predictors yi = b0 + b1 xi1 + b2 xi2 + ei b0 : Y value when X1 and X2 equal 0 b1 : effect of X1 on Y controlling for X2 b2 : effect of X2 on Y controlling for X1 Multiple linear regression Fitness Example Response: O2 Uptake Summary of Fit RSquare 0,761424 RSquare Adj 0,744383 Root Mean Square Error 2,693374 Mean of Response 47,37581 Observ ations (or Sum Wgts) Model explains 76% of the variation around the mean of 02 uptake. 31 Parameter Estimates Term Estimate Std Error t Ratio Prob>|t| Lower 95% 76,191947 Upper 95% Intercept 93,088766 8,248823 11,29 <,0001 Run Time -3,140188 0,373265 -8,41 <,0001 -3,90478 -2,375595 Run Pulse -0,073509 0,050514 -1,46 0,1567 -0,176983 0,0299637 Ef fect Test Source Nparm DF Sum of Squares F Ratio Prob>F Run Time 1 1 513,41745 70,7746 <,0001 Run Pulse 1 1 15,36208 2,1177 0,1567 109,98559 Evaluate the effect of Run Time and Run Pulse on O2 uptake If you take an effect away from the model, then the SSError will be higher. Difference in SS is used to construct an F-test on whether the contribution of the variable is significant. Prediction Equation: O2 uptake = 93,089 – 3,14 Run time – 0.074 Run Pulse Multiple Regression Whole Model Leverage Plot Graphical method to view the wholemodel hypothesis using a scatterplot of actual response values against the predicted values. The vertical distance from a point to the 45° line is the residual error. The idea is to get a feel for how much better the slope line fits than the horizontal line at the mean. If the confidence curves cross the horizontal line, the whole model F test is significant. Mulitple Regression Effect Leverage Plots partial plot, partial regression leverage plot, added variable plot Plot shows how each effect contributes to the fit after all the other effects have been included in the model. The distance from each point to the sloped line measures the residual for the full model. The distance from each point to the horizontal line measures the residual for a model without the effect (reduced model). Multiple regression models Interaction model with two predictor variables Y = β0 + β1 X1 + β2 X2 + β12 X1X2 + ε The change in the response associated with X1 depends on the level of X2 (and vice versa) Multiple regression models Quadratic model with two predictor variables Y = β0 + β1 X1 + β2 X2 + β12 X1X2 + β11 X12 + β22 X22 + ε Quadratic models can only represent three basic types of shapes: mountains, valleys, saddles. Multiple regression models • Model terms may be divided into the following categories – – – – – Constant term Linear terms / main effects (e.g. X1) Interaction terms (e.g. X1X2) Quadratic terms (e.g. X12) Cubic terms (e.g. X13) • Models are usually described by the highest term present – Linear models have only linear terms – Interaction models have linear and interaction terms – Quadratic models have linear, quadratic and first order interaction terms – Cubic models have terms up to third order. What you will learn • Multiple Linear Regression – – – – – – – – – – Basic concepts Multiple regression models The model-building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative Predictor variables Interaction effects Practical examples The model-building process Source: Applied Linear Statistical Models, Neter, Kutner, Nachtsheim, Wasserman The model-building process Aerobic fitness can be evaluated using a special test that measures the oxygen uptake of a person running on a treadmill for a prescribed distance. However, it would be more economical to evaluate fitness with a formula that predicts oxygen uptake using simple measurements such as running time and pulse. The table shows the partial listing of the measurements for 31 subjects. Age Weight O2 uptake Runtime Rest pulse Run pulse Max Pulse 38 81,87 60,055 8,63 48 170 186 38 89,02 49,874 9,22 55 178 180 40 75,07 45,313 10,07 62 185 185 40 75,98 45,681 11,95 70 176 180 42 68,15 59,571 8,17 40 166 172 44 85,84 54,297 8,65 45 156 184 Objective Find the regression equation that allows us to make the most reliable predictions of O2 uptake. Selection of predictor variables Objective • Goal is to find the “best” model that is able to predict well over the range of interest. Many variables (especially in exploratory observational studies) may contribute to the response variation. Include them all in the model ? • No, the model must also be parsimonious – A simple model with only a few relevant explanatory variables is easier to understand and use than a complex model – Pareto principle – a few variables contribute the most of information • Reducing the number of variables reduces ‘multicollinearity’ • Increasing ratio observations/variables reduces the variability of b and improves prediction. • Gathering and maintaining data on many factors is difficult and expensive • In the words of Albert Einstein : “Make things as simple as possible but no simpler” Model Selection Methods • Find the ‘best’ model by comparing all possible regression models using all combinations of the explanatory variables. • Automatic model selection methods – Forward selection – Backward elimination – Stepwise selection Model selection methods All Possible Subsets • A regression model is estimated for each possible subset of the predictor variables. The constant term is included in each of the subset models. • If there are k possible terms to be included in the regression model, there are 2k possible subsets to be estimated. • Purpose of the all subsets approach is to identify a small group of models that are “good” according to a specified criterion, so that further detailed examination can be done of these models. Determining the Best Model • A variety of statistics have been developed to help determine the “best” subset model. These include : – – – – – – Coefficient of determination : R2 Adjusted R2 Relative PRESS Mallows Cp Akaike information criterion (AIC) Swarz Information Criterion (SIC or BIC) • Residual analysis and various graphical displays also help to select the “best” subset regression model. Coefficient of Determination • R2 measures the proportion of the total variation in the response that is explained by the regression model, i.e. R2 SSModel SSError 1 SSTotal SSTotal • Values of R2 close to 1 indicate a good fit to the data. However, R2 can be arbitrarily increased by adding extra terms in the model without necessarily improving the predictive ability of the model. • To correct for the number of terms in the model, Adjusted R2 is defined as SSError / df Error 2 2 with Radj 1 Radj 1 SSTotal / dfTotal 2 • Large difference between R2 and Radj indicates the presence of unnecessary terms in the model. Relative PRESS • The PRedictive Error Sums of Squares is given by, n 2 ˆ ( Y Y ) i (i ) PRESS = i 1 where Yˆ represents the predicted value of Yi using the model that was fitted (i ) with the ith observation deleted. • Relative Press is similar to R2 and 2 Radj Relative PRESS = 1 – PRESS / SSTotal • It can be shown that Relative PRESS ≤ 1 Mallows Cp Statistic Cp (n p) s 2p s 2 (n 2 p) • n is the number of observations • p is the number of terms in the model (including the intercept) • sp2 is the estimate of error from the subset model containing p terms • s2 is the estimate of error from the model containing all possible terms. s2 is assumed to be a “good” estimate of the experimental error. If a model with p terms is adequate, Cp ≈ p Otherwise, if the model contains unnecessary terms, Cp > p AIC and BIC AIC (Akaike Information Criterion) and BIC (Swarz Information Criterion) are two popular model selection methods. They not only reward goodness of fit, but also include a penalty that is an increasing function of the number of estimated parameters. This penalty discourages overfitting. The preferred model is the one with the lowest value for AIC or for BIC. These criteria attempt to find the model that best explain the data with a minimum of free parameters. The AIC penalizes free parameters less strongly than does the Schwarz criterion. AIC = 2k + n [ln (SSError / n)] BIC = n ln (SSError / n) + k ln(n) Model selection methods Stepwise regression • While the All Subsets procedure is the only way to guarantee that the “best” subset is chosen, other techniques have been developed that are less computationally intensive. • Stepwise Regression is an automatic model selection procedure which enters or removes terms sequentially. • The basic steps in the procedure are: – Compute an initial regression model – Enter terms that significantly improve the fit (p-value less than the p-toenter value), or remove terms that do not significantly harm the fit (pvalue greater than the p-to-remove value) – Compute the new regression model – Stop when entering or removing terms will not significantly improve the model • Unfortunately, the order in which the terms are entered or removed can lead to different models. Variable Selection Fitness Example p-values to enter or remove a variable from the model model comparison criteria Overview of the variable selection procedure Best model Best Model with Leverage Plot Fitness Example Response: Oxy Summary of Fit RSquare 0,835425 RSquare Adj 0,810106 Root Mean Square Error 2,321437 Mean of Response 47,37581 Observ ations (or Sum Wgts) 31 Parameter Estimates Term Estimate Std Error t Ratio Prob>|t| 11,65754 8,34 <,0001 Intercept 97,185202 Age -0,189218 0,09439 -2,00 0,0555 Runtime -2,775606 0,341602 -8,13 <,0001 RunPulse -0,345272 0,118209 -2,92 0,0071 MaxPulse 0,2714364 0,134383 2,02 0,0538 Ef fect Test Source F Ratio Prob>F Age Nparm 1 DF 1 Sum of Squares 21,65647 4,0186 0,0555 Runtime 1 1 355,78610 66,0199 <,0001 RunPulse 1 1 45,97614 8,5314 0,0071 MaxPulse 1 1 21,98663 4,0799 0,0538 Multiple linear regression Example from cardiology Mauri et al, Circulation 2005 Multiple linear regression Example from cardiology Mauri et al, Circulation 2005 What you will learn • Multiple Linear Regression – – – – – – – – – Basic concepts Multiple regression models The model-building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative Predictor variables Practical examples What you will learn • Multiple Linear Regression – Model diagnostics and Remedial measures • • • • • • Assumptions Scaling of Residuals Distribution of Residuals Unequal Variances Outliers and Influential Observations Multicollinearity – Remedial Measures – Model Validation Building the Regression Model Diagnostics Remember that the residuals are useful for : Validating the assumptions of the model Diagnosing departures from those assumptions Identifying corrective methods to refine the model ei Yi Yˆi Linear regression Assumptions Least square assumptions Linear relation between X and Y : E(εi) = 0 for all i Homoscedasticity (constant variance): Var(εi) = σ2 for all i Uncorrelated residuals : E(εi εj) = σ2 for all i ≠ j Significance tests assumptions Residuals are normally distributed : εi ≈ N ( 0, σ2 ) for all i Scaling of Residuals • Raw – the unscaled residuals These are in the original units of the response. They are used to diagnose extreme values in magnitude and direction. • Standardized – to make the residuals more comparable Standardized residuals are calculated dividing the residuals by their standard deviation. They are approximately normal with a mean of zero and a standard deviation of one. Hence approximately 95% of the standardized residuals should be between + 2σ. Examining residuals Growth example The picture you hope to see is the residuals scattered randomly about a mean of zero. Look for patterns and for points that violate this random scatter. The plot above is suspicious. Why? Violation against which assumption? Examining residuals Regression line with extreme points excluded. Now the residuals are scattered randomly around a mean of zero. But excluding points is probably not the best solution. Examining residuals Comparison of Linear and Second Order Polynomial Fit Ratio = b0 + b1 age + b2 age2 + residual There still apppears to be a pattern in the residuals. Continue by fitting a model with higher order terms. Examining residuals Comparison of Linear and Higher Order Polynomial Fit The residuals of the fourth order polynomial fit do not show any pattern anymore. Violation Linearity Assumption Remedial measures • Polynomial regression: previous example • Transformation (Box-Cox) of the independent variable • Broken line (or piecewise) regression Distribution of residuals Normality check • Graphical Methods – Histogram, Box-plot – Normal Probability Plot (NPP) • Formal Tests given in many stat. packages – Shapiro-Wilk statistic W for small samples (< 50) – Kolmogorov-Smirnov test • If violation against the normality assumption of Y we can try to solve the problem using transformations of Y (use NPP and Box-Cox to decide which transformations). Distribution of residuals Normality check Polynomial 4th order model Simple regression 0,10 .01 .05 .10 .25 .50 .75 .90 .95 0,08 .99 .01 .05 .10 .25 .50 .75 .90 .95 .99 0,06 0,05 0,04 -0,00 0,02 -0,05 0,00 -0,10 -0,02 -0,15 -0,04 -0,20 -0,06 -0,25 -0,08 -3 -2 -1 0 1 2 3 -3 Normal Quantile -2 Normal Quantile Test f or Normality Test f or Normality Shapiro-Wilk W Test Shapiro-Wilk W Test W 0,862849 W Prob<W <,0001 -1 Conclusions ? 0,973852 Prob<W 0,3949 0 1 2 3 Normality check Example from cardiology: late loss Mauri et al, Circulation 2005 Normality check Example from cardiology: late loss Mauri et al, Circulation 2005 Why graphics are important ? Linear Fit Statistical reports for four analysis Linear Fit Y 1 = 3,00009 + 0,50009 X1 Y 2 = 3,00091 + 0,5 X2 Summary of Fit Summary of Fit RSquare 0,666542 RSquare 0,666242 RSquare Adj 0,629492 RSquare Adj 0,629158 Root Mean Square Error 1,236603 Root Mean Square Error 1,237214 Mean of Response 7,500909 Mean of Response 7,500909 Observ ations (or Sum Wgts) 11 Observ ations (or Sum Wgts) Analy sis of Variance Source DF Model What do you expect about the underlying data ? C Total Analy sis of Variance Sum of Squares Mean Square F Ratio Source DF Model 1 27,500000 27,5000 17,9656 9 13,776291 1,5307 Prob>F 10 41,276291 1 27,510001 27,5100 17,9899 9 13,762690 1,5292 Prob>F Error 10 41,272691 0,0022 C Total Error Parameter Estimates Term 11 Sum of Squares Mean Square 0,0022 Parameter Estimates Estimate Std Error t Ratio Prob>|t| Term Intercept 3,0000909 1,124747 2,67 0,0257 Intercept X1 0,5000909 0,117906 4,24 0,0022 X2 Estimate 3,0009091 0,5 Std Error t Ratio Prob>|t| 1,125302 2,67 0,0258 0,117964 4,24 0,0022 Linear Fit Linear Fit Y 4 = 3,00173 + 0,49991 X4 Y 3 = 3,00245 + 0,49973 X3 Summary of Fit Summary of Fit RSquare 0,666324 RSquare 0,666707 RSquare Adj 0,629249 RSquare Adj 0,629675 Root Mean Square Error 1,236311 Root Mean Square Error 1,235695 Mean of Response 7,500909 Mean of Response 7,5 Observ ations (or Sum Wgts) Observ ations (or Sum Wgts) 11 Model Error C Total DF 11 Analy sis of Variance Analy sis of Variance Source F Ratio Sum of Squares Mean Square F Ratio Source Model 1 27,470008 27,4700 17,9723 9 13,756192 1,5285 Prob>F Error 10 41,226200 0,0022 C Total DF Sum of Squares Mean Square F Ratio 1 27,490001 27,4900 18,0033 9 13,742490 1,5269 Prob>F 10 41,232491 0,0022 Parameter Estimates Parameter Estimates Std Error t Ratio Prob>|t| Term Std Error t Ratio Prob>|t| Intercept 3,0024545 1,124481 2,67 0,0256 Intercept 3,0017273 1,123921 2,67 0,0256 X3 0,4997273 0,117878 4,24 0,0022 X4 0,4999091 0,117819 4,24 0,0022 Term Estimate Estimate Why graphics are important ? Regression lines for four analysis Unequal Variances • Model Assumption: var (εi) = var (yi) is a constant σ2 • Heteroscedasticity (unequal variances) does not bias the estimates of the regression parameters β but it causes variances of parameter estimates to be large and can affect R2, s2 and tests substantially. • Detect heteroscedasticity through plots of the (standardized) residuals against ŷ • Remedial actions : – Variance stabilizing transformations of yi (eg. square root, logarithm) – Weighting the regression parameters (WLS) Unequal Variances example Residuals e ŷ Outliers and Influential Observations • Sometimes, while most of the observations fit the model, some of the observations clearly do not. This occurs when there is something wrong with the observations or if the model is faulty. • A point has great influence when it has a large effect on the parameter estimates. • There are two types of influential observations – Outliers: extreme observations of the dependent variable that exhibit large residuals – Leverage points: observation with extreme value on one of the independent variables • Outliers are detected by examining various types of residuals. • Leverage points are detected with the leverage hii This measure describes how far away a point is from the centroid of all points in the space of the independent variables. So, leverage is a measure of remoteness. • Influence is assessed by examining the residuals and the leverages. Outliers and Influential Observations Influential Outlier Influential Leverage point Outliers and Influential diagnostics • For detecting leverage points we examine the leverage hii of the observations. ( X i X )2 1 hii n n 2 ( X X ) j j 1 • leverage value of ith observation increases with increasing devation of Xi of the mean of this variable. • hii takes values between 0 and 1 and the sum is p (number of parameters) • hii with values greater than 2p/n are called leverage points and need further investigation Outliers and Influential diagnostics • For detecting outliers that do not belong to the model, Studentized (deleted) residuals are mostly used. e * i s(i ) ei t n p 1 1 hii s(i) is equivalent to the standard deviation s if least squares is run after deleting the ith case. hii is the leverage Outliers and Influential diagnostics • Not all influential points have large ei* s. • Additional measures are defined, measures that tell us how much parameter b or ŷ would change if a given point were deleted. • Most used measures are : DFBETAS DFFITS ei2 hii COOK’s Distance = 2 2 k 1s (1 h ) ii • Criteria can be defined to decide if point is influential. “Large” Cook’s Distance values are the most influential and should be investigated. Checking for Influential Observations Contour curves show Cook’s Distances corresponding to the 50th, 75th and 95th influence percentiles. As a result, 5% of the residuals will always fall outside the 95th percentile curve. Multicollinearity • The quality of estimates, as measured by their variances, can be seriously affected if the independent variables are closely related (highly correlated) to each other. • An obvious method of assessing the degree to which each independent variable is related to all other independent variables is to examine R2 • Popular measures: – tolerance TOLj = 1 – Rj2 – variance inflation factor VIFj = TOLj -1 Remedial measures Overview • Depending on the nature of the problem, one or more of the following may be appropriate : – – – – – – – – Consider Transforming the data Consider using Weighted Least Squares Consider using Robust Regression Use a more complicated equation e.g. add quadratic or cubic terms to the model Add an omitted predictor variable Consider more complicated models e.g. time series models Consider variable reduction techniques Consider ridge regression Model validation • After remedial measures have been taken and diagnostics analyzed to make sure that the remedial measures were succesful, the final step of the modelbuilding proces is the validation of the selected regression model. • Three basic ways of validating a regression model are: – Collection of new data to check predictive ability of the model → preferred method, but not practical – Comparison of the results with theory, simulation results or previous empirical results – Split the study data into model-building (training) and validation data set randomly : cross-validation. Validation data set is used to re-estimate and compare the regression coefficients. Sample Size • For the linear regression model the desired sample size is determined via an analysis of the power of the F-test. • For this analysis we need following info : – the number of predictors you want to analyze (rule of thumb: number of observations must be at least 15 times number of predictors in the study) – the significance level alpha (type 1 error) – the size of the effect in the population (use measures for proportion of explained variation such as R2 for the whole model) • The bigger the expected effect, the smaller the size of the sample. What you will learn • Multiple Linear Regression – – – – – – – – – Basic concepts Multiple regression models The model-building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative predictor variables Practical examples Categorical predictors • So far we have utilized only quantitative predictor variables in the regression models. • Qualitative predictor variables can also be incorporated in the linear regression model by using indicator variables. • Indicator variables or dummy variables are variables that take only two values eg. 0 and 1 or -1 and 1. • Let’s have a look at the simplest example : one qualitative predictor with two levels: dichotomous predictor. Dichotomous predictor Fitness example Regression model with one dichotomous predictor variable. yi = β0 + β1 xi + ε with xi = -1 for male and xi = 1 for female Response: Oxy This is a model for the familiar two-sample testing problem : H0: µ1 = µ2 against H1: µ1 ≠ µ2 Summary of Fit RSquare 0,234692 RSquare Adj 0,208302 Root Mean Square Error 4,740032 Mean of Response 47,37581 Observ ations (or Sum Wgts) 31 Parameter Estimates Term Estimate Std Error t Ratio Prob>|t| Intercept 47,293867 0,851778 55,52 <,0001 Sex[F-M] 2,5401333 0,851778 2,98 0,0057 Ef fect Test Source Sex Nparm 1 DF 1 Sum of Squares 199,81246 F Ratio Prob>F 8,8932 0,0057 What can you conclude from the statistical output about the mean response for the males versus the females ? Dichotomous predictor Leverage Plots Polychotomous predictor • To incorporate a categorical variable with more than two levels in the regression model, we will create several dummy variables. • For a variable with g categories, we need to incorporate only g -1 dummy variables in the model. The category for which the dummy variable is not in the model is the reference category. Category A B X1 (A) 1 0 X2 (B) 0 1 X3 (C) 0 0 C 0 0 1 Polychotomous predictor Example Let’s examine the effect of drug treatment (a, d or placebo) on the response (LBS=bacteria count) in 30 subjects. Which statistical method would you use to tackle this problem ? Polychotomous predictor Example Response: LBS Summary of Fit RSquare 0,227826 RSquare Adj 0,170628 Root Mean Square Error 6,070878 Mean of Response 7,9 Observ ations (or Sum Wgts) 30 Parameter Estimates Term Estimate Std Error t Ratio Prob>|t| 7,9 1,108386 7,13 <,0001 Drug[a-placebo] -2,6 1,567494 -1,66 0,1088 Drug[d-placebo] -1,8 1,567494 -1,15 0,2609 Intercept Ef fect Test Source Drug Nparm 2 DF 2 Sum of Squares 293,60000 F Ratio Prob>F 3,9831 0,0305 From linear regression to the general linear model. Coding scheme for the categorical variable defines the interpretation of the parameter estimates. Polychotomous predictor Example - Regressor construction • Terms are named according to how the regressor variables were constructed. • Drug[a-placebo] means that the regressor variable is coded as 1 when the level is “a”, - 1 when the level is “placebo”, and 0 otherwise. • Drug[d-placebo] means that the regressor variable is coded as 1 when the level is “d”, - 1 when the level is “placebo”, and 0 otherwise. • You can write the notation for Drug[a-placebo] as ([Drug=a][Drug=Placebo]), where [Drug=a] is a one-or-zero indicator of whether the drug is “a” or not. • The regression equation then looks like: Y = b0 + b1*((Drug=a)-(Drug=placebo)) + b2*(Drug=d)-(Drug=placebo)) + error Polychotomous predictor Example – Parameters and Means • With this regression equation, the predicted values for the levels “a”, “d” and “placebo” are the means for these groups. • For the “a” level: Pred y = 7.9 + -2.6*(1-0) + -1.8*(0-0) = 5.3 For the “d” level: Pred y = 7.9 + -2.6*(0-0) + -1.8(1-0) = 6.1 For the “placebo” level: Pred y = 7.9 + -2.6(0-1) + -1.8*(0-1) = 12.3 • The advantage of this coding system is that the regression parameter tells you how different the mean for that group is from the means of the means for each level (the average response across all levels). • Other coding schemes result in different interpretations of the parameters. What did you learn - hopefully • Multiple Linear Regression – – – – – – – – – Basic concepts Multiple regression models The model-building process Selection of predictor variables Model diagnostics Remedial measures Model validation Qualitative predictor variables Practical examples Linear regression: do-it-yourself with SPSS Scatterplot Linear regression Linear regression Linear regression Questions? Take home messages • Multiple regression models are generally used to study the effect of continuous independent variables on a continuous response variable, simultaneously. • Before modelling, look carefully at the data and always start the analysis by plotting all variables individually and against each other. Evaluate correlations. • Keep in mind the model assumptions from the start. • Building the ‘best’ regression model is an iterative process. • Be careful with automatic variable selection procedures. • Examine residuals and leverages to diagnose and remedy the model. • Validate the model using an independent data set. And now a real break… For further slides on these topics please feel free to visit the metcardio.org website: http://www.metcardio.org/slides.html