Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
6.1.4 AIC, Model Selection, and the Correct Model o Any model is a simplification of reality o If a model has relatively little bias, it tends to provide accurate estimates of the quantities of interest o Best model is often the simplest (less parameters)- model parsimony Akaike Information Criterion (AIC)- alternative to significance tests to estimate quantities of interest o Criterion for choosing between competing statistical models o AIC judges a model by how close its fitted values tend to be to the true values o The AIC selects the model that minimizes: AIC = -2(maximized log likelihood – # parameters in the model) o This penalizes a model for having too many parameters o Serves the purpose of model comparison only; does not provide diagnostic about the fit of the model to the data AIC = -2(maximized log likelihood – # parameters in the model) In SAS: AIC = -2LogL + 2p Crab Example : Table 6.2 (p. 215): The best models have smallest AIC’s o Best models have main effects, COLOR and WIDTH (AIC = 197.5) Parameter D F Estimate Standard Error Wald Chi-Square Pr > ChiSq PROC LOGISTIC (Backward Elimination) : Intercept 1 -12.3508 2.6287 22.0749 <.0001 proc logistic descending ; class color spine / param = ref ; model y = width weight color spine / selection = backward lackfit ; width 1 0.4972 0.1017 23.8872 <.0001 Backward Elimination Procedure Step 0. The following effects were entered: Intercept width weight color spine Step 1. Effect spine is removed Step 2. Effect weight is removed Analysis of Maximum Likelihood Estimates Model Fit Statistics Intercept Only Intercept and Covariates AIC 227.759 201.202 -2 Log L 225.759 185.202 Criterion In our case, AIC is equal in all steps: 227.759 = -2LogL + 2p = 225.759 + 2(1), where p = 1 6.1.5 Using Causal Hypotheses to Guide Model Building o Rather than using selection techniques, such as stepwise, which look at significance levels of each parameter, use theory and common sense to build a model (Add and remove parameters that make sense) o A time ordering among variables may suggest causal relationships Example : (table 6.3, p. 217) In a British study, 1036 men and women (married and divorced) were asked whether they’ve had premarital and/or extramarital sex. We want to determine whether G = gender, P = premarital sex, and E = extramarital sex are factors in whether a person is M= married or divorced. Simple Model : G→P→E→M Any of these is an explanatory variable when a variable listed to its right is the response Complex Model (Triangular) : (Fig. 3.1, p. 218) 1st stage : predicts G has a direct effect on P 2nd stage : predicts P and G have direct effects on E 3rd stage : predicts E has direct effect on M ; P has direct and indirect effects on M; G has indirect effects through P and E Table 6.4 : Goodness of Fit Tests for Model Selection 1st Stage : predicts Gender has a direct effect on Premarital Sex ˆ 100 x 219 .27 141x576 The estimated odds of premarital sex for females is .27 times that for males. data causal2 ; input gender $ PMS TOTALPMS ; datalines ; F 100 676 M 141 360 ; Model (Response P, no Actual Explanatory) PROC GENMOD DATA = CAUSAL2 DESCENDING ; CLASS GENDER ; MODEL PMS/TOTALPMS = / DIST = BIN LINK = LOGIT; Model (Response P, Actual Explanatory G) PROC GENMOD DATA = CAUSAL2 DESCENDING ; CLASS GENDER ; MODEL PMS/TOTALPMS = GENDER / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; PMS Yes No Total Female 100 576 676 Male 141 219 360 241 795 1036 Total Criteria For Assessing Goodness Of Fit Criterion DF Value Value/DF Deviance 1 75.2594 75.2594 Pearson Chi-Square 1 78.1753 78.1753 Log Likelihood -561.9568 Criteria For Assessing Goodness Of Fit Criterion DF Value Deviance 1 0.000 Pearson Chi-Square 1 0.000 Log Likelihood -524.3271 Value/DF Goodness of Fit as a Likelihood-Ratio Test The L-R statistic -2(L0 – L1) test whether certain model parameters are zero by comparing the log likelihood L1 for the fitted model M1 with L0 for the simpler model M0 (formula p. 187) For the example, we will use the fact -2(L0 – L1) = G2(M0) - G2(M1) using SAS output. 1st Stage : G2 = G2(M0) - G2(M1) = 75.2594 – 0.0000 = -2(L0 – L1) = -2(-561.9568 – (-524.3271) Df = 1 – 0 = 1, so χ2 p-value < .001 and effect on pre marital sex suggesting is a better model. 75.2594 = 75.2594 there is evidence of a gender having G as an explanatory variable 2nd Stage : predicts Gender and Premarital Sex have direct effects on Extramarital Sex PMS GENDER Female Male EMS Yes No Total Yes 21 79 100 No 40 536 576 Yes 39 102 141 No 21 198 219 data causal3 ; input gender $ PMS $ EMS TOTALEMS ; datalines ; F Y 21 100 F N 40 576 M Y 39 141 M N 21 219 ; Model (Response E, no Actual Explanatory) PROC GENMOD DATA = CAUSAL3 DESCENDING ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Model (Response E, P Actual Explanatory) PROC GENMOD DATA = CAUSAL3 ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = PMS / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Model (Response E, G+P Actual Explanatory) PROC GENMOD DATA = CAUSAL3 DESCENDING ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = GENDER PMS / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS Model (Response E, no Actual Explanatory) Model (Response E, P Actual Explanatory) Criteria For Assessing Goodness Of Fit Criteria For Assessing Goodness Of Fit Criterion D F Value Value/DF Deviance 3 48.9244 16.3081 Pearson Chi-Square 3 56.7739 18.9246 Log Likelihood Criterion DF Value Value/DF Deviance 2 2.9080 1.4540 Pearson Chi-Square 2 2.9542 1.4771 Log Likelihood -350.4605 -373.4687 Model (Response E, G+P Actual Explanatory Criteria For Assessing Goodness Of Fit Criterion DF Value Value/DF Deviance 1 0.0008 0.0008 Pearson Chi-Square 1 0.0008 0.0008 Log Likelihood ; -349.0069 Model E = 1 vs. E = P G2(M0) - G2(M1) = 48.9244 – 2.9080 = 46.016 -2(L0 – L1) = -2(-373.4687–(-350.4605) = 46.016 df = 3-2= 1, so χ2 p-value < .001, so there is evidence of a P effect on E Model E = P vs. E = G+P G2 = G2(M0) - G2(M1) = 2.9080 - .0008 = 2.9 df = 2-1 = 1, so χ2 p-value > .10 so only weak evidence occurs that G had a direct effect as well as indirect effect on E. So E = P is a sufficient model. 3rd stage : predicts Extramarital Sex has direct effect on Marriage ; Premarital Sex has direct and indirect effects on Marriage; Gender has indirect effects through PMS and EMS PMS EMS GENDER Female Yes Yes No Yes No No Yes 17 4 21 No 54 25 79 Yes 36 4 40 214 322 536 Yes 28 11 39 No 60 42 102 Yes 17 4 21 No 68 130 198 No Male Divorced Model M = E + P vs. M = E*P G2 = G2(M0) - G2(M1) = 18.1596 – 5.2455 so χ2 p-value < .10 so the interaction Model M = E*P vs. M = E*P + G G2 = G2(M0) - G2(M1) = 5.2455 - .6978 = so χ2 .025 < p-value < .05 so adding G data causal ; input gender $ PMS $ EMS $ DIVORCED TOTAL ; datalines ; F Y Y 17 21 F Y N 54 79 F N Y 36 40 F N N 214 536 M Y Y 28 39 M Y N 60 102 M N Y 17 21 M N N 68 198 ; = 12.91, with df = 5-4= 1 EMS*PMS is a better model to predict Divorce 4.5477, with df = 4-3= 1 to interaction EMS*PMS fits slightly better. Conclusion for Causal Relationships Good alternative for model building by using common sense to hypothesize relationships 6.1.6 New Model-Building Strategies for Data Mining o Data mining is the analysis of huge data sets, in order to find previously unsuspected relationships which are of interest or value o Model Building is challenging o There are alternatives to traditional statistical methods, such as automated algorithms that ignore concepts such as sampling error and modeling o Significance tests are usually irrelevant, since nearly any variable has significant effect if n is sufficiently large o For large n, inference is less relevant than summary measures of predictive power