Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
The Impact of Family Income on SAT Scores Bryan LaBrecque PADM 9040 Research methods in Public administration April 9, 2012 Br. Baracskay Introduction The Scholastic Aptitude Test (SAT) has long been a measuring stick used by colleges to aid in the selection process of prospective students. In an annual right of spring (and often summer and fall), high school students anxiously await their test scores. Colleges use these scores in several ways: 1) to sift through the thousands of applicants, 2) to predict prospective students’ first year grades, 3) to select scholarship awards and other types of financial assistance, and 4) as means of academic pride between colleges. The SAT clearly carries a great deal of “weight” in the academic world, but is also not without its detractors or its controversy. In recent years, naysayers have identified several biases that they believe exist in the system, which render the test scores less worthy. Issues such as racial composition, cultural biases, age distributions, wealth, and the availability of quality secondary education, have all been mentioned as reasons to either eliminate the SAT altogether or at minimum, reduce its weighting in the selection process. While the foundation of the SAT exam has remained relatively constant, the test has received several “makeovers” over the years, in an attempt to quell some of the controversy. On might ask why this topic is important. Standardized tests have been in existence for a hundred years or so and impact the students and their schooling from kindergarten through higher education. Logically, there must be a reasonable amount of benefit to the metrics associated with them. What makes the SAT such a polarizing test is the impact it has on a young person’s future earnings potential. Scores of articles and research papers have dedicated themselves to understanding the value of a college degree as it relates to wage compensation. 2 This paper will not attempt to supplement that research but will use their outcome – college degrees having a strong positive association with earnings potential – as the motivation for better understanding some key drivers for SAT performance results. Clearly, students with higher SAT scores will have more opportunities to attend prestigious colleges and universities, providing them with higher earnings potential. The question under consideration in this research paper is whether or not higher scores on the SAT can “purchased” and if so, what ramifications are associated with this result. In other words, this paper will investigate the many variables associated with the SAT and attempt to isolate the impact of wealth – or family income – on those scores. The analysis uses secondary data, available from a number of sources, and utilizes statistically based research methodology to attempt to project family income as a cause of successful testing. Before embarking on the analysis itself, several reference articles and contemporary research was reviewed to better understand other’s views and to take advantage of work previously accomplished and knowledge previously gained on the topic. What follows is a brief literature review. Literature review The subject of SAT score bias has received as much research attention as there are theories as to why the scores differ from student to student. Among the more prevalent research regarding the Scholastic Aptitude Test involves comparative studies. These range over a myriad of variables including racial bias, test preparation bias, gender bias, geographic bias, testing bias and income 3 bias (the principle topic of this paper). Since each of these variables may or may not have an impact on the income study, a better understanding of each was needed to assure that their impact was not overlooked. A review of a select few of these studies is helpful in laying the groundwork for this study. Among the more frequent studies, geographic impact – specifically state to state comparison studies – has garnered a great deal of interest, especially from the both the media and state politicians who are each jockeying for good news/bad news stories. Authors Wainer, Holland, Swinton and Wang produced a compelling analysis of state-by-state SAT comparisons using the 1984 and 1985 “State Education Statistics” released by then Secretary of Education Bell. This table compared the 50 states on several interesting and related educational variables. According to the authors, the very nature of the data points be exhibited side-by-side, compels one to begin comparing numbers – especially SAT scores v. many of the aforementioned variables. The authors contend that this compulsion could lead to some very faulty conclusions. In particular, Wainer et al, are concerned that the impact of participation rates both between states and within states is glossed over. The primary concern of this missing input is the self-selection error associated with the data. Ultimately, the authors conclude that while “the correlation between income and SAT Verbal is substantial”, the self selection process – “those with both high income and high SAT capability tend to take the test…” (Wainer et al, 1985, 307)” - skews the data. Stephan F. Gohmann expands on the issues of comparing geographic SAT scores in his journal article. He also expresses considerable concern over the research methodology and the impact of selection bias on the data set. He suggests the need to inject a “selection” correction to consider 4 the impact of self-selection. In his article, he produced a table of SAT scores, by state, with their predicted rankings without selection correction. He then updated the equation with one of several forms of corrections, resulting in extremely different residual rankings. “For example, Alabama’s ranking fell from its observed value of 26 to 29 when the standard estimating equation residual is used. When the selection equation is used, Alabama is ranked sixth.” (Gohmann, 1988, 145) Gohmann strongly suggests that participation rate and the bias of self-selection are keys to predicting SAT scores. Another instrumental variable in SAT scores results from familiarization with the test itself. This situation arises when students have taken the PSAT previously or have participated in public or private SAT tutoring, or even via the SAT Test-taking Booklet itself. Authors Powers and Alderman (who, as members of the Educational Testing Service, have authored several articles regarding standardized testing) tested the concept of test familiarization on SAT test scores to help understand the impact of preparation classes. They concluded that while familiarization with a test such as the SAT assists students both academically and emotionally, preparation courses do not significantly impact test scores. “In summary, this study strongly suggests that the opportunity for self-study of additional, more extensive information about a test like the SAT is welcomed by test candidates, but that any additional benefits beyond those resulting from previous 5 exposure to the test and from familiarization information available in test bulletins is not likely to effect the test scores of typical candidates.” (Alderman and Powers, 1983 , 79). Alderman and Powers team up again in another article slightly more pertinent to the topic of this paper, concerning themselves with the impact of special preparation for the SAT – a potential proxy for family income. In that article, the authors studied students at eight secondary schools, assigned some to a treatment group (special preparation for the SAT) and others to a control group (delayed access to the same preparation), in an attempt to determine if the additional preparation impacted scores significantly. In this study, Alderman and Powers concluded that “regular school practices or characteristics exert a stronger influence on test scores than does special preparation for the test.” (Alderman and Powers, 1980, 250). Finally, like many other researchers, Rebecca Zwick asserts that test scores are positively associated with socioeconomic factors, but through her own research determines that wealth is also a measure of most other previous educational achievement. In that sense, the SAT is not any more biased than other measures of academic capability. “This evidence fails to support the claim that score gaps result mainly from the inclusion of esoteric test content that is more familiar to upper-crust test-takers or from inequalities in access to coaching services.” (Zwick, 2002, 310) 6 Research Design and Hypothesis A fairly standard statistical methodology will be utilized in the research for this assessment. As the intent of the study is to determine the strength of relationship between SAT scores and household income, the mean scores and their associated standard deviations will be computed for several increasing income levels, in a given year. The mean was selected because the data follows a fairly normal distribution with very little skew. The Standard Deviation is computed for each income level to determine the variability of data points. One point of emphasis here is that since the secondary data being used represents the population of data, random sampling errors do not result, significantly simplifying the analysis. Variables In building the analysis, it is critical to first understand the difference between the independent variable(s) and the dependent variable. In our case, SAT scores (both Math and Verbal) represent our dependent variable. Several independent variables, when applied appropriately will result in the specific scores. For this study, our independent variables are: Family Income High School GPA Parental Education level High School GPA – A student’s high school GPA represents his or her grasp of specific subject knowledge and is an indicator of work ethic, organization skills and learning capacity. As such it 7 is reasonable to assume that this metric is a contributor to the ability of a student to achieve scores on the SAT. Family Income – Many experts believe that household income affects students’ ability to score on the SAT. Higher income often means that the student lives in a more affluent area which equally often results in better schools and resources. In addition, higher incomes usually result in more disposable income to be used for computers, internet access, tutoring sessions and SAT preparation courses. Parental Education Levels – Parents with college degrees tend to focus their children on the need for college and as a result may create an environment more conducive to success on the SAT. Data Manipulation The data itself represents the population of SAT takers and is presented as a mean over a range of family incomes. Since averages were used and there is no sampling error (since there is no sampling done), the data was manipulated to produce a single set of discrete data. First, the mean of test scores of SAT Verbal and SAT Math were combined and the resulting figures were paired with the mid-point of the family income level. In addition, since we used the population of data versus random sampling, the need for null hypotheses is eliminated because there are no sampling errors to be concerned over. The means of the data represent the true differences. 8 Research Hypothesis Since we are attempting to determine our relationship strength between family income and SAT scores we will set our research hypothesis thusly: “Family income is a key and significant driver of higher test scores and has a distinctly positive relationship.” Regression From there simple regression will be run as a foundational predictor. This will produce a best-fit line through our data set and allow the researcher to make a broad projection of SAT outcomes based on our equation. While a multiple regression analysis would provide a more refined association, the complexity of the analysis goes beyond the preliminary nature of this paper. The use of multiple regression and control variables will be discussed in the assessment of the analysis results and possible next steps. A simple regression of our data set will provide us with the coefficient of the independent variable (slope) as well as the constant (y-intercept) of our best-fit linear equation and Correlation Once the independent variable and dependent variable are established, the methodology continues by determining the correlation between the variables. Correlation is determined by computing the Pearson R. The Pearson product-moment correlation coefficient, as it is properly known, will range from -1.00 to 1.00. The closer the Pearson R is to -1.00 or 1.00, the stronger 9 the relationship between the two variables, with a negative coefficient indicating an inverse relationship and a positive coefficient denoting a direct relationship. The association between the two variables is further refined by determining the Coefficient of Determination. The Coefficient of Determination is computed by squaring the Pearson R value and represents a percentage of how much variance on one variable is accounted for by the variance on another. In our case, the Coefficient of Determination will help us understand the percent of variance on SAT scores which is accounted for by the variance on family income. Practical Significance Finally, once we have determined the association and correlation strength, we will assess the overall practical significance of the variances. Analysis of Data Data The data used to assess our hypothesis came from the “2009 College-Bound Seniors Total Group Profile Report” as provided by The College Board. As previously discussed the data, being categorical in nature, was manipulated to create quantitative data sets more useful to the researcher. The data, in its delivered form, is shown in charts 1 and 2 below. 10 Chart 1 – Mean of SAT–Math scores shown as a function of ordinal levels of family income Chart 2 – Mean of SAT–Verbal scores shown as a function of ordinal levels of family income The data shown above are the mean scores within each income level computed from the population of test-takers in 2009. The total number of participants was 1,530,128 students. 11 Scatter Plot Chart 3 below, represents a scatter plot graph of the combine SAT scores paired with the midpoint of the family income ordinal level. Chart 3 – Scatter Plot of Mean of SAT–Combined scores shown as a function of the mid-point of the ordinal levels of family income Regression Analysis A single regression analysis was performed on the x-y dataset in order to establish the best-fit line within the scatter plot. This analysis provides significant insight into the association between the two test variables – SAT scores and Family Income. The line itself is defined by the slope (coefficient of independent variable) and its y-intercept (constant). The correlation between the two variables (independent and dependent) is also determined as are various other measures of relationship. Table I below, provides the results of the regression. Each of the outcomes will be discussed further. 12 SAT SCORES v. FAMILY INCOME REGRESSION ANALYSIS SUMMARY OUTPUT Regression Statistics Multiple R 0.889160251 R Square 0.790605952 Adjusted R Square 0.764431696 Standard Error 38.29168357 Observations 10 Intercept X Variable 1 Coefficients Standard Error t Stat 925.2515152 24.30934806 38.06155199 0.001158485 0.000210789 5.495951186 P-value 2.49296E-10 0.000576492 Table I – Single Regression Analysis results The best-fit line determined from the scatter data by the linear regression analysis is defined by its slope (.00115) and its y-intercept (925.25). Applying these coefficients to the equation results in the following linear equation: Y = 925.25 + .00115*X Where Y represents the combined SAT scores (dependent variable) and X represents the family income (independent variable). The equation states - in mathematical terms - that for every $1 of extra income in a family, there is an associated rise in SAT score of .00115 points. More appropriately, a $1,000 rise in income is associated with a rise of 1.15 points on the SAT. 13 The correlation R is measured at .8891. This number, being close to 1.00 indicates a strong positive association between family income and SAT scores. We can be more concise by using the Coefficient of Determination (R-Squared). The R squared for our study is .7906, which in terms of percent indicates that 79% of the variance of SAT scores is accounted for by the variance in family income. This level of R-squared leaves us with 21% of the variation in SAT scores not accounted for by family income. We can further refine the correlation results by understanding that with only a single independent variable we need to adjust our result downward. The Adjusted R square of 76.4% represents this downward adjustment in association. As the linear regression analysis draws a best fit line amongst a scatter plot of data, some error will result as a result of the “fitting” process. Our residual – standard error – in this study was 38.29 pts – not an insignificant disagreement. This essentially dictates that our study regression (least squares) line met the predictive criteria within +/- 3%. Conclusion This study has taken the first step in analyzing impact of family income on average SAT scores. The results clearly show that there exists a high correlation between test scores and family income. The correlation is positive in nature, which one would assume. Those with higher incomes, generally have more resources for their children, live in more affluent neighborhoods with better equipped schools. 14 Can we, though, make a leap of faith to identify family income as the cause of higher test scores? Causation is a very difficult proposition to prove. Our hypothesis stated that family income IS a significant driver of higher test scores, but without expanding this study to include more independent variable as well as control variables, family income may not be as significant as other variables such as geographic location or parental education. In addition a path diagram should be assembled to assess both common cause and potential as an intervening variable. Ultimately a multiple regression analysis should be performed using all the available variables. Even at that, the controversy over income levels and scores may not abate. The impact of these studies have far-reaching ramifications for public policy. Should we spend more resources on preparing students for the test? Should we otherwise spend our resources on improving schools in lower income areas? Is there truly an income driver here or would we be wasting resources attempting to impact the scores and lessen the score gaps? Only time - and a great deal more study - will tell. 15 References: Remler, Dahlia K., and Gregg G. Van Ryzin. 2011. Research Methods in Practice. Thousand Oaks, CA: Sage. Pyrczak, Fred. 2010. Making Sense of Statistics: A Conceptual Overview. Los Angeles: Pyrczak Publishing. Rosenberg, Kenneth M., 2007. The Excel Statistics Companion. Belmont, CA: WadsworthCengage. Alderman, Donald L. and Donald E. Powers.1983. “Effects of Test Familiarization on SAT Performance.” Journal of Educational Measurement 20 (Spring): 71-79 Alderman, Donald L. and Donald E. Powers. 1980. “The Effects of Special Preparation on SATVerbal Scores.” American Educational Research Journal 17 (Summer): 239-251 Gohmann, Stephan F. 1988. “Comparing SAT Scores: Problems, Biases, and Corrections.” Journal of Educational Measurement 25 (Summer): 137-148 16 Lee, Young-joo, and Vicky M. Wilkins. 2011. “More Similarities or More Differences? Comparing Public and Non-Profit Managers’ Job Motivations.” Public Administration Review (January-February): 45-56 Wainer, Howard, and Paul W. Holland and Spencer Swinton and Min Hwei Wang. 1985. “On State Education Statistics”. Journal of Educational Statistics 10 (Winter): 293-325 Zwick, Rebecca. 2002. “Is the SAT a ‘Wealth Test’?” .The Phi Delta Kappan 84 (December): 307-311 No author. 1998. “Why Family Income Differences Don’t Explain the Racial Gap in SAT Scores”. The Journal of Blacks in Higher Education 20 (Summer): 6-8 No author. 2007. “Does the SAT Merely Test a Student’s Ability to Pay Tuition?”. The Journal of Blacks in Higher Education 56 (Summer): 36-37 17