Download The Scholastic Aptitude Test (SAT)

The Impact of Family Income on SAT Scores
Bryan LaBrecque
PADM 9040 Research methods in Public administration
April 9, 2012
Br. Baracskay
Introduction
The Scholastic Aptitude Test (SAT) has long been a measuring stick used by colleges to aid in
the selection process of prospective students. In an annual right of spring (and often summer and
fall), high school students anxiously await their test scores. Colleges use these scores in several
ways: 1) to sift through the thousands of applicants, 2) to predict prospective students’ first year
grades, 3) to select scholarship awards and other types of financial assistance, and 4) as means of
academic pride between colleges.
The SAT clearly carries a great deal of “weight” in the academic world, but is also not without
its detractors or its controversy. In recent years, naysayers have identified several biases that they
believe exist in the system, which render the test scores less worthy. Issues such as racial
composition, cultural biases, age distributions, wealth, and the availability of quality secondary
education, have all been mentioned as reasons to either eliminate the SAT altogether or at
minimum, reduce its weighting in the selection process. While the foundation of the SAT exam
has remained relatively constant, the test has received several “makeovers” over the years, in an
attempt to quell some of the controversy.
On might ask why this topic is important. Standardized tests have been in existence for a
hundred years or so and impact the students and their schooling from kindergarten through
higher education. Logically, there must be a reasonable amount of benefit to the metrics
associated with them. What makes the SAT such a polarizing test is the impact it has on a young
person’s future earnings potential. Scores of articles and research papers have dedicated
themselves to understanding the value of a college degree as it relates to wage compensation.
2
This paper will not attempt to supplement that research but will use their outcome – college
degrees having a strong positive association with earnings potential – as the motivation for better
understanding some key drivers for SAT performance results. Clearly, students with higher SAT
scores will have more opportunities to attend prestigious colleges and universities, providing
them with higher earnings potential.
The question under consideration in this research paper is whether or not higher scores on the
SAT can “purchased” and if so, what ramifications are associated with this result. In other
words, this paper will investigate the many variables associated with the SAT and attempt to
isolate the impact of wealth – or family income – on those scores. The analysis uses secondary
data, available from a number of sources, and utilizes statistically based research methodology to
attempt to project family income as a cause of successful testing.
Before embarking on the analysis itself, several reference articles and contemporary research
was reviewed to better understand other’s views and to take advantage of work previously
accomplished and knowledge previously gained on the topic. What follows is a brief literature
review.
Literature review
The subject of SAT score bias has received as much research attention as there are theories as to
why the scores differ from student to student. Among the more prevalent research regarding the
Scholastic Aptitude Test involves comparative studies. These range over a myriad of variables
including racial bias, test preparation bias, gender bias, geographic bias, testing bias and income
3
bias (the principle topic of this paper). Since each of these variables may or may not have an
impact on the income study, a better understanding of each was needed to assure that their
impact was not overlooked. A review of a select few of these studies is helpful in laying the
groundwork for this study.
Among the more frequent studies, geographic impact – specifically state to state comparison
studies – has garnered a great deal of interest, especially from the both the media and state
politicians who are each jockeying for good news/bad news stories. Authors Wainer, Holland,
Swinton and Wang produced a compelling analysis of state-by-state SAT comparisons using the
1984 and 1985 “State Education Statistics” released by then Secretary of Education Bell. This
table compared the 50 states on several interesting and related educational variables. According
to the authors, the very nature of the data points be exhibited side-by-side, compels one to begin
comparing numbers – especially SAT scores v. many of the aforementioned variables. The
authors contend that this compulsion could lead to some very faulty conclusions. In particular,
Wainer et al, are concerned that the impact of participation rates both between states and within
states is glossed over. The primary concern of this missing input is the self-selection error
associated with the data. Ultimately, the authors conclude that while “the correlation between
income and SAT Verbal is substantial”, the self selection process – “those with both high income
and high SAT capability tend to take the test…” (Wainer et al, 1985, 307)” - skews the data.
Stephan F. Gohmann expands on the issues of comparing geographic SAT scores in his journal
article. He also expresses considerable concern over the research methodology and the impact of
selection bias on the data set. He suggests the need to inject a “selection” correction to consider
4
the impact of self-selection. In his article, he produced a table of SAT scores, by state, with their
predicted rankings without selection correction. He then updated the equation with one of
several forms of corrections, resulting in extremely different residual rankings.
“For example, Alabama’s ranking fell from its observed value of 26 to 29 when the
standard estimating equation residual is used. When the selection equation is used,
Alabama is ranked sixth.” (Gohmann, 1988, 145)
Gohmann strongly suggests that participation rate and the bias of self-selection are keys to
predicting SAT scores.
Another instrumental variable in SAT scores results from familiarization with the test itself.
This situation arises when students have taken the PSAT previously or have participated in
public or private SAT tutoring, or even via the SAT Test-taking Booklet itself. Authors Powers
and Alderman (who, as members of the Educational Testing Service, have authored several
articles regarding standardized testing) tested the concept of test familiarization on SAT test
scores to help understand the impact of preparation classes. They concluded that while
familiarization with a test such as the SAT assists students both academically and emotionally,
preparation courses do not significantly impact test scores.
“In summary, this study strongly suggests that the opportunity for self-study of
additional, more extensive information about a test like the SAT is welcomed by test
candidates, but that any additional benefits beyond those resulting from previous
5
exposure to the test and from familiarization information available in test bulletins is not
likely to effect the test scores of typical candidates.” (Alderman and Powers, 1983 , 79).
Alderman and Powers team up again in another article slightly more pertinent to the topic of this
paper, concerning themselves with the impact of special preparation for the SAT – a potential
proxy for family income. In that article, the authors studied students at eight secondary schools,
assigned some to a treatment group (special preparation for the SAT) and others to a control
group (delayed access to the same preparation), in an attempt to determine if the additional
preparation impacted scores significantly. In this study, Alderman and Powers concluded that
“regular school practices or characteristics exert a stronger influence on test scores than does
special preparation for the test.” (Alderman and Powers, 1980, 250).
Finally, like many other researchers, Rebecca Zwick asserts that test scores are positively
associated with socioeconomic factors, but through her own research determines that wealth is
also a measure of most other previous educational achievement. In that sense, the SAT is not any
more biased than other measures of academic capability.
“This evidence fails to support the claim that score gaps result mainly from the inclusion
of esoteric test content that is more familiar to upper-crust test-takers or from inequalities
in access to coaching services.” (Zwick, 2002, 310)
6
Research Design and Hypothesis
A fairly standard statistical methodology will be utilized in the research for this assessment. As
the intent of the study is to determine the strength of relationship between SAT scores and
household income, the mean scores and their associated standard deviations will be computed for
several increasing income levels, in a given year. The mean was selected because the data
follows a fairly normal distribution with very little skew. The Standard Deviation is computed
for each income level to determine the variability of data points. One point of emphasis here is
that since the secondary data being used represents the population of data, random sampling
errors do not result, significantly simplifying the analysis.
Variables
In building the analysis, it is critical to first understand the difference between the independent
variable(s) and the dependent variable. In our case, SAT scores (both Math and Verbal) represent
our dependent variable. Several independent variables, when applied appropriately will result in
the specific scores. For this study, our independent variables are:

Family Income

High School GPA

Parental Education level
High School GPA – A student’s high school GPA represents his or her grasp of specific subject
knowledge and is an indicator of work ethic, organization skills and learning capacity. As such it
7
is reasonable to assume that this metric is a contributor to the ability of a student to achieve
scores on the SAT.
Family Income – Many experts believe that household income affects students’ ability to score
on the SAT. Higher income often means that the student lives in a more affluent area which
equally often results in better schools and resources. In addition, higher incomes usually result
in more disposable income to be used for computers, internet access, tutoring sessions and SAT
preparation courses.
Parental Education Levels – Parents with college degrees tend to focus their children on the need
for college and as a result may create an environment more conducive to success on the SAT.
Data Manipulation
The data itself represents the population of SAT takers and is presented as a mean over a range
of family incomes. Since averages were used and there is no sampling error (since there is no
sampling done), the data was manipulated to produce a single set of discrete data. First, the
mean of test scores of SAT Verbal and SAT Math were combined and the resulting figures were
paired with the mid-point of the family income level. In addition, since we used the population
of data versus random sampling, the need for null hypotheses is eliminated because there are no
sampling errors to be concerned over. The means of the data represent the true differences.
8
Research Hypothesis
Since we are attempting to determine our relationship strength between family income and SAT
scores we will set our research hypothesis thusly:
“Family income is a key and significant driver of higher test scores and has a distinctly
positive relationship.”
Regression
From there simple regression will be run as a foundational predictor. This will produce a best-fit
line through our data set and allow the researcher to make a broad projection of SAT outcomes
based on our equation. While a multiple regression analysis would provide a more refined
association, the complexity of the analysis goes beyond the preliminary nature of this paper. The
use of multiple regression and control variables will be discussed in the assessment of the
analysis results and possible next steps.
A simple regression of our data set will provide us with the coefficient of the independent
variable (slope) as well as the constant (y-intercept) of our best-fit linear equation and
Correlation
Once the independent variable and dependent variable are established, the methodology
continues by determining the correlation between the variables. Correlation is determined by
computing the Pearson R. The Pearson product-moment correlation coefficient, as it is properly
known, will range from -1.00 to 1.00. The closer the Pearson R is to -1.00 or 1.00, the stronger
9
the relationship between the two variables, with a negative coefficient indicating an inverse
relationship and a positive coefficient denoting a direct relationship.
The association between the two variables is further refined by determining the Coefficient of
Determination. The Coefficient of Determination is computed by squaring the Pearson R value
and represents a percentage of how much variance on one variable is accounted for by the
variance on another. In our case, the Coefficient of Determination will help us understand the
percent of variance on SAT scores which is accounted for by the variance on family income.
Practical Significance
Finally, once we have determined the association and correlation strength, we will assess the
overall practical significance of the variances.
Analysis of Data
Data
The data used to assess our hypothesis came from the “2009 College-Bound Seniors Total
Group Profile Report” as provided by The College Board. As previously discussed the data,
being categorical in nature, was manipulated to create quantitative data sets more useful to
the researcher. The data, in its delivered form, is shown in charts 1 and 2 below.
10
Chart 1 – Mean of SAT–Math scores shown as a function of ordinal levels of family income
Chart 2 – Mean of SAT–Verbal scores shown as a function of ordinal levels of family income
The data shown above are the mean scores within each income level computed from the
population of test-takers in 2009. The total number of participants was 1,530,128 students.
11
Scatter Plot
Chart 3 below, represents a scatter plot graph of the combine SAT scores paired with the midpoint of the family income ordinal level.
Chart 3 – Scatter Plot of Mean of SAT–Combined scores shown as a function of the mid-point
of the ordinal levels of family income
Regression Analysis
A single regression analysis was performed on the x-y dataset in order to establish the best-fit
line within the scatter plot. This analysis provides significant insight into the association
between the two test variables – SAT scores and Family Income. The line itself is defined by the
slope (coefficient of independent variable) and its y-intercept (constant). The correlation
between the two variables (independent and dependent) is also determined as are various other
measures of relationship. Table I below, provides the results of the regression. Each of the
outcomes will be discussed further.
12
SAT SCORES v. FAMILY INCOME
REGRESSION ANALYSIS
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.889160251
R Square
0.790605952
Adjusted R Square
0.764431696
Standard Error
38.29168357
Observations
10
Intercept
X Variable 1
Coefficients Standard Error
t Stat
925.2515152
24.30934806 38.06155199
0.001158485
0.000210789 5.495951186
P-value
2.49296E-10
0.000576492
Table I – Single Regression Analysis results
The best-fit line determined from the scatter data by the linear regression analysis is defined by
its slope (.00115) and its y-intercept (925.25). Applying these coefficients to the equation results
in the following linear equation:
Y = 925.25 + .00115*X
Where Y represents the combined SAT scores (dependent variable) and X represents the family
income (independent variable). The equation states - in mathematical terms - that for every $1 of
extra income in a family, there is an associated rise in SAT score of .00115 points. More
appropriately, a $1,000 rise in income is associated with a rise of 1.15 points on the SAT.
13
The correlation R is measured at .8891. This number, being close to 1.00 indicates a strong
positive association between family income and SAT scores. We can be more concise by using
the Coefficient of Determination (R-Squared). The R squared for our study is .7906, which in
terms of percent indicates that 79% of the variance of SAT scores is accounted for by the
variance in family income. This level of R-squared leaves us with 21% of the variation in SAT
scores not accounted for by family income.
We can further refine the correlation results by understanding that with only a single independent
variable we need to adjust our result downward. The Adjusted R square of 76.4% represents this
downward adjustment in association.
As the linear regression analysis draws a best fit line amongst a scatter plot of data, some error
will result as a result of the “fitting” process. Our residual – standard error – in this study was
38.29 pts – not an insignificant disagreement. This essentially dictates that our study regression
(least squares) line met the predictive criteria within +/- 3%.
Conclusion
This study has taken the first step in analyzing impact of family income on average SAT scores.
The results clearly show that there exists a high correlation between test scores and family
income. The correlation is positive in nature, which one would assume. Those with higher
incomes, generally have more resources for their children, live in more affluent neighborhoods
with better equipped schools.
14
Can we, though, make a leap of faith to identify family income as the cause of higher test scores?
Causation is a very difficult proposition to prove. Our hypothesis stated that family income IS a
significant driver of higher test scores, but without expanding this study to include more
independent variable as well as control variables, family income may not be as significant as
other variables such as geographic location or parental education. In addition a path diagram
should be assembled to assess both common cause and potential as an intervening variable.
Ultimately a multiple regression analysis should be performed using all the available variables.
Even at that, the controversy over income levels and scores may not abate. The impact of these
studies have far-reaching ramifications for public policy. Should we spend more resources on
preparing students for the test? Should we otherwise spend our resources on improving schools
in lower income areas? Is there truly an income driver here or would we be wasting resources
attempting to impact the scores and lessen the score gaps? Only time - and a great deal more
study - will tell.
15
References:
Remler, Dahlia K., and Gregg G. Van Ryzin. 2011. Research Methods in Practice. Thousand
Oaks, CA: Sage.
Pyrczak, Fred. 2010. Making Sense of Statistics: A Conceptual Overview. Los Angeles: Pyrczak
Publishing.
Rosenberg, Kenneth M., 2007. The Excel Statistics Companion. Belmont, CA: WadsworthCengage.
Alderman, Donald L. and Donald E. Powers.1983. “Effects of Test Familiarization on SAT
Performance.” Journal of Educational Measurement 20 (Spring): 71-79
Alderman, Donald L. and Donald E. Powers. 1980. “The Effects of Special Preparation on SATVerbal Scores.” American Educational Research Journal 17 (Summer): 239-251
Gohmann, Stephan F. 1988. “Comparing SAT Scores: Problems, Biases, and Corrections.”
Journal of Educational Measurement 25 (Summer): 137-148
16
Lee, Young-joo, and Vicky M. Wilkins. 2011. “More Similarities or More Differences?
Comparing Public and Non-Profit Managers’ Job Motivations.” Public Administration
Review (January-February): 45-56
Wainer, Howard, and Paul W. Holland and Spencer Swinton and Min Hwei Wang. 1985. “On
State Education Statistics”. Journal of Educational Statistics 10 (Winter): 293-325
Zwick, Rebecca. 2002. “Is the SAT a ‘Wealth Test’?” .The Phi Delta Kappan 84 (December):
307-311
No author. 1998. “Why Family Income Differences Don’t Explain the Racial Gap in SAT
Scores”. The Journal of Blacks in Higher Education 20 (Summer): 6-8
No author. 2007. “Does the SAT Merely Test a Student’s Ability to Pay Tuition?”. The Journal
of Blacks in Higher Education 56 (Summer): 36-37
17