Download RATING SYSTEMS

RATING SYSTEMS
Andrew Sanfratello
Mohegan Lake, NY
Teacher’s Guide — Getting Started
Purpose
In this two-day lesson, students will model rating systems like those used in many sports. They are asked to
consider the various factors that the human mind employs to “rate” one team over another; they will then
model a way to consider these factors in order to make a systematic, mathematical rating method. Note that
even professional rating systems often are disputed for their “accuracy”: such is the nature of both mathematical modeling and sports!
Begin with the description of the situation: you are trying to compare teams or players, but not every
team/player plays the other, so there is no clear “clean-cut” method. How can you devise a system to do
this?
Prerequisites
Students should understand basic probability concepts such as the computation and meaning of “rate of
success”. Students should be able to interpret the meaning of expressions in an equation or function.
Materials
Required: Internet access (for research), calculators.
Suggested: None.
Optional: None.
Worksheet 1 Guide
The first three pages of the lesson constitute the first day’s work. Students consider the factors that they
think should be included when comparing one team or player to another. They use this intuition to create a
simple model for a rating system. Students are introduced to the Elo Rating System, one of the first systems
of its kind, which was developed for chess players. They perform Internet research to determine what is
included in the system and compare the system to their model, which they try to refine.
Worksheet 2 Guide
The fourth and fifth pages of the lesson constitute the second day’s work. Students consider the different
factors of the Elo system and make judgments about them based on both mathematics and intuition. Students then consider another rating system, RPI, and make decisions about its effectiveness based on their
experience and intuition.
CCSSM Addressed
F-BF.1: Write a function that describes a relationship between two quantities.
S-MD.5: (+) Weigh the possible outcomes of a decision by assigning probabilities to payoff values and finding expected values.
S-MD.6: (+) Use probability to make fair decisions.
S-MD.7: (+) Analyze decisions and strategies using probability concepts.
141
RATING SYSTEMS
Student Name:_____________________________________________ Date:_____________________
In many professional games and sports, players or teams are rated in relation to others. This rating helps
determine which players or teams are a good match for one another and helps determine who might win in
a matchup between any two.
Leading Question
How would you devise a system to determine rating?
142
RATING SYSTEMS
Student Name:_____________________________________________ Date:_____________________
1. What factors would you consider in determining the rating of each person or team?
2. If ratings are determined by the previous games won and/or lost, how could you use an opponents’
rating to determine a new rating for a player or team?
3. Can you create a model, such as a function, that would determine
the increase or decrease for each opponent in a match depending
on who wins?
143
What should be true about
the model? Are there any
properties that should always
be true in any rating system?
RATING SYSTEMS
Student Name:_____________________________________________ Date:_____________________
4. Does your model include the problem of one of the opponents being previously unrated? How might
you handle the situation of an unrated player? Incorporate this into your model if you haven’t already.
5. One of the first rating systems was devised for chess players and is known as the Elo Rating System,
named after Arpad Elo, its creator. The Elo system has three elements that help to determine the “Per
Game Rating Change”: K-factor, Expected Result, and Score. Research these factors and determine the
meaning of each one.
6. What similarities or differences do the Elo system and your model have? Are there changes that you
would make to your model now that you know how the Elo system works?
144
RATING SYSTEMS
Student Name:_____________________________________________ Date:_____________________
Recall what your last model looked like and the information you found on the Elo rating.
7. How does the K-factor affect the rating of a player? Is it reasonable to use a fixed number? Did you use a
fixed number in your initial rating system from question 3?
8. How is the Expected Result calculated? Did you use a similar mathematical method in your model?
9. The “Rule of 400” states that if two players are more than 400
points apart, then to determine the Expected Result, you assume
that they are exactly 400 points apart. Why would this rule come
to exist?
145
Think of the case where a
very skilled player plays a
bad, inexperienced player.
RATING SYSTEMS
Student Name:_____________________________________________ Date:_____________________
10. In college basketball, RPI (Rating Percentage Index) is calculated with three factors: Winning Percentage (WP), Opponents’ Average Winning Percentage (OWP), and Opponents’ Opponents’ Average Winning Percentage (OOWP). The weights used are 25%, 50%, and 25%, respectively. What similarities or
differences does this have with the prior models?
11. A team has two options: they can play 5 other teams with an average winning percentage of 80% and
an OWP of 90% and they’ll likely win 1 out of the 5 games, or they can play 5 teams with an average
winning percentage of 40% and an OWP of 50% and win 4 out of the 5 games. Which scenario will generate a greater RPI?
12. Is the weighting applied to the three factors in RPI appropriate? How might you change the weighting
and/or include factors to alter the weighting?
146
RATING SYSTEMS
Teacher’s Guide — Possible Solutions
The solutions shown represent only some possible solution methods. Please evaluate students’ solution
methods on the basis of mathematical validity.
1. Answers will vary but may include opponents’ prior record and/or rating, the number of games played,
“strength of schedule”, and (if applicable) home/away records.
2. Generally, if Team A is rated and Team B is new and unrated, then if B beats A, B will have a higher rating than A. If A beats B, B will have a lower rating than A.
3. Answers will vary, but one way is to award the team or player with 1 point for a win, 0.5 points for a tie,
and 0 points for a loss. The team or player with the most points will have the best rating.
4. This model does not account for unrated opponents. However, unrated teams or players may become
rated by earning enough points to surpass a rated team or player.
5. K-factor is a number applied to a player (which varies according to the player’s rating) and is used to
balance highly rated players from increasing their rating easily. Expected Result is the expected score
given a player’s rating and the opponent’s rating before they have actually played. Score is the actual
result that occurs after the players have played each other. Scores in chess are 1 point for a win, 0.5
points for a tie, or 0 points for a loss.
6. The model above does not take into account various factors that the Elo system does. For example, it
does not take into account opponents’ rating or ability nor does it account for the advantage a highly
rated player has over a very lowly rated or unrated player. It does, like Elo, consider ties to be “half-win”
and “half-loss”.
7. K-factor diminishes the value of individual games played by players with more experience. This causes
more fluctuations with novice players’ ratings, but more stability with expert players’ ratings.
8. Expected Result is calculated for Player A with rating X against Player B with rating Y with the formula
1
EA =
Y −X
1 + 10 400
.
9. The Rule of 400 prevents highly rated players from gaining points on their rating from playing people
who are greatly below their skill level, and thus falsely boosting their rating.
10. Answers will vary, although as compared with the Elo system, both have three variables taken into
account, although they are all quite different.
11. The first choice of games gives the team an RPI of 0.675 while the second choice of games (despite winning more of them) produces an RPI of only 0.525.
12. The RPI has a very high focus on “strength of schedule” — how well opponents perform — and performance of the team itself only accounts for 25% of the rating. Other factors to include might be based
on score differences or how well a team does in “away” games, which are said to be more difficult to
win.
147
RATING SYSTEMS
Teacher’s Guide — Extending the Model
One of the outcomes of this lesson is a formula that is used for rating chess players: we have seen that
the Expected Result for Player A when Player A with rating X plays Player B with rating Y is given by the
formula
1
EA =
Y −X .
1 + 10 400
We see that it is based on a logistic curve. What do we expect from a formula like that?
1) If A is a lot better than B, we expect that it will give an answer close to 1. Well, suppose that X is a lot bigger than Y, say X–Y = 360. Then the exponent (Y–X)/400 is –0.9, 10–0.9 is about 1/8, so EA is about 0.89.
On the other hand, if we reverse the abilities of A and B, that is, set X–Y = –360, then EA = 0.11. This fits
our expectation of symmetry.
2) If the players are equal in ability, then X = Y, the exponent is 0, and EA = 0.5. This fits our expectation of
equality between the two players.
3) Suppose it has been observed that A wins about 1 time out of 3 against B. The difference in rating is
expected to be Y–X = 120 since then the exponent is about 0.3, and 100.3 = 2. This expectation should
work both ways: players that win 1 time out of 3 against an opponent (or equally rated opponents)
should have a rating 120 less than these opponents, and also, players whose rating is 120 less than an
opponent’s should win 1 time out of 3.
4) This Expected Result can actually be interpreted as the probability that A will win plus half the probability of a draw. This implies that for all X and Y, EA + EB = 1. The algebraic exercise to show that this is true
is not quite a one-step trivial exercise, and should not be missed.
5) All of this depends, of course, on how a player’s rating is computed and adjusted. An important issue is
human behavior given the rating system and the natural desire to improve one’s rating.
How does a formula like this compare to a formula in physics, such as the formula for the range of a batted
ball hit with velocity V at an angle a with the horizontal? Yes, that formula usually ignores air resistance and
the height of the batter, but we can defend it on the basis of the principles of mechanics. We could correct
for the height of the batter, and even for air resistance, and really believe the answer. The philosophy behind
our formula for rating chess players is different. We want the formula to act as a probability, and to behave
in certain limiting ways. We want it to agree with our rating system. If it has the effects we desire, we accept
it, not because in some deeper sense we know it to be correct, but because it has the right shape and gives
results we like and can use.
Mathematical models in many aspects of social science often satisfy similar expectations. The shape of the
curve is right, the way we use it to optimize behavior or expenditures makes sense, and we don’t expect the
numbers to be exactly right. For example, a manufacturer expects that there is flexibility in the use of capital
versus the use of labor in a given production program. If we have more machinery and automation, we have
fewer laborers. People like to use a formula of the form
K αL1–α = C
where K stands for capital, L stands for labor, C is a constant, and α is an exponent between 0 and 1 chosen
to be reasonable for the particular industry under consideration. We don’t expect this formula to be exact,
but it’s the right shape and it fits real data at a couple of points pretty well. What can you do with it? Well,
for example, you can draw a line of fixed expenditure on the same plot and get a pretty good idea of the mix
of capital and labor that will give you the most product for your money. But don’t forget maintenance!
This, as people like to say, is not rocket science, but it’s typical of the kind of models that can be created and
used outside of the “hard” sciences.
148