Download 7 repeated games.pptx

Introduction to Game Theory
7. Repeated Games
Dana Nau
University of Maryland
Nau: Game Theory 1
Repeated Games
  Used by game theorists, economists, social and behavioral scientists
as highly simplified models of various real-world situations
Iterated Prisoner’s Dilemma
Iterated Chicken Game
Roshambo
Iterated Battle
of the Sexes
Repeated
Ultimatum Game
Repeated Stag Hunt
Repeated
Matching
Nau:Pennies
Game Theory 2
Finitely Repeated Games
Prisoner’s Dilemma:
  In repeated games, some game G is played
multiple times by the same set of agents
2
C
D
C
3, 3
0, 5
D
5, 0
1, 1
1
  G is called the stage game
•  Usually (but not always),
G is a normal-form game
  Each occurrence of G is called
an iteration or a round
  Usually each agent knows what all
the agents did in the previous iterations,
but not what they’re doing in the
current iteration
  Thus, an imperfect-information
game with perfect recall
  Usually each agent’s
payoff function is additive
Iterated Prisoner’s Dilemma,
with 2 iterations:
Agent 1:
Agent 2:
Round 1:
C
C
Round 2:
D
C
3+5 = 5
3+0 = 3
Total payoff:
Nau: Game Theory 3
Strategies
  The repeated game has a much bigger strategy space than the stage game
  One kind of strategy is a stationary strategy:
  Use the same strategy
at every iteration
  More generally, an
Iterated Prisoner’s Dilemma with 2 iterations:
agent’s play at each stage
may depend on the history
  What happened in
previous iterations
Nau: Game Theory 4
Backward Induction
  If the number of iterations is finite and known, we can use backward
induction to get a subgame-perfect equilibrium
  Example: finitely many repetitions of the Prisoner’s Dilemma
  In the last round,
the dominant strategy is D
  That’s common knowledge
  So in the 2nd-to-last round,
D also is the dominant strategy
Agent 1:
Agent 2:
Round 1:
D
D
Round 2:
D
D
Round 3:
D
D
Round 4:
this argument is vulnerable to
both empirical and theoretical criticisms
D
D
  …
  The SPE is (D,D) on every round
  As with the Centipede game,
Nau: Game Theory 5
Backward Induction when G is 0-sum
  As before, backward induction works much better in zero-sum games
  In the last round, equilibrium is the minimax profile
•  Each agent uses his/her minimax strategy
  That’s common knowledge
  So in the 2nd-to-last round, it
again is the minimax strategies
…
  The SPE is (D,D) on every round
Nau: Game Theory 6
Infinitely Repeated Games
  An infinitely repeated game in extensive form would be an infinite tree
  Payoffs can’t be attached to any terminal nodes
  Payoffs can’t be the sums of the payoffs in the stage games (generally infinite)
  Two common ways around this problem
  Let r (1)i , r (2)i , … be an infinite sequence of payoffs for agent i
  Agent i’s average reward is
k
lim ∑ j=1 ri( j ) / k
k→∞
  Agent i’s future discounted reward is the discounted sum of the payoffs, i.e.,
∞
∑
j ( j)
β
ri where€β (with 0 ≤ β ≤ 1) is a constant called the discount factor
j=1
  Two ways to interpret the discount factor:
€
1.  The agent cares more about the preset than the future
2.  The agent cares about the future, but the game ends at any round with
probability 1 − β
Nau: Game Theory 7
Example
  Some well-known strategies for the Iterated Prisoner’s Dilemma:
»  AllC: always cooperate
»  AllD (the Hawk strategy):
TFT
or
Grim AllD
TFT Tester
C
C
C
D
C
D
C
C
D
D
D
C
C
C
D
D
C
C
C
C
D
D
C
C
C
C
D
D
C
C
C
C
D
D
C
C
C
C
D
D
C
C
...
...
…
…
…
…
always defect
»  Grim: cooperate until the
other agent defects,
then defect forever
»  Tit-for-Tat (TFT):
cooperate on the first move.
On the nth move, repeat
the other agent (n–1)th move
»  Tester: defect on move 1. If the
other agent retaliates, play TFT.
Otherwise, randomly intersperse
cooperation and defection
AllC,
AllC,
Grim, Grim,
or TFT or TFT
  If the discount factor is large enough, each of the following is a Nash equilibrium
  (TFT, TFT), (TFT,GRIM), and (GRIM,GRIM)
Nau: Game Theory 8
Equilibrium Payoffs for Repeated Games
  There’s a “folk theorem” that tells what the possible equilibrium payoffs
are in repeated games
  It says roughly the following:
  In an infinitely repeated game whose stage game is G, there is a
Nash equilibrium whose average payoffs are (p1, p2, …, pn)
if and only if
  G has a mixed-strategy profile (s1, s2, …, sn) with the following
property:
•  For each i, si’s payoff would be ≥ pi if the other agents used
minimax strategies against i
Nau: Game Theory 9
Proof and Examples
Example 2:
IPD with
(p1, p2) = (2.5,2.5)
Grim
Other
agent
Agent 1 Agent 2
gives each agent i the average payoff pi,
given certain constraints on (p1, p2, …, pn)
C
C
D
C
C
C
C
D
•  In this equilibrium, the agents cycle in
lock-step through a sequence of game
outcomes that achieve (p1, p2, …, pn)
C
C
D
C
C
C
C
D
C
D
D
D
D
C
D
D
D
C
D
C
D
C
D
D
…
…
  Use the definitions of minimax and best-
response to show that in every equilibrium,
an agent’s average payoff ≥ the agent’s
minimax value
  Show how to construct an equilibrium that
•  If any agent i deviates, then the others
punish i forever, by playing their
minimax strategies against i
  There’s a large family of such theorems,
for various conditions on the game
…
Example 1:
IPD with
(p1, p2) = (3,3)
…
  The proof proceeds in 2 parts:
Nau: Game Theory 10
Zero-Sum Repeated Games
  For two-player zero-sum repeated games, the folk theorem is still true, but it
becomes vacuous
  Suppose we iterate a two-player zero-sum game G
  Let V be the value of G (from the Minimax Theorem)
  If agent 2 uses a minimax strategy against 1, then 1’s maximum payoff is V
•  Thus max value for p1 is V, so min value for p2 is –V
  If agent 1 uses a minimax strategy against 2, then 2’s maximum payoff is –V
•  Thus max value for p2 is –V, so min value for p1 is V
  Thus in the iterated game, the only Nash-equilibrium payoff profile is (V,–V)
  The only way to get this is if each agent always plays his/her minimax strategy
•  If agent 1 plays a non-minimax strategy s1 and agent 2 plays his/her best
response, 2’s expected payoff will be higher than –V
Nau: Game Theory 11
Roshambo (Rock, Paper, Scissors)
Rock
Paper
Scissors
Rock
0, 0
–1, 1
1, –1
Paper
1, –1
0, 0
–1, 1
Scissors
–1, 1
1, –1
0, 0
A2
A1
  Nash equilibrium for the stage game:
  choose randomly, P=1/3 for each move
  Nash equilibrium for the repeated game:
  always choose randomly, P=1/3 for each move
  Expected payoff = 0
  Let’s see how that works out in practice …
Nau: Game Theory 12
Roshambo (Rock, Paper, Scissors)
Rock
Paper
Scissors
Rock
0, 0
–1, 1
1, –1
Paper
1, –1
0, 0
–1, 1
Scissors
–1, 1
1, –1
0, 0
A2
A1
  1999 international roshambo programming competition
www.cs.ualberta.ca/~darse/rsbpc1.html
  Round-robin tournament:
•  55 programs, 1000 iterations
for each pair of programs
•  Lowest possible score = –55000,
highest possible score = 55000
  Average over 25 tournaments:
•  Highest score (Iocaine Powder): 13038
•  Lowest score (Cheesebot): –36006
  Very different from the game-theoretic prediction
Nau: Game Theory 13
  A Nash equilibrium strategy is best for you
if the other agents also use their Nash equilibrium strategies
  In many cases, the other agents won’t use Nash equilibrium strategies
  If you can forecast their actions accurately, you may be able to do
much better than the Nash equilibrium strategy
  Why won’t the other agents use their Nash equilibrium strategies?
  Because they may be trying to forecast your actions too
  Something analogous can happen in non-zero-sum games
Nau: Game Theory 14
Iterated Prisoner’s Dilemma
Prisoner’s Dilemma
  Multiple iterations of the Prisoner’s Dilemma
Cooperate
Defect
Cooperate
3, 3
0, 5
Defect
5, 0
1, 1
P1
  Widely used to study the emergence of
cooperative behavior among agents
  e.g., Axelrod (1984), The Evolution of Cooperation
P2
Nash equilibrium
  Axelrod ran a famous set of tournaments
  People contributed strategies
encoded as computer programs
  Axelrod played them against each other
If I defect now, he might punish
me by defecting next time
Nau: Game Theory 15
TFT with Other Agents
  In Axelrod’s tournaments, TFT usually did best
»  It could establish and maintain cooperations with many other agents
»  It could prevent malicious agents from taking advantage of it
TFT AllC
TFT AllD
TFT Grim
TFT TFT
TFT Tester
C
C
D
C
C
C
C
C
D
C
C
D
D
C
C
C
C
D
C
C
C
D
D
C
C
C
C
C
C
C
C
D
D
C
C
C
C
C
C
C
D
C
C
C
C
C
C
C
C
D
D
D
C
C
C
C
C
C
C
C
C
D
D
C
C
C
C
C
C
...
...
…
…
…
…
…
…
…
…
C
Nau: Game Theory 16
Example:
  A real-world example of the IPD, described in Axelrod’s book:
  World War I trench warfare
  Incentive to cooperate:
  If I attack the other side, then they’ll retaliate and I’ll get hurt
  If I don’t attack, maybe they won’t either
  Result: evolution of cooperation
  Although the two infantries were supposed to be enemies, they
avoided attacking each other
Nau: Game Theory 17
IPD with Noise
  In noisy environments,
  There’s a nonzero probability (e.g., 10%)
C
C
C
C
C
D C
Noise
…
…
that a “noise gremlin” will change some
of the actions
•  Cooperate (C) becomes Defect (D),
and vice versa
  Can use this to model accidents
  Compute the score using the changed
action
  Can also model misinterpretations
  Compute the score using the original
action
Did he really
intend to do that?
Nau: Game Theory 18
Example of Noise
  Story from a British army officer in World War I:
  I was having tea with A Company when we heard a lot of shouting and went
out to investigate. We found our men and the Germans standing on their
respective parapets. Suddenly a salvo arrived but did no damage.
Naturally both sides got down and our men started swearing at the Germans,
when all at once a brave German got onto his parapet and shouted out:
“We are very sorry about that; we hope no one was hurt. It is not our
fault. It is that damned Prussian artillery.”
  The salvo wasn’t the German infantry’s intention
  They didn’t expect it nor desire it
Nau: Game Theory 19
Noise Makes it Difficult
to Maintain Cooperation
  Consider two agents
who both use TFT
  One accident or
misinterpretation
can cause a long
string of retaliations
Retaliation
Retaliation
C
C
C
C
D
C
D
C
C
C
C
D C
C
D
C
D
Noise"
Retaliation
Retaliation
...
...
Nau: Game Theory 20
Some Strategies for the Noisy IPD
  Principle: be more forgiving in the face of defections
  Tit-For-Two-Tats (TFTT)
»  Retaliate only if the other agent defects twice in a row
•  Can tolerate isolated instances of defections, but susceptible to exploitation
of its generosity
•  Beaten by the TESTER strategy I described earlier
  Generous Tit-For-Tat (GTFT)
»  Forgive randomly: small probability of cooperation if the other agent defects
»  Better than TFTT at avoiding exploitation, but worse at maintaining cooperation
  Pavlov
»  Win-Stay, Lose-Shift
•  Repeat previous move if I earn 3 or 5 points in the previous iteration
•  Reverse previous move if I earn 0 or 1 points in the previous iteration
»  Thus if the other agent defects continuously, Pavlov will alternatively cooperate
and defect
Nau: Game Theory 21
Discussion
  The British army officer’s story:
  a German shouted, ``We are very sorry about that; we hope no one was
hurt. It is not our fault. It is that damned Prussian artillery.”
  The apology avoided a conflict
  It was convincing because it was consistent with the German infantry’s
past behavior
  The British had ample evidence that the German infantry wanted to
keep the peace
  If you can tell which actions are affected by noise, you can avoid reacting
to the noise
  IPD agents often behave deterministically
  For others to cooperate with you it helps if you’re predictable
  This makes it feasible to build a model from observed behavior
Nau: Game Theory 22
The DBS Agent
  Work by my recent PhD graduate, Tsz-Chiu Au
  Now a postdoc at University of Texas
  From the other agent’s recent behavior, build a model π of the other
agent’s strategy
  Use the model to filter noise
  Use the model to help plan our next move
Au & Nau. Accident or intention:
That is the question (in the iterated
prisoner’s dilemma). AAMAS, 2006.
Au & Nau. Is it accidental or
intentional? A symbolic approach to the
noisy iterated prisoner’s dilemma. In G.
Kendall (ed.), The Iterated Prisoners
Dilemma: 20 Years On. World
Scientific, 2007.
Nau: Game Theory 23
Modeling the other agent
  A set of rules of the following form
if our last move was m and their last move was m'
then P[their next move will be C]
  Four rules: one for each of (C,C), (C,D), (D,C), and (D,D)
  For example, TFT can be described as
  (C,C)
1, (C, D)
1, (D, C )
0, (D, D)
0
  How to get the probabilities?
  One way: look at the agent’s behavior in the recent past
  During the last k iterations,
  What fraction of the time did the other agent cooperate at iteration j
when the agents’ moves were (x,y) at iteration j–1?
Nau: Game Theory 24
Modeling the other agent
  π can only model a very small set of strategies
  It doesn’t even model the Grim strategy correctly:
  If Grim defects, it may be defecting because of something that
happened many moves ago
  But we’re not trying to model an agent’s entire strategy, just its recent
behavior
  If an agent’s behavior changes, then the probabilities in π will change
  e.g., after Grim defects a few times, the rules will give a very low
probability of it cooperating again
Nau: Game Theory 25
Noise Filtering
  Suppose the applicable rule is
deterministic
  P[their next move will be C] = 0 or 1
  If the other agent’s next move
isn’t what the rule predicts, then
  Assume the
observed action
is noise
  Behave as if the
action were what
the rule predicted
The other
agent
cooperates
when I do
So I won’t
retaliate here. I
think these
defections are
actually noise
C
C
C
C
C
C
C D C
C
C
C D C
C
C
C
C
:
:
Nau: Game Theory 26
I am Grim. If
you ever betray
me, I will never
forgive you.
Change of Behavior
  Anomalies in observed behavior can be due
either to noise or to a genuine change of
behavior
  Changes of behavior occur because
  The other agent can change its strategy
anytime
  E.g., if noise affects one of Agent 1’s
actions, this may trigger a change in
Agent 2’s behavior
•  Agent 1 does not know this
  How to distinguish noise from a real change
of behavior?
C
C
C
C
CD
C
C
D
C
D
C
D
:
D
:
D
:
:
These
moves are
not noise
Nau: Game Theory 27
Detection of a Change of Behavior
Temporary tolerance:
  When we observe unexpected
behavior from the other agent
  Don’t immediately decide
whether it’s noise or a real
change of behavior
  Instead, defer judgment
for a few iterations
  If the anomaly persists, then
recompute π based on the
other agent’s recent behavior
The other
agent
cooperates
when I do
The defections
might be accidents,
so I shouldn’t lose
my temper too soon
I think the other
agent’s has really
changed, so I’ll
change mine too
C
C
C
C
C
C
D
D
:
C
C
C
D
D
D
D
D
:
Nau: Game Theory 28
Move generation
  Modified version of game-tree search
  Use the policy π to predict probabilities of the other agent’s moves
  Compute expected utility) for move x as
u1(x) = ∑ y∈{C,D} u1(x,y) × P(y | π, previous moves)
where x = my move, y = other agent’s move
  Choose the move with the highest expected utility
(C,C)
:
:
:
:
:
(C,D) (D,C)
:
:
:
:
Current
Iteration
(D,D)
:
:
:
Next
Iteration
:
:
:
:
Iteration
after next
Nau: Game Theory 29
Suppose we have the rules
1. (C,C) → 0.7
2. (C,D) → 0.4
3. (D,C) → 0.1
4. (D,D) → 0.1
Example
(C,C)
C
C
C
D
D
C
C
C
??
(C,D) (D,C)
(D,D)
  Suppose we search to depth 1
u1(C) = 0.7 u1(C,C) + 0.3 u1(C,D) = 2.1 + 0 = 2.1
u1(D) = 0.7 u1(D,C) + 0.3 u1(D,D) = 3.5 + 0.3 = 3.8
»  So D looks better
  Is D really what we should choose?
Rule 1 predicts
P(C) = 0.7, P(D) = 0.3
Nau: Game Theory 30
Suppose we have the rules
1. (C,C) → 0.7
2. (C,D) → 0.4
3. (D,C) → 0.1
4. (D,D) → 0.1
Example
(C,C)
C
C
C
D
D
C
C
C
??
Rule 1 predicts
P(C) = 0.7, P(D) = 0.3
(C,D) (D,C)
(D,D)
  It’s not wise to choose D
»  On the move after that, the opponent will
retaliate with P=0.9
»  The depth-1 search didn’t see this
  But if we search to depth d>1, we’ll see it
  C will look better and we’ll choose it instead
  In general, it’s best look far ahead
»  e.g., 60 moves
Nau: Game Theory 31
How to Search Deeper
  Game trees grow exponentially with search depth
»  How to search to the tree deeply?
  Key assumption: π accurately models the other agent’s future behavior
  Then we can use dynamic programming
»  Makes the search polynomial in the search depth
»  Can easily search to depth 60
»  Equivalent to solving an acyclic MDP of depth 60
  This generates fairly good moves
(C,C)
Current
iteration
(C,D) (D,C)
(D,D)
Next
iteration
iteration
after next
:
:
:
:
Nau: Game Theory 32
20th Anniversary IPD Competition
http://www.prisoners-dilemma.com
  Category 2: IPD with noise
  165 programs participated
  DBS dominated the
top 10 places
  Two agents scored
higher than DBS
  They both used
master-and-slaves strategies
Nau: Game Theory 33
Master & Slaves Strategy
  Each participant could submit up to 20 programs
  Some submitted programs that could recognize each other
  (by communicating pre-arranged sequences of Cs and Ds)
  The 20 programs worked as a team
•  1 master, 19 slaves
  When a slave plays with its master
•  Slave cooperates, master defects
My goons give
me
all their
money …
=> maximizes the master’s payoff
  When a slave plays with
an agent not in its team
•  It defects
… and they
beat up
everyone else
=> minimizes the other agent’s payoff
Nau: Game Theory 34
Comparison
  Analysis
  Each master-slaves team’s average score was much lower than DBS’s
  If BWIN and IMM01 had each been restricted to ≤ 10 slaves,
DBS would have placed 1st
  Without any slaves, BWIN and IMM01 would have done badly
  In contrast, DBS had no slaves
  DBS established cooperation
with many other agents
  DBS did this despite the noise,
because it filtered out the noise
Nau: Game Theory 35
Summary
  Finitely repeated games – backward induction
  Infinitely repeated games
  average reward, future discounted reward
  equilibrium payoffs
  Non-equilibrium strategies
  opponent modeling in roshambo
  iterated prisoner’s dilemma with noise
•  opponent models based on observed behavior
•  detection and removal of noise
•  game-tree search against the opponent model
  20th anniversary IPD competition
Nau: Game Theory 36