Download OU - The Computer Science Department

CASE − Cognitive Agents for Social Environments
Yu Zhang
Trinity University | Laboratory for Distributed Intelligent Agent Systems
2
Outline
•
•
•
•
•
•
Introduction
CASE — Agent-Level Solution
CASE — Society-Level Solution
Experiment
A Case Study
Conclusion and Future Work
Trinity University | Laboratory for Distributed Intelligent Agent Systems
3
Introduction
•
•
•
•
•
Multi-Agent Systems
MAS for Social Simulation
Research Goal
Existing Approaches
Our Approach
Trinity University | Laboratory for Distributed Intelligent Agent Systems
4
Multi-Agent Systems
Society
Multi-Agent
Agent
KB
KB
KB
KB
KB
Agents
Societies
HighFrequency
Interactions
Interactions are
decentralized
KB
KB = Knowledge Base
Trinity University | Laboratory for Distributed Intelligent Agent Systems
5
Simulating Social Environments
1998
1997
1996
1995
1992
First international conference on
computer simulation and the
social sciences. Hope is that
computer simulation will achieve
a disciplinary synthesis among
the social sciences.
Journal of Artificial
Societies and Social
Simulation first
published.
Santa Fe Institute becomes well
known for developing ideas about
complexity and studying them
utilizing computer simulations of
real-world phenomena.
Series of workshops held in
Italy and USA. Field becomes
more theoretically and
methodologically grounded.
First ‘Simulating Societies’
workshop held.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
6
Research Goal
Understanding how the decentralized
interactions of agents could generate
social conventions.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
Current Approaches
Agent Level
Focuses on the
self-interested
agents.
Society Level
Focuses on
static social
structures.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
7
8
Our Approach
Cognitive Agents for Social Environments
Network
 Social
 Bounded
Convention
Rationality
 Perception
 Action
Environment
Trinity University | Laboratory for Distributed Intelligent Agent Systems
9
Related Work
Sugarscape
CASE
COGENT SOAR
Meso-Level
Schelling’s Agent behavior
Segregation realistic but not too
computationally
Model
complex
Top-Down
Agent
Complexity
ACT-R
CLARION
Bottom-Up
Trinity University | Laboratory for Distributed Intelligent Agent Systems
10
Outline
•
•
•
•
•
•
Background and Objective
CASE — Agent-Level Solution
CASE — Society-Level Solution
Experiment
A Case Study
Conclusion and Future Work
Trinity University | Laboratory for Distributed Intelligent Agent Systems
11
Our Approach
Cognitive Agents for Social Environments
Network
 Social
 Bounded
Convention
Rationality
 Perception
 Action
Environment
Trinity University | Laboratory for Distributed Intelligent Agent Systems
12
Rationality vs. Bounded Rationality
Rationality means that agents calculate a
utility value for the outcome of every
action.
Bounded Rationality means that agents
use intuition and heuristics to determine if
one action is better than another.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
13
Daniel Kahneman
Courtesy Google Image
Trinity University | Laboratory for Distributed Intelligent Agent Systems
14
Two-Phase Decision Model
Framing
Evaluation
Criteria
Anchoring
Selective
Attention
Phase I - Editing
Phase II - Evaluation
Accessibility
State
Similarity
Decision Mode
Two Modes
of Function
Intuition
Deliberation
Action
Trinity University | Laboratory for Distributed Intelligent Agent Systems
15
Phase I - Editing
Phase II - Evaluation
Trinity University | Laboratory for Distributed Intelligent Agent Systems
16
Phase I - Editing
• Framing: decide evaluation criteria based on one’s
attitude toward potential risk and reward.
• Anchoring: build selective attention on information.
– Salience of information:
A piece of information
i is used

C
i 
, iI
 C I is used  1
Context of the current decision
A set of all information
– Anchored information:
I*= {i | i > threshold}
• Accessibility: determine state similarity only by I*.
Accessibility relation
st ~ sm if distancec, I * (st , sm )  threshold
Current state A memory state
Trinity University | Laboratory for Distributed Intelligent Agent Systems
17
Phase II - Evaluation
• Intuition
– If st ~ sm, the optimal decision policy *(st) and
*(sm) should be close too.
• Deliberation
– Optimize *(st).
Time discount factor
 *( st )  arg max E[



time i  0
 reward( si ) |  ], 0    1
i
Expected value function
A given policy
Trinity University | Laboratory for Distributed Intelligent Agent Systems
18
Outline
•
•
•
•
•
•
Background and Objective
CASE — Agent-Level Solution
CASE — Society-Level Solution
Experiment
A Case Study
Conclusion and Future Work
Trinity University | Laboratory for Distributed Intelligent Agent Systems
19
Our Approach
Cognitive Agents for Social Environments
Network
 Social
 Bounded
Convention
Rationality
 Perception
 Action
Environment
Trinity University | Laboratory for Distributed Intelligent Agent Systems
20
Social Convention
A social law is a restriction on the set
of actions available to agents.
A social convention is a social law
that restricts the agent’s behavior to
one particular action.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
Hard-Wired Design vs. Emergent
Design
Hard-wired design means that social
conventions are given to agents off-line
before the simulation.
Emergent design is a run time solution
that agents decide the most suitable
conventions giving the current state of
the system.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
21
Generating Social Conventions:
Existing Rules
Highest Cumulative Reward
•An agent switches to a new
action if the total payoff from
that action is higher than the
payoff obtained from the
currently-chosen action.
Simple Majority
•An agent switches to a new
action if they have observed
more instance of it in other
agents than the present action.
•Not rely on global statistics
about the system.
•Rely on global statistics about
the system.
•Guaranteeing convergence in
a 2-person 2-choice
symmetric coordination game.
•Convergence has not been
proved.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
22
Generating Social Conventions:
Our Rule
Generalized Simple Majority
Definition. Assume an agent has K neighbors and
that KA neighbors are in state A. If the agent is in
state B, it will change to state A with probability
f  (K A ) 
1
1 e
2  ( 2 K A / K 1)
.
Theorem. When →, change to state A when
more than K/2 neighbors are in state A, in a 2person 2-choice symmetric coordination game.
Trinity University | Laboratory for Distributed Intelligent Agent Systems
23
24
Outline
•
•
•
•
Background and Objective
CASE — Agent-Level Solution
CASE — Society-Level Solution
Experiment
– Evaluating the Agent-Level Solution
– Evaluating the Society-Level Solution
• A Case Study
• Conclusion and Future Work
Trinity University | Laboratory for Distributed Intelligent Agent Systems
25
Evaluating the Agent-Level Solution
The Ultimatum Game
The Bargaining Game
Agent a
I'll take x,
you get 10x
a gets x,
b gets 10x
Both get 0
Accept Agent b
I'll take x,
you get 10x
a gets x, Accept
b gets 10x
Negotiate I'll take y,
you get
10y
Reject
Reject or
Run out of steps
Both get 0
Trinity University | Laboratory for Distributed Intelligent Agent Systems
26
Phase I - Editing
• Framing
– 11 states: $0, $1, …, $10
• Anchoring
– Use 500 iterations of Q-learning to develop anchored states
• Accessibility
– st ~ sm if distance * (st , sm )  0
c, I
Trinity University | Laboratory for Distributed Intelligent Agent Systems
27
Q-Learning
•
•
•
•
Well studied reinforcement learning algorithm
Converges to optimal decision policy
Works in unknown environments
Estimates long-term reward from experience
expected discounted reward
old value
Q( st , at )  Q( st , at )    [rewardt   max Q( st 1 , a)  Q( st , at )]
a
max future value
old value
learning rate discount factor
Trinity University | Laboratory for Distributed Intelligent Agent Systems
Phase II - Evaluation
• 1000
iterations of
play with
intuitive or
deliberative
decisions
Trinity University | Laboratory for Distributed Intelligent Agent Systems
28
29
Results of the Ultimatum Game
Human
Players
CASE Agents’ Results
Rational
Players
Number Time Accepted
Human Players’
Results
&
Rational
Players’ Results
Two-Phase
Intuition Only
Deliberation Only
Split Value
Trinity University | Laboratory for Distributed Intelligent Agent Systems
30
Results of the Bargaining Game
Split Value
10
8
6
4
2
0
Iteration
Negotiation Size
Human Players’ Results
Iteration
Split Value
Negotiation Size
CASE Agents’ Results
Iteration
Iteration
Results of the Bargaining Game by Human Players are with kind permission of Springer Science
Trinity University | Laboratory for Distributed Intelligent Agent Systems
31
Outline
•
•
•
•
Background and Objective
CASE — Agent-Level Solution
CASE — Society-Level Solution
Experiment
– Evaluating the Agent-Level Solution
– Evaluating the Society-Level Solution
• A Case Study
• Conclusion and Future Work
Trinity University | Laboratory for Distributed Intelligent Agent Systems
32
Evaluating the Society-Level Solution
2-person 2-choice symmetric coordination Game
A
B
A
1
-1
B
-1
1
Two Optimal Decisions: (A,A) and (B,B)
Trinity University | Laboratory for Distributed Intelligent Agent Systems
33
Evaluating the Society-Level Solution
• Intuitive and deliberative decisions
• N agents (N2) with random initial state, A or B,
with probability 50%
• Agents connected by classic networks or complex
networks
• Evaluating two rules
– Highest cumulative reward (HCR)
– Generalized simple majority (GSM)
• Performance measure: T90%
– The time it takes that 90% of the agents use the same
convention
Trinity University | Laboratory for Distributed Intelligent Agent Systems
34
Classic Networks
Complete Network KN
Lattice Ring CN,K
Random Network RN,P
• Nodes fully
connected to each
other
• Nodes fully
connected to its K
neighbors
• Local clustering
•Nodes connected with
equal probability
N=8
N=100 K=6
N=100 P=5%
Trinity University | Laboratory for Distributed Intelligent Agent Systems
35
Complex Networks
Small-World Network WN,K,P
•Start with a CN,K graph and
rewire every link at random
with P
•Local clustering & randomness
N=100 K=6 P=5%
Scale-Free Network SN,K,
•P(K) is a power law
P(K) ~ K-
•Large networks can self-organize
into a scale free state,
independent of the agents
N=100 K=6 =2.5
Trinity University | Laboratory for Distributed Intelligent Agent Systems
36
Evaluating Highest Cumulative Reward
Network Topology
SN
CN
KN
Name
Scale-free network
Lattice ring
Complete network
Size  103 (N)
1, 2.5, 5, 7.5, 10, 25, 50
0.1, 0.25, 0.5, 0.75, 1
1, 2.5, 5, 7.5, 10, 25, 50
Parameter
=2.5 <K>=12
K=12
None needed
WN
Small-world network
1, 2.5, 5, 7.5, 10
P=0.1 <K>=12
Lattice ring
T90%=O(N2.5)
Small-world
T90%=O(N1.5)
Scale-free/
Complete
T90%=O(NlogN)
Trinity University | Laboratory for Distributed Intelligent Agent Systems
37
Evaluating Generalized Simple Majority
Network Topology
CN
KN
SN
Name
Lattice ring
Complete network
Scale-free network
Size  103 (N)
0.1, 0.25, 0.5, 0.75
1, 2.5, 5, 7.5, 10, 25, 50, 75, 100
1, 2.5, 5, 7.5, 10, 25, 50, 75, 100
Parameter
K=12
None needed
=2.5 <K>=12
SN
Scale-free network
1, 2.5, 5, 7.5, 10, 25, 50, 75, 100
=3.0 <K>=12
WN
Small-world network
1, 2.5, 5, 7.5, 10, 25, 50
P=0.1 <K>=12
Lattice ring
T90%=O(N2.5)
Small-world
T90%=O(N1.5)
Scale-free/
Complete
T90%=O(N)
Trinity University | Laboratory for Distributed Intelligent Agent Systems
38
Evaluating HCR vs. GSM
Network Topology
Small-World Network
Lattice ring
N
104
<K>
12
P
P=0.05, 0.09 … 0.9 (P=0.09)
Random network
Trinity University | Laboratory for Distributed Intelligent Agent Systems