* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
From: AAAI-93 Proceedings. Copyright © 1993, AAAI (www.aaai.org). All rights reserved. University of Michigan Ann Arbor, MI 48109 durfee/[email protected] A rational agent in a multiagent world must decide on its actions based on the decisions it expects others to make, but it might believe that they in turn might be basing decisions on what they believe the initial agent will decide. Such reciprocal rationality leads to a nesting of models that can potentially become intractable. To solve such problems, game theory has developed techniques for discovering rational, equilibrium solutions, and AI has developed computational, recursive methods. These different approaches can involve different solution concepts. For example, the Recursive Modeling Method (RMM) finds different solutions than game-theoretic methods when solving problems that require mixed-strategy equilibrium solutions. In this paper, we show that a crucial difference between the approaches is that RMM employs a solution concept that is overeager. This eagerness can be reduced by introducing into RMM second-order knowledge about what it knows, in the form of a flexible function for mapping relative expected utility of an option into the probability that the agent will pursue that option. This modified solution concept can allow RMM to derive the same mixed equilibrium solutions as game-theory, and thus helps us delineate the types of knowledge that lead to alternative solution concepts. Rational decisionmaking in a multiagent context is a difficult problem when an agent has knowledge that leads it to view other agents as being rational as well. With such reciprocaZ rationality, an agent must decide on its action(s) given what it believes the rational agents around it will do, but inferring that requires it to infer what each of those agents will believe it (and ‘This research was sponsored in part by the NSF under grants IRI-9015423 and IRI-9158473, by DARPA under contract DAAE-07-92-C-R012, and by the Golda Meir Fellowship and the Alfassa Fund administered by the Hebrew University of Jerusalem. Jerusalem, Israel [email protected] the others) will do, which in turn requires it to assess what each of those agents will believe about what each of the agents will believe the others will do, and so on. Reciprocal rationality can thus lead to an indefinite (and theoretically infinite) number of levels of recursive modeling. In the Recursive Modeling Method ( for example, we have developed a procedure whereby agents can build this recursive nesting of models and can use the resultant modeling hierarchy to infer rational courses of action for others and for themselves [Gmytrasiewicz et al., 19911. RMM provides a rigorous foundation for such decisionmaking situations, based on RMM’s concept of a solution. Rowever, as we show in this paper, RMM’s original formulation leads to decisions that differ from those that would be derived by traditional game-theoretic methods, because those methods employ a different solution concept. In addition, the original solution concept in RMM could lead to cases where the algorithm that uses the recursive models could arrive at different results depending on the number of recursive levels examined. In our original RMM, we described how such behavior must eventually become cyclic given finite knowledge, and that when such a cycle occurs RMM can probabilistically mix the results to arrive at a single, overall decision. While this is the best that original RMM can do, the question arises as to whether a different solution concept could avoid such cyclic behavior, and could converge more naturally on a single solution. Moreover, the assumption that RMM must run out of knowledge at some finite level might be overly restrictive for some applications, and a solution concept that clearly defines the behavior of RMM in the limit of infinity would be desirable. In this paper, we suggest these characteristics of MM stem from the fact that its solution concept leads to what we call “overeager reciprocal rationality.” In a nutshell, our argument is that rationality based on expected payoffs given probabilistic models of what others will do should be tempered by the degree of confidence in those models. We describe one way of introducing this additional knowledge into RMM by using a more flexible function for generating those istributed Problem Solving 225 /. probabilistic models. Not only can this avoid overeager rationality that leads to oscillations, but that in fact it can make RMM’s results converge in the limit of infinite recursion. More broadly, however, the contributions of this paper extend beyond RMM to examine the nature of game-theoretic rationality as employed in a multiagent reasoning systems [Rosenschein and Breese, 1989; Rosenschein and Genesereth, 1985; Rosenschein et al., 1986; Zlotkin and Rosenschein, 19911. In particular, the concepts of equilibria and mixed-strategies play a central (although often debated) role in the gametheoretic literature [Shubik, 19821. In this paper, we show how the recursive algorithm that RMM employs to model reciprocal rationality and the more traditional equilibria solutions can converge, given a particular choice of solution concept. We begin by defining the game-theoretic notions of equilibria and mixed strategies, showing how rationality is embodied in these models. We then look at original RMM’s solution to reciprocal rationality problems and its characteristics. Then we suggest a less eager rationality assumption, embodied in a function for computing probabilities over agents’ moves, and describe how introducing it into the original RMM formulation avoids overeager rationality. We conclude by analyzing the performance of our approach and highlighting important open problems. Game Theoretic Game theoreticians have developed a number of techniques for determining rational combinations of moves for games represented in normal (or strategic) form, where each combination of strategies chosen by the players leads to a particular payoff for each player. The most common solution method involves using (iterated) dominance techniques [Rasmusen, 1989; Shubik, 19821, where players remove from consideration any moves that are clearly inferior (dominated) no matter which of the non-dominated moves others take. By alternating between the players’ viewpoints, the number of rational moves can be decreased; in the ca+se where there is a single rational strategy for each player, it can be found. The combination of rational strategies for the players represents an equilibrium solution, since none of the players has an incentive to deviate unilaterally from its own strategy. In cases where there are multiple equilibrium moves, converging on a single move (a pure strategy) for each player is more complicated. For example, consider the game summarized in Figure la, where each combination of moves leads to a matrix entry giving a payoff for P (lower left) and Q (upper right). In this game, there are two equally good moves for both P and Q. They could each maximize their payoff at 2 by either choosing options ad or bc. The trouble is, which of these will they choose? One way would be to have P and Q communicate and agree on one of the 226 Durfee c a P lzlb d 0 0 2 2 2211 (a) with 2 pure solutions Figure 1: Matrices (b) with no pure solutions for Example Games two possible combinations. Or if they shared common knowledge about how to break such ties, they could employ that knowledge here. But more generally, P and Q cannot be assured of converging for certain on one joint course of action. Game theory says that a mixed strategy equilibrium might thus be a useful solution concept here, where agents adopt probabilistic combinations of the separate pure strategies. One possible derivation for such a mixed strategy, patterned after [Rasmusen, 19891, is the following: Assume that P and Q will adopt mixed strategies (pa pb) and (pC pd), respectively. ’ The expected payoff of P (recall that Pa + Pb = pc + pd = 1) is then: JwaYofbl = 2PaPd = Pa +Pc + 2pbpc -3PaPc + PbPd + 1. Differentiating the above with respect to pa, and postulating that the result be zero (for the maximization problem), allows us to conclude that in the mixed strategy equilibrium Q must select move c with probability p, = l/3. That is, it is assumed that a mixed strategy for P is optimal, and P will only adopt a mixed strategy if pc = l/3 (if higher, P will always choose b, and lower leads to a). By the same arguments’, the strategy for player P is pa = l/3. So, with Pa = Pc = l/3, P and Q would expect payoffs of 4/3. Mixed strategies also play a role in games with no equilibrium solutions, such as the game in Figure lb. In this game, the mixed strategy equilibrium solution has player P choose a with probability l/2 and b with probability l/2, while player Q chooses c with proba2/3 [Brandenburger, bility l/3 and d with probability 19921. As Brandenburger points out, the concept of mixed strategy, where agents choose probabilistically among possibilities, is somewhat troubling in this kind of game because, for example, if Q believes that P is playing the expected mixed strategy of (l/2 l/2), then there is no incentive for Q to play his mixed strategy ‘In the notation in the rest of this paper, a mixture over strategies, represented as (pa pb), indicates that the option listed first in the matrix has probability p, and the option listed second has probability pb. of (l/3 2/3) over any other mixed (or pure) strategy. Brandenburger cites a history of work involved in viewing mixed strategies not as probabilistic strategy selection by agents playing a game, but instead as probabilistic conjectures that agents have about the pure strategies of others. It is this viewpoint that RMM takes as well. ecwsive Analyses of normal form games, as described above, can employ various methods, including iterated dominance and adopting assumptions about the agents and solving for optimal mixed strategies. Different analyses might use different solution concepts, and thus (as seen above) different decisions can be rational in the context of different solution concepts. In the Recursive Modeling Method (RMM), our goal has been to develop a general, algorithmic solution to the problem of recursive reciprocal reasoning that generates the recursion explicitly. In RMM, one agent represents how it thinks another sees the situation by hypothesizing the game matrix (or matrices) that the other agent is seeing. It can also hypothesize the matrices that the other agent might think the first agent sees, and so on, building a hierarchy of models representing, to increasing depths, the view of how the agent thinks that agents think that agents think . . . that agents perceive the situation. Beginning at the leaf node(s) of this hierarchy, probabilities over agents’ choices from among their options can be propagated upward until, at the top, the agent trying to decide what to do has a more informed model of what others are likely to do. Note that RMM works by assigning an equiprobability distribution among the options at the leaf nodes, corresponding not to a belief that an agent will flip a coin to decide on its strategy, but corresponding rather to the fact that it does not know how the agent will decide on its strategy, leading to the equiprobabilistic distribution that contains no information. RMM then chooses the option(s) with the maximum expected utility at each successive level above. RMM helps agents converge on informed rational For example, in situachoices in many situations. tions with single equilibrium solutions, RMM easily converges on solutions computed with iterated dominance. However, in situations without such clear solutions, RMM often must probabilistically mix solutions. Recall the example with two pure equilibrium solutions (Figure la). In RMM, the tugging between two solutions leads to oscillating views from level to level. From P’s perspective: If he does not consider what Q will prefer to do, then he will choose option b since it has the higher average payoff. If he considers Q but does not consider how Q considers P, then he will infer that Q will choose option d (highest average) and so P should choose a. If he considers Q and how Q considers P but no deeper, then he will infer that Q will think P will choose b, so that Q will choose c, so that P should in fact choose b. This oscillating of P deciding on b and then a continues no matter how deep in the hierarchy P searches: If P elaborates the recursion an even number of times, he will prefer b (and expect a payoff of 2 since he expects Q to take move c), while he will prefer a (and again expect a payoff of 2) if he elaborates the recursion an odd number of times. Wow do we reconcile this oscillation? Well, what does is to probabilistically mix the derived expectations, and to work from there. So in the example above, when P runs RMM, it will recognize that half the time it expects Q to take action c, and half the time it expects Q to take d. It ascribes a strategy of that, if Q is equally (112 V) to Q, and determines likely to take either action, P should take action b (for an expected payoff of 1.5) rather than action a (which has an expected payoff of 1). Q, when it runs RMM, will go through the same reasoning to decide that it should take action d. Thus, each agent will have decided on a single action, and expect a payoff of 1.5; as external observers, however, we can clearly see that their true payoffs will really be 1. Had they instead each derived the mixed strategy of (l/3 2/3), however, we know that they could each expect a payoff of 1.33. Our derivation of this mixed strategy in the previous section assumed additional knowledge that allowed an agent to adopt a mixed strategy based on the understanding that the other agent would be adopting a mixed strategy as well. As we show below, we can incorporate addit,ional knowledge within RMM to change its solution concept such that agents using RMM can derive this mixed strategy. First, however, let us also revisit the case with no equilibrium solutions (Figure lb). Were, the results of RMM for successive depths of the recursion will cycle among the four possible joint moves indefinitely. P would believe that Q is equally likely to take action c or d, and so P would choose action a with an expected payoff of 1. Q, on the other hand, would see P as equally likely to take action a or b, and so Q would be indifferent among its choices, adopting a mixed strategy of (l/2 l/2). Th is d i ff ers from the mixed strategies of (l/2 l/2) for P and (l/3 l/3) for Q derived game theoretically in the previous section. In summary, RMM’s decisions mirror the gametheoretic solutions when the choice is clear, but when several choices are equally rational depending on where one starts in the hierarchy, then RMM treats the possible strategies conjectured for other agents as equally likely. This sometimes leads RMM to adopt strategies that differ from game-theoretic solutions, as we have seen. As we explain next, the reason why RMM makes different conclusions than game theory is because RMM’s solution concept does not consider second-order knowledge, and thus RMM’s rationality is overeager. Distributed Problem Solving 227 J 1 OL 0 0; 0 -J 0 dative expected payoff 05 &live expected payelf (4 04 Figure 2: Functions Mapping Overeager Reci We argue here that RMM’s solution concept commits too strongly and quickly to the its conjectures-the probabilistic mixtures over options. At the leaves, RMM assigns an equiprobable distribution over the options of the other agent(s). Given this distribution, however, RMM immediately considers desirable only the options that will maximize expected payoffs based on this equiprobable distribution. Because the initial distribution was based on ignorance, it seems premature to place such weight in it, given the opportunity to bring more knowledge to bear at the higher levels of the hierarchy. In other words, RMM applies the utility maximization concept of rationality too eagerly. Let us look at the probability calculations given relative expected payoffs, graphed for the simple case of two options, shown in Figure 2. At the leaf nodes, we have a flat probability function (Figure 2a), meaning that all options have equal probability because RMM has no knowledge about the relative expected payoffs below the leaf nodes. Above the leaf nodes, we have a step probability function (Figure 2b), which places certainty in the option with the higher expected payoff. Clearly, these are two extremes of a more general mapping function of relative expected payoff to probability. Consider the more general function to compute probability of option i, given its expected payoff relative to the payoffs of all of the options J (where all payoffs are assumed non-negative): pi = Payoff(i)” /x Payoff(j)” (1) jEJ In this function, when k: is 0 we get the extreme of a flat probability function, while as it: approaches 00, the function approaches the step function extreme. When k: is 1, the function specifies a linear relationship between payoffs and probabilities-that if option a has twice the payoff of option b, for example, then it is 228 Durfee 4 1 (4 Relative Payoff to Probability. twice as likely to be chosen. The function is graphed for the simple case with two options in Figure 2c. This function can be incorporated simply into the RMM calculations, provided we can specify a choice for L. The choice of k: represents knowledge, and, more importantly, how k changes at different depths of the hierarchy corresponds to second-order knowledge about probabilities at each level. In the original RMM, k was implicitly specified as 0 for computing the probability distribution feeding into the leaves of the hierarchy, and 00 at all levels above. This abrupt change, however, is what we claim makes the original solution concept in RMM overeager. Instead, k should be 0 at the leaves and become successively larger as one moves up the hierarchy, because k represents knowledge about the certainty RMM should place in conjectures about strategies. Toward the leaves, RMM should only lean slightly toward favoring options that do well given the conjectures at those levels, because those conjectures are based on ignorance. As we work toward the root of the hierarchy, the probabilistic conjectures are based on more and more information (propagated from the levels below), and thus RMM should commit more heavily at these levels. We can think of this approach as a variant of simulated annealing, where early in the process (near the leaves) the algorithm biases the search for rational strategies slightly but still keeps its search options open. As it moves up the hierarchy, the algorithm becomes increasingly committed to better options based on the evidence accumulated from below. Besides avoiding overeager rationality, this modification also provides a new approach to dealing with the possibility of infinite recursion in RMM. As RMM recurses increasingly deeper, k gets closer and closer to 0, and the influence of deeper levels of knowledge (about what I know about what you know about what I know.. .) diminishes. In a practical sense, there will be a finite level at which a computing system will lack (defvar *power-reduction-rate* .8) (defun simple-rmm (matrix1 matrix2 levels &optional (power 1)) (let* ((column-probs (if (= levels 0) (make-equiprobability-vector (length (first matrixl))) (simple-r-mmmatrix2 matrix1 (l- levels) (modify-power power levels)) )) (rows-exp-utils (mapcar #* (lambda (row) (compute-expected-utility row column-probs)) matrixl))) (mapcar #'(lambda (utility-for-row) (compute-probability utility-for-row rows-exp-utils power)) rows-exp-utils))) (defun compute-expected-utility (payoffs probs) (cond ((null payoffs) 0) (t (+ (* (first payoffs) (first probs)) (compute-expected-utility (rest payoffs) (rest probs)))))) (defun compute-probability (payoff all-payoffs power) (let ((prob (if power (/ (expt payoff power) (float (let ((current-sum 0)) (dolist (a-payoff all-payoffs current-sum) (setf current-sum (+ current-sum (expt a-payoff power)))) current-sum))) ; else, nil power means assume original RMM formulation (let ((max-payoff (apply I'max all-payoffs))) (if (= max-payoff payoff) (float (/ 1 (count max-payoff all-payoffs))) 0.0))))) (if (<= prob l.Oe-6) 0.0 prob))) (defun modify-power (power level) ; this version ignores the level.... (when power (* power *power-reduction-rate*))) Figure 3: Code Fragment For Simple sufficient resolution to distinguish deeper levels of the hierarchy, while in a theoretical sense, RMM is wellbehaved as it recurses toward infinite levels. Thus, rather than appealing to arguments of finite amounts of knowledge bounding any recursion, we can instead clearly define the behavior of RMM as it moves toward infinite recursive levels. While it is clear that, in the modified RMM probability calculation, k: should approach 0 as more levels are explored, it is less clear what value of k makes sense as we approach the root of the hierarchy. Driving k toward higher values will cause RMM to “lean” increasingly heavily as it goes upward, until it leans hard enough to commit to a specific choice. This is desirable for problems with single equilibrium points, but, when mixed strategies are most rational, having values of k too high will lead to oscillations just like in the unmodified RMM. The remaining questions, therefore, are whether a proper selection of k can lead to appropriate mixed strategies, and if so, how is that selection done. ixed Strategies T Key functions in a much simplified version of RMM, which does not consider possible horizontal branching representing uncertainty about alternative payoff matrices other agents might subscribe to (see [Gmytrasiewicz et al., 1991]), are shown in Figure 3. Note that this example implementation uses a very simple method to change the value of k at successively lower levels of the hierarchy: it multiplies the value of k at the previous level by a constant (less than 1). This approach allows the algorithm to asymptotically approach 0 toward the leaves assuming sufficient recursive levels and an initial value of k that is not too large. RMM Implementation To see how modifying RMM affects its performance, we begin with the example having two equilibria in Figure la. Our game-theoretic analysis determined the optimal mixed strategy would be for P and Q to each select its first option with probability l/3, and its second option with probability 2/3. Recall, however, that this analysis was based on knowledge that P and Q both assume that the other was adopting a mixed strategy. The modified RMM algorithm outlined in Figure 3 does not assume this knowledge. In Figure 4 we show the probability that P (and Q since the game is symmetric) assigns to its first option derived by modified RMM for runs involving 100 levels of recursion on incrementally larger values of k. The probability of the second option is simply 1 minus that of the first option. As expected, when k is 0 throughout, the equiprobable mixed strategy is returned. As k increases, however, note that a player will adopt a mixed strategy that approaches the solution computed gametheoretically as (l/3 2/3). Beyond a certain value of k, however, modified RMM diverges from this solution because its solution concept leans so heavily toward particular options even if they are only marginally better, forcing the system into the oscillatory behavior seen in the original RMM formulation. The other problematic case in which there were no equilibrium solutions (Figure lb) provides similar results. That is, as k increases, the mixed-strategy solutions derived by RMM approach those derived gametheoretically (P playing (l/2 l/2) and Q (l/3 2/3)), and then begin to diverge and oscillate. The value of k at which convergence ceases differs between this problem and the previous problem, and as yet we have no method to predict ahead of time the value of k that will Distributed Problem Solving 229 next step is to examine more precisely the second-order knowledge captured in the new mapping function, using that understanding to analytically derive parameter settings (values of k) that will lead to the optimal converging mixed-strategy equilibria, and possibly embodying that knowledge more explicitly in the RMM algorithm by propagating probabilities about probabilities through the RMM hierarchy. Beyond this, however, is the ongoing challenge of characterizing the different rational solution concepts, so that developers of rational autonomous systems understand the strengths and limitations of the solution concepts that they implement . 0.9 0.8 0.7 0.6 0.5 0.4 1tJ 0.2 eferences 0.1 0 5 10 15 20 25 k Figure 4: First Option Probability: 2-Equilibria Game. lead to convergence. Our current formulation thus uses an iterative method: assuming that the new mapping function captures the correct knowledge when modified RMM converges, our formulation increases the value of k until divergence begins. Conclusions Our results illustrate how adding additional knowledge about the uncertainty in probabilities over other agents’ strategies can lead to a different solution concept, causing reciprocal rationality to be less eager. Under appropriate conditions, this new solution concept can converge on mixed equilibrium solutions that can lead to better actual performance than the solutions derived by the original RMM solution concept. More importantly, the modified solution concept allows us to bring together the equilibrium solution models common in game-theoretic approaches with the recursive modeling approach embodied in RMM. By including the possibility of representing second-order knowledge about the probabilistic conjectures about strategies, we can implement solution concepts ranging from the sometimes overeager rationality of the original RMM, to the less eager solution concepts that approximate those used in game-theoretic methods, all the way to completely indecisive strategies (when k = 0). More generally, our results in this paper serve to underscore how the results of rational decisionmaking are dependent upon the underlying solution concepts and their associated knowledge, even when following the basic concept of maximizing expected utility. Much work along these lines remains to be done, however. With regard to the work presented in this paper, a clear 230 Durfee Brandenburger, rium in games. 6(4):83-101. Adam 1992. Knowledge and equilib- The Journal of Economic Perspectives Gmytrasiewicz, Piotr J.; Durfee, Edmund H.; and Wehe, David K. 1991. A decision-theoretic approach to coordinating multiagent interactions. In Proceed- ings of the Twelfth International Joint Conference on Artijcial Intelligence. Rasmusen, Eric 1989. Games andInformation, an Introduction to Game Theory. Basil Blackwell. Rosenschein, Jeffrey S. and Breese, John S. 1989. Communication-free interactions among rational agents: A probablistic approach. In Gasser, Les and Huhns, Michael N., editors 1989, Distributed Artificial Intelligence, volume 2 of Research Notes in Artificial Intelligence. Pitman. 99-l 18. Rosenschein, Jeffrey S. and Genesereth, Michael R. 1985. Deals among rational agents. In Proceedings of the Ninth International Joint Conference on Arti$cial Intelligence, Los Angeles, California. 91-99. (Also published in Readings in Distributed Artificial Intelligence, Alan H. Bond and Les Gasser, Morgan Kaufmann, 1988.). editors, pages 227-234, Rosenschein, Jeffrey S.; Ginsberg, Matthew L.; and Genesereth, Michael R. 1986. Cooperation without communication. In Proceedings of the Fifth National Conference on Artificial Intelligence, Philadelphia, Pennsylvania. 5 l-57. Shubik, Martin 1982. Game Theory in the Social Sciences: Concepts and Solutions. MIT Press. Zlotkin, Gilad and Rosenschein, Jeffrey S. 1991. Cooperation and conhict resolution via negotiation among autonomous agents in non-cooperative domains. IEEE Transactions on Systems, Man, and Cybernetics 21(6). (Sp ecial Issue on Distributed AI).