Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Evolving Neural NPCs with Layered Influence Map in the Real-time Simulation Game ‘Conqueror’ Su-Hyung Jang, Student Member, IEEE and Sung-Bae Cho, Senior Member, IEEE making strategies. The IM function could be a summation of the natural resources present in that square, the distance to the closest enemy, or the number of friendly units in the vicinity [3]. Abstract—AI in computer games has been highlighted recently and the importance of game platforms that support the investigation of various kinds of AI techniques increases. In this paper, we developed a real-time simulation game called ‘Conqueror’ for providing with a better AI game platform to many AI researchers and applied it to produce a neural NPCs with layered influence maps. The layered influence map algorithm is used to analyze the situation in terms of distribution of influences between friends and enemies, while the neural network is adopted as a basic representation of the NPC. The experiment has been conducted with various factors and verified the usefulness of the proposed game platform. II. BACKGROUND A. Related Work The part of interpretation of the current situation is very important in order to generate the optimal strategy for realtime strategy games. There are various techniques for situation analysis. C. Ong [4] attempted to evaluate situations of Chinese Chess by expressing the location of the horses on the whole map with numbers. This method enables to identify all the factors making influences on the game by reading the whole situation. This is appropriate for a method that each object has no autonomy and that a small number of object adjustments is carried out with central control. However, for a case that the influence one object makes on the whole game is small as there are many objects in the game, the strategies to be taken by the object find more chances that the information of the whole game is not necessary, and rather unnecessary information may force excessive computation resources and interrupt with an optimal strategy adoption. S. Lucas [5] used the situation information of the map in the Pac-Man game after selectively collecting and processing them. An expert system should be utilized to effectively use data . As more parts are tuned at the development stage, it leads to more overhead and trials and errors become mandatory in order to find effective encoding. I. INTRODUCTION Inexpensive yet powerful computer hardware and advanced graphics technique have made it possible to enlarge computer game genre. One of them is real-time strategy simulation games. A good artificial intelligence (AI) on RTS-games means a series of interesting decisions. But most of the research focused on games that can be described in a compact form using symbolic representations such as board and card games. Recently, many AI technologies are applied to design NPC’s behaviors of the RTS-game. One of them is Neural Evolution (NE) [1]. This approach is particulaly well-suited for video games. NE works well in the high dimensional space, diverse population can be maintained as individual networks behave consistently, adaptation can take place in real time, and memory can be implemented through recurrency [2]. RTS games include Starcraft, Dawn of War, Supreme Ruler, Earth 2160, and Age of Empires. Players are given cities, armies, buildings, and abstract resources such as money, gold, and saltpeter. They play by both allocating these resources to produce more units and buildings, and by assigning objectives and commands to their units. Units carry out player orders automatically, and the game is usually resolved with the destruction of other players’ assets [3]. In the paper, we propose a strategy generation method using influence map. An influence map (IM) is a grid placed over the world, which has values assigned to each square based on some functions which represent a spatial feature or concept. Influence maps evolved out of work done on spatial reasoning within the game of Go and have been used sporadically since then in games such as Age of Empires. Influence maps combine together to form spatial decision B. Influence Map Interpretation of the current situation is very important in order to draw out the optimal strategies for real-time strategy games. IM shows overall reverse relationships of the game and is useful for strategic evaluation and decision-making based on the current game status. An Influence map is comprised of multiple layers superposed on the geographic expression of the game map as shown in Figure 1, and each layer represents various variables of the game. When an agent carries out a certain decision-making, the whole or parts of these layers are put together along with appropriate weights to make into a map. Values constituting the map indicate an approximate degree of suitability of the relevant position of the game map for decision making at present [3]. Sung-Bae Cho is with the Department of Computer Science. Yonsei University, 262 Seoungsanno, Sudaemoon-ku Seoul 120-749, Korea(phone +82-2-2123-3877 ;email : [email protected]) Sung-Bae Cho is with the Department of Computer Science. Yonsei University, 262 Seoungsanno, Sudaemoon-ku Seoul 120-749, Korea(phone +82-2-2123-2720 ; email : [email protected]) 978-1-4244-2974-5/08/$25.00 ©2008 IEEE 385 Fig. 1. Inffluence Map I III. THE GAM ME: ‘CONQUERO OR’ ‘Conqueror’’ developed in n this researcch is a real-tiime sstrategic simulaation game, in which two nattions expand thheir o own territory and take awaay the other. Each nation has h s soldiers who inndividually bu uild towns and fight against the e enemies, whilee a town contiinually producces soldiers foor a g given period. The motivatio on of this gam me is to obseerve m multi-agents’ b behaviors and to t demonstratee AIs in compuuter g games, especially the usefuln ness of evolutioonary approachhes. F Figure 2 showss a snapshot of o ‘Conqueror.’ Following ruules s summerize the game. 1) Resourcces given to eaach country at the beginningg of the gam me are one soldier and one village. 2) Energy increases for the t village as time goes by unntil the ennergy reaches up to 100, annd one soldierr is producced and the eneergy is initializzed. 3) Produceed soldiers may y create other villages, destrruct villagees of the counteerpart, or makee combats agaiinst the couunterpart soldiers. 4) Energy is given to the t soldiers buut decreases as a a villagee is created or destructed, or a combat is maade and as time paasses, the energy increaases automatically. 5) Combatts can be madee in alliance wiith other soldieers, and thhe energy of th he soldiers in alliance a equalss to the tottal sum of each h soldier's energgy. 6) Structurres such as paalisades can reduce the fightting strengtth of the countterpart. Fig. 2. A Snapshot of o the Developed Game G 386 The size s of the mapp can be adjustted from 7 × 7 to 30 × 30 accordinng to the requeest of users and the high and loow for each positionn can be deciided. Each cooordinate is ann exclusive space where w two unitts cannot enterr simultaneously and two buildinggs cannot be constructed at a the same time. t NPC supportts two types of o characters and a each charracter may generate a high-level of emergent behaviors b suchh as attack, ment to both sides, construuction of thee building, movem destructtion of a buildiing, merger, etcc. The Repository R mannages informattion by modulaating all the informaation on the game, agent, map, etc. neecessary to progresss the game. For F increasing the reality off the game, various interaction rulles are designedd. Those rules are applied u period, andd each rule hass the effect on action and every unit reactionn for not causinng a super strateegy. Scores aree calculated by seveeral measures suuch as buildingg Towns scoress, attacking scores, producing Units scores, andd the summatioon of them. t are diveerse measures to evaluate a strategy, Since there various types of strateegies might be explored by evvolutionary approacches. IV. NEURAL NPC WITH W LIM As shown in Figuure 3, all the agents withinn the game me and each derived d the identifieed the situatioon of the gam optimizzed strategies by utilizing the proposeed layered influencce map (LIM) technique. t All the agents creaated within the gam me decide thhe optimal beehaviors among attack, construction, destructiion, evasion, annd merger throough neural networkk utilizing LIM and makke movementss. Layered informaation used for LIM at this time t is of buiildings and troops. The weight off neural networrk used here was w directly i genes. Thhus created genes g were encodedd and made into consisteently evaluateed using geneetic algorithm m to select excellennt groups, crosssover, repeat processes of addition a of mutatioons, and grradually explore neural networks commannding strategiees with excellennt evaluation. Fig. 3. Behavior B Decisionn Flowchart using Neural N Network annd Influence Map 2008 IEEE Symposium on Computational Intelligence and Games (CIG'08) Weighting and summing the influence layers to attain a final set of desirability value is advantageous as it is a relatively simple approach. Moreover, it is fully transparent in that the developer knows exactly how and why it makes its decision, as these parameters have been set by hand. However, this method of calculating the desirability value has certain limitations. First, the developer needs to choose which layers are relevant to the decision that is being made. This might seem like a simple task, but it is often difficult to know exactly which factors an expert is considering when making a decision, such as an expert game player choosing which location to attack. Therefore, the process of choosing the relevant variables that need to be considered for the decision is a matter of trial and error. As a result, the process is time consuming and might mean that useful and important information is left out. Second, when the chosen layers are summed together, it is possible that important information might be lost. For example, consider a situation in which the AI has units in a certain cell that are adding a positive influence, while the human player has units in the same cell that are adding a negative influence. When these opposing influences are added together they cancel each other out, and it seems as though there are no units adding influence in that cell. However, the information that both forces have units in this cell could be quite important to a strategic decision, but by simply adding them together it can go unnoticed. Third, finding the correct weighting for each layer for each decision is also a matter of trial and error, and as such can require a great deal of tweaking to get right. The only way to find a suitable set of weight is to guess initial values and then hand-tune until the AI seems to be behaving reasonably [6]. In order to overcome such limits, in this paper, behavior evaluation is conducted for a case of considering each variable only by separating situational variables, and behavior strategies that would satisfy all the variables as much as possible after considering weights by variables of values evaluated are sought after. This is because the weight is not identical in terms that all the situations have influences on strategies selection. In this paper, we adopt the evolutionary method using the genetic algorithm to generate behavior systems that are accommodated to several environments. The chromosome is encoded as a string that represents a weight of neural network that determine behavior of agent as shown in Figure. 4. The crossover, mutation operations and the roulette wheel selection are used to reproduce a new population. The crossover and mutation of neural networks are shown in Figure. 5. In crossover, we first choose two individual and replace those parts of them. In mutation, we select a part of the network and replace with new values. As repeating this process, we can get improved strategy of behaviors. Fig. 4. Chromosome of NN Encoding Fig. 5. Genetic Operation V. EXPERIMENT AND RESULT A. Experiment Operator used for the genetic algorithm was designed with 50 individuals, 0.7 of cross-over rate, and 0.15 of mutation rate with roulette wheel selection. Situational variables of LIM were divided into the status of buildings and status of troops, and the weight given for troops was 0.8, and that for buildings was 0.2. The value of the position where the biggest influence of the friends is 5, 1 point to be deducted for each fall while the value of the position where the biggest influence of enemies is -5, 1 point to be added for each fall. Where the influences of friends and enemies are overlapped, the total sum of each influence is to be established. Evaluation is based on how effectively the game progresses against the system of random strategy selection. Based on the status of buildings and the status of troops used for LIM, grade (A) is produced. This grade, not evaluated as an absolute value, but the relative grade has been figured out (1) by comparing with grades of the system of random strategy selection (B). This means the dominance rate of the game. The feasibility has been verified by comparing strategy performance generated by utilizing LIM like this with strategy performance generated by utilizing general IM. Fitness = A / ( A + B ) 2008 IEEE Symposium on Computational Intelligence and Games (CIG'08) (1) 387 B. Test Resuult Fig. 6. Dominance Rate Changes agaainst Random Strattegy based on F E Evolutionary Compputation and Win Rate Changes agaainst Random Strattegy b based on Evolutionnary Computation When input values of evo olutionary neurral network were w innput as themseelves without processing, p the game dominannce r in the game against AI haaving random strategy selecttion rate r rule was connverged at the t performannce rate bellow a approximately 60%. On the contrary, c when LIM was usedd as innput values of evolutionary neural n networkk, the performannce r rapidly increaseed at the beginn ning and show wed approximattely 8 80% of game dominance d ratee in about 30 generations g as can c b seen from thhe figures 6. Also be A in terms of comparisonn of s success rate, peerformance sh howed below 80% 8 when sim mple d data entry was used, but therre was successs rate higher thhan 9 90% through evvolution when LIM was usedd. Also the geneeration of diveerse strategies according to the weeight after tesst results by adjusting situational variable's weeight which is included in the Influencee Map has beeen obbserved as shoown in Figure. 7. While grooup strategy was w beeing selected tooward the direcction of multiple constructionn of trooops productioon base when the weight shhares of buildinngs annd troops were given at the raatio of 8:2, therre was a tendenncy thaat soldiers conncentrated on combats ratheer than buildinngs coonstruction whhen the weightt shares betweeen buildings and a trooops were designated as 2:8 8. Neverthelesss, it was reveaaled thaat there was similar s perform mance in termss of final succcess ratte. comparred with the case of having input values entered as simple data in the games using the evolutionnary neural networkk. Strategy seelection utiliziing neural neetworks by makingg information into i influence by factors off the game broughtt about generaation of far more m excellentt strategies comparred with neuural networkss utilizing unnprocessed informaation as itselff. For this stuudy tests tookk place by applyinng the same reesolution to thhe map in the game and Influencce Map, but we will studdy performancce changes accordinng to adjustm ment of the am mount of inpuut data by changinng resolution of o Influence Maap in the futurre. Also we plan too compare annd evaluate Layered L Influuence Map perform mances under coonditions of utiilizing techniquues such as NEAT under u indirect encoding, not direct encodinng of neural networkk only. AC CKNOWLEDGEM MENTS This research was supported s by MKE, M Korea under u ITRC IITA-20008-(C1090-08801-0011). REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] Fig. 7. Win Rate Chhanges on Evolutio F onary Computationn of Various Situaation R Ratio [14] VI. CON NCLUSIONS [15] Layered Inflluence Map haas been proposed as a way for s situation perceeption input vaalues utilized for the real-tiime s strategy and alsso as AI test simulator in thiis paper. Throuugh thhis the feasibiility has been verified from tests to see how h m much of perform mance enhanceement was achhieved when innput v values were utiilized by proceessing them into Influence Map M 388 [16] [17] R. Mikkulainen, M B. Bryant B and R. Coornelius and I. Kaarpov and K. Stannley and C.Yong, "Computational Intelligence in Game," G IEEE Com mputational Intellig igence Society, pp.. 281-300, 2006. K. Stanley, S B. Bryantt, “Real-time neurroevolution in the NERO video gam me,” Evolutionaryy Computation, IEEE IE Transactionns Vol.9, pp. 653-668 ,Dec.2005. C. Miles M and S. Louiss, “Towards The Co-Evolution C of Innfluence Map Treee Based Strateegy Game Playyer," IEEE Sym mposium on Com mputational Intellig igence and Games,, 2006 C. Ong, O H. Quek, K. Tan and A. Tayy, “Discovering Chinese C Chess Straategies through Cooevolutionary Appproaches,” IEEE Symposium S on Com mputational Intellig igence and Games ,2007. C. Ong, O H. Quek, K.. Tan and A. Tayy, “Evolving a Neeural Network Loccation Evaluator to Play Ms. Pacc-Man,” IEEE Syymposium on Com mputational Intellig igence and Games ,2005. S. Rabin, R AI GAME PROGRAMMING G WIDSOM II, Charles C River Meddia, 2005. D. B. Fogel, "A platform p for evoolving intelligentlly interactive adversaries," Biosysteem pp. 72-83,85, 2006. P. Byl B Programmingg Believable Chaaracters for Compputer Games, Chaarles river media 2004. K. Kanev, K "Design annd simulation of innteractive 3D compputer games," Com mput. & Graphics,, Vol. 22, no. 2-3, pp. p 281-300, 19988. A. Tychsen, T M. Hitchhens and T.Broluund and M.Kavaklli "The Game Masster," 2005 Interractive. Entertainnment Conferencce, Vol. 20, pp.2215-222. 2005. K. Mackin, "Evolvving Intelligent Multi-agent Syystems using Unssupervised Agent Communication C annd Behavior Trainiing," Systems, Mann, and Cyberneticss, 2000 IEEE Interrnational Conferennce, Vol. 4 pp. 2411-2414, 2000. K "Modelliing and simulationn of a group of mobile m robots" G. Klancar, Simulation Modellingg Practice and Theeory, Vol, 15, pp.6447-658, 2007. N. Stahl, "Gamebotss: A 3D virtual world w test-bed for multi-agent reseearch", INF389 - Artificial A Life, 20066 Z. Kobti, S. Sharrma, “A multi-aagent architecturee for game playying”,Computationnal Intelligence and Games, 2007. IEEE Sym mposium on CIG 2007, 2 pp. 276 - 2811 April 2007. K. Chellaphilla, C D. Fogel, F “Evolution, neural networkss, games, and inteelligence,” Proceeddings of the IEEE, 87(9), pp 1471-14496, 1999 D. Fogel, K. Chellaapilla, “Verifying anaconda’s expeert rating by com mpeting against Chinook: C experim ments in co-evolvving a neural checckers player,” Neuurocomputing, 42(1-4), pp 69-86, 20002 J. Hong, H “Evolutionn of emergent behaviors b for shhooting game charracters in Robocoode,” Congress onn Evolutionary Coomputation, 1, pp. 634-638, 2004 2008 IEEE Symposium on Computational Intelligence and Games (CIG'08)