Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Machine Learning Algorithms for Packet Routing in Telecommunication Networks Edwill Nel, C.W. Omlin Department of Computer Science, University of the Western Cape 7535 Bellville, South Africa Tel: +(27)(21) 9592406, Fax: +(27)(21) 959-3006 {enel, comlin}@uwc.ac.za Abstract— Routing effects the overall performance of a communication network’s throughput and average packet delay. Traditionally a centralized routing scheme is used but scales badly to increased sized networks. Network usage is also evolving and in certain criterion a guaranteed Quality of Service (QoS) is required. Routing strategies is required to adapt to changing loads and topologies. A search for alternative methods of routing packets has resulted in machine learning (ML) as a good approach to adaptive routing. ML methods are able to learn and adapt to a changing environment. We have targeted three ML methods that can be good routing strategies; 1) Reinforcement learning (RL) can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. QRouting is derived from RL’s Q-Learning algorithm. Routing decisions are made on a node-to-node basis. 2) Ant-Based Routing - AntNet is a distributive adaptive routing algorithm that is inspired by the stigmergy model of ants in nature. ”ants” in a communication network explores the network and rapidly learn good routes. 3) Particle Swarm Routing (PSR) is based on the social behavior of bird flocking or fish schooling. PSR initializes a population of random solutions and searches for optima by updating generations. We propose to implement these algorithms on a Java simulated network and test various algorithmic aspects with traditional routing algorithms. We will empirically compare the scalability, average packet delay, queue waiting time and convergence time under various conditions. I. I NTRODUCTION Modern communication networks are growing, developing and evolving at a rapid rate. Telecommunication systems provide a myriad of services that use much of the network’s resources inefficiently. These services make traffic patterns and network behavior unpredictable, unreliable and unstable. A distributive adaptive multi-agent system can give a solution to the widening adversities of network traffic management and current network routing strategies. In this paper we propose to implement three distributive adaptive machine learning algorithms for routing in a packet switching network. We wish to implement, test and evaluate these algorithms against current standard routing algorithms. We will build a Java Network Simulator and Routing Algorithm tester to test these algorithms. Our research will show that machine learning techniques is a viable approach to adaptive routing in a packet switched communication network. II. R ELATED W ORK Boyan and Littman [1] considered a straightforward application of Q-learning to packet routing. The ”Q-Routing” algorithm, without having to know in advance the network topology and traffic patterns, and without the need for any centralized routing control system, is able to discover efficient routing policies in a dynamically changing network. Andrag and Omlin [2] examined the distributed, adaptive traffic routing algorithm called Q-Routing. In the Q-Routing algorithm, each node maintains a Q-table for estimating the average packet delivery time via its neighbors to all destinations. These delivery time estimates are incrementally updated based on local information of neighboring nodes. Each node routes the packet to the neighbor with the minimum estimated delay. Their results showed that Q-Routing was able to route packets more efficiently at higher network loads than the static Shortest Path algorithm. They also found Q-Routing to be more stable than a straightforward implementation of the distributed Bellman-Ford algorithm, using queue length as metric. Subramanian, Druschel and Chen [3] have investigated two distributive routing algorithms for data networks based on simple biological ”ants” that explore the network and rapidly learn good routes. These algorithms have space and computational overheads that are competitive with traditional packet-routing algorithms [4]. They showed that the performance of their algorithms scaled well with increase in network size using a realistic topology. Di Caro and Dorigo [5] introduced AntNet, a routing algorithm for communication networks. AntNet is an adaptive, distributed, mobile-agent-based algorithms which was inspired by work on the ant-colony metaphor. They applied AntNet in a datagram network and compared it with both static and adaptive state-of-the-art routing algorithms. AntNet showed both very good performances and robustness under all the experimental conditions with respect to its competitors. White [6] wrote a technical report, describing how biologically inspired agents can be used to solve control and management problems in Telecommunications. He describes that the collection of agents, or swarm system, deal only with local knowledge and exhibits a form of distributed control with agent communication effected through the environment. III. M ACHINE L EARNING METHODS FOR PACKET ROUTING A. Q-Routing Q-Learning [1][2] is a reinforcement learning algorithm that is able to learn an optimal sequence of actions in an environment which maximizes rewards received from the environment. Q-Routing is an adaptation from Q-Learning that is able to distributively route packets in a network. Locally stored information is used by each node to make a routing decision. Routing information is stored in a Q-table as the estimated time traveled by each neighboring node, to each destination node. When a packet is sent the receiving node sends the estimate for the remaining travel time for this packet back to the sending node as a reinforcement signal to update the corresponding Q-value. • Qx (d, y) is the time that node x estimates it will take to deliver a packet to node d via its neighbor y. • When y receives the packet, it sends back a message (to node x), containing its (i.e. y’s) best estimate of the time remaining to get the packet to d, i.e. t = min(Qy (d, z))overallz ∈ neighbors(y) • (1) x then updates Qx (d, y) by: [Qx (d, y)]N EW = [Qx (d, y)]OLD +η.(s + q + t − [Qx (d, y)]OLD ) B. Ant-based Routing Ant-Based Routing [3][5] is a novel variation of reinforcement learning that is based on simple biological ”ants”. These ”ants” explore the network and rapidly learn optimal routes inspired by the stigmergy model of communication observed in ant colonies. This algorithm is more resilient than traditional routing algorithms, in the sense that random corruption of routes has limited effect on the computation of the packetroutes. The AntNet system with its multi-agent approach is well suited to address the stochastic distributed multi-objective routing problem. D. Empirical Study We will investigate and compare Q-Routing, Ant-Based Routing and Particle Swarm Routing against each other and against standard Bellman-Ford and shortest path strategies. We will look at the potential benefits of machine learning algorithms by investigating the following characteristics; • Efficiency savings • Space savings • Reduction in network traffic • Robustness and fault-tolerance IV. C ONCLUSION In this paper we propose the use of Machine Learning (ML) techniques for routing packets in communication networks. Distributive, adaptive routing strategies are essential in networks with unpredictable changes in traffic patterns and network topologies. We will empirically show that ML techniques adapted for routing perform better than other routing methods. We propose to investigate and evaluate the strengths and weaknesses of the ML algorithms with standard routing algorithms. We will compare the average packet delay time, average queue waiting time and the convergence time of the network after topology changes. ACKNOWLEDGMENT The authors would like to thank Telkom for providing financial support towards this project. R EFERENCES C. Particle Swarm Routing Particle swarm optimization [6][7] (PSO) is a population based stochastic optimization technique developed by Dr. Russ Eberhart and Dr. James Kennedy in 1995, inspired by social behavior of bird flocking or fish schooling. PSO is initialized with a group of random solutions and then searches for optima by updating generations. In every iteration, each particle is updated by the following two ”best” values. The first one is the best solution it has achieved so far. This value is called pbest. Another ”best” value that is tracked by the particle swarm optimizer is the best value, obtained so far by any particle in the population. This best value is a global best and called gbest. When a particle takes part of the population as its topological neighbors, the best value is a local best and is called lbest. After finding the two best values, the particle updates its velocity and positions with the following equations (2) and (4); v[] = v[] + c1 ∗ rand() ∗ (pbest[] − present[]) + (3) Inherent properties of swarm intelligence as observed [8] in nature include: massive system scalability, emergent behavior and intelligence from low complexity local interactions, autonomy and stigmergy or communication through the environment. (2) c2 ∗ rand() ∗ (gbest − present[]) (3) present[] = present[] + v[] (4) [1] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in Advances in Neural Information Processing Systems (J. D. Cowan, G. Tesauro, and J. Alspector, eds.), vol. 6, pp. 671–678, Morgan Kaufmann Publishers, Inc., 1994. [2] W. Andrag and C. Omlin, “Reinforcement learning for routing in communication networks,” Master’s thesis, University of Stellenbosch, 2003. [3] D. Subramanian, P. Druschel, and J. Chen, “Ants and reinforcement learning: A case study in routing in dynamic networks,” in IJCAI (2), pp. 832–839, 1997. [4] D. E. Comer, Internetworking with TCP/IP, vol. 1, ch. 16. Prentice Hall, 2000. [5] G. D. Caro and M. Dorigo, “Ant colonies for adaptive routing in packetswitched communications networks,” Lecture Notes in Computer Science, vol. 1498, pp. 673–??, 1998. [6] T. White, “Routing with swarm intelligence,” 1997. [7] R. E. James Kennedy, “A new optimizer using particles swarm theory,” 1995. [8] I. Kassabalidis, M. El-Sharkawi, R. M. II, P. Arabshahi, and A. Gray, “Swarm intelligence for routing in communication networks,” November 2001.