Download Machine Learning Algorithms for Packet Routing in

Machine Learning Algorithms for Packet Routing in
Telecommunication Networks
Edwill Nel, C.W. Omlin
Department of Computer Science, University of the Western Cape
7535 Bellville, South Africa
Tel: +(27)(21) 9592406, Fax: +(27)(21) 959-3006
{enel, comlin}@uwc.ac.za
Abstract— Routing effects the overall performance of a communication network’s throughput and average packet delay.
Traditionally a centralized routing scheme is used but scales
badly to increased sized networks. Network usage is also evolving
and in certain criterion a guaranteed Quality of Service (QoS)
is required. Routing strategies is required to adapt to changing
loads and topologies. A search for alternative methods of routing
packets has resulted in machine learning (ML) as a good approach
to adaptive routing. ML methods are able to learn and adapt to
a changing environment. We have targeted three ML methods
that can be good routing strategies; 1) Reinforcement learning
(RL) can be viewed as browsing a set of policies while evaluating
them by trial through interaction with the environment. QRouting is derived from RL’s Q-Learning algorithm. Routing
decisions are made on a node-to-node basis. 2) Ant-Based Routing
- AntNet is a distributive adaptive routing algorithm that is
inspired by the stigmergy model of ants in nature. ”ants” in a
communication network explores the network and rapidly learn
good routes. 3) Particle Swarm Routing (PSR) is based on the
social behavior of bird flocking or fish schooling. PSR initializes
a population of random solutions and searches for optima by
updating generations. We propose to implement these algorithms
on a Java simulated network and test various algorithmic aspects
with traditional routing algorithms. We will empirically compare
the scalability, average packet delay, queue waiting time and
convergence time under various conditions.
I. I NTRODUCTION
Modern communication networks are growing, developing
and evolving at a rapid rate. Telecommunication systems
provide a myriad of services that use much of the network’s
resources inefficiently. These services make traffic patterns and
network behavior unpredictable, unreliable and unstable. A
distributive adaptive multi-agent system can give a solution
to the widening adversities of network traffic management and
current network routing strategies. In this paper we propose
to implement three distributive adaptive machine learning
algorithms for routing in a packet switching network. We
wish to implement, test and evaluate these algorithms against
current standard routing algorithms. We will build a Java
Network Simulator and Routing Algorithm tester to test these
algorithms. Our research will show that machine learning
techniques is a viable approach to adaptive routing in a packet
switched communication network.
II. R ELATED W ORK
Boyan and Littman [1] considered a straightforward application of Q-learning to packet routing. The ”Q-Routing”
algorithm, without having to know in advance the network
topology and traffic patterns, and without the need for any
centralized routing control system, is able to discover efficient
routing policies in a dynamically changing network.
Andrag and Omlin [2] examined the distributed, adaptive
traffic routing algorithm called Q-Routing. In the Q-Routing
algorithm, each node maintains a Q-table for estimating the
average packet delivery time via its neighbors to all destinations. These delivery time estimates are incrementally updated
based on local information of neighboring nodes. Each node
routes the packet to the neighbor with the minimum estimated
delay. Their results showed that Q-Routing was able to route
packets more efficiently at higher network loads than the static
Shortest Path algorithm. They also found Q-Routing to be more
stable than a straightforward implementation of the distributed
Bellman-Ford algorithm, using queue length as metric.
Subramanian, Druschel and Chen [3] have investigated two
distributive routing algorithms for data networks based on simple biological ”ants” that explore the network and rapidly learn
good routes. These algorithms have space and computational
overheads that are competitive with traditional packet-routing
algorithms [4]. They showed that the performance of their
algorithms scaled well with increase in network size using a
realistic topology.
Di Caro and Dorigo [5] introduced AntNet, a routing algorithm for communication networks. AntNet is an adaptive,
distributed, mobile-agent-based algorithms which was inspired
by work on the ant-colony metaphor. They applied AntNet
in a datagram network and compared it with both static and
adaptive state-of-the-art routing algorithms. AntNet showed
both very good performances and robustness under all the
experimental conditions with respect to its competitors.
White [6] wrote a technical report, describing how biologically inspired agents can be used to solve control and
management problems in Telecommunications. He describes
that the collection of agents, or swarm system, deal only with
local knowledge and exhibits a form of distributed control with
agent communication effected through the environment.
III. M ACHINE L EARNING METHODS FOR PACKET ROUTING
A. Q-Routing
Q-Learning [1][2] is a reinforcement learning algorithm
that is able to learn an optimal sequence of actions in an
environment which maximizes rewards received from the environment. Q-Routing is an adaptation from Q-Learning that is
able to distributively route packets in a network. Locally stored
information is used by each node to make a routing decision.
Routing information is stored in a Q-table as the estimated time
traveled by each neighboring node, to each destination node.
When a packet is sent the receiving node sends the estimate for
the remaining travel time for this packet back to the sending
node as a reinforcement signal to update the corresponding
Q-value.
• Qx (d, y) is the time that node x estimates it will take to
deliver a packet to node d via its neighbor y.
• When y receives the packet, it sends back a message (to
node x), containing its (i.e. y’s) best estimate of the time
remaining to get the packet to d, i.e.
t = min(Qy (d, z))overallz ∈ neighbors(y)
•
(1)
x then updates Qx (d, y) by:
[Qx (d, y)]N EW = [Qx (d, y)]OLD +η.(s + q + t −
[Qx (d, y)]OLD )
B. Ant-based Routing
Ant-Based Routing [3][5] is a novel variation of reinforcement learning that is based on simple biological ”ants”. These
”ants” explore the network and rapidly learn optimal routes
inspired by the stigmergy model of communication observed
in ant colonies. This algorithm is more resilient than traditional
routing algorithms, in the sense that random corruption of
routes has limited effect on the computation of the packetroutes. The AntNet system with its multi-agent approach is
well suited to address the stochastic distributed multi-objective
routing problem.
D. Empirical Study
We will investigate and compare Q-Routing, Ant-Based
Routing and Particle Swarm Routing against each other and
against standard Bellman-Ford and shortest path strategies.
We will look at the potential benefits of machine learning
algorithms by investigating the following characteristics;
• Efficiency savings
• Space savings
• Reduction in network traffic
• Robustness and fault-tolerance
IV. C ONCLUSION
In this paper we propose the use of Machine Learning
(ML) techniques for routing packets in communication networks. Distributive, adaptive routing strategies are essential
in networks with unpredictable changes in traffic patterns and
network topologies. We will empirically show that ML techniques adapted for routing perform better than other routing
methods. We propose to investigate and evaluate the strengths
and weaknesses of the ML algorithms with standard routing
algorithms. We will compare the average packet delay time,
average queue waiting time and the convergence time of the
network after topology changes.
ACKNOWLEDGMENT
The authors would like to thank Telkom for providing
financial support towards this project.
R EFERENCES
C. Particle Swarm Routing
Particle swarm optimization [6][7] (PSO) is a population
based stochastic optimization technique developed by Dr. Russ
Eberhart and Dr. James Kennedy in 1995, inspired by social
behavior of bird flocking or fish schooling. PSO is initialized
with a group of random solutions and then searches for optima
by updating generations. In every iteration, each particle is
updated by the following two ”best” values. The first one is the
best solution it has achieved so far. This value is called pbest.
Another ”best” value that is tracked by the particle swarm
optimizer is the best value, obtained so far by any particle in
the population. This best value is a global best and called gbest.
When a particle takes part of the population as its topological
neighbors, the best value is a local best and is called lbest. After
finding the two best values, the particle updates its velocity and
positions with the following equations (2) and (4);
v[] = v[] + c1 ∗ rand() ∗ (pbest[] − present[]) + (3)
Inherent properties of swarm intelligence as observed [8]
in nature include: massive system scalability, emergent behavior and intelligence from low complexity local interactions,
autonomy and stigmergy or communication through the environment.
(2)
c2 ∗ rand() ∗ (gbest − present[])
(3)
present[] = present[] + v[]
(4)
[1] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing
networks: A reinforcement learning approach,” in Advances in Neural Information Processing Systems (J. D. Cowan, G. Tesauro, and J. Alspector,
eds.), vol. 6, pp. 671–678, Morgan Kaufmann Publishers, Inc., 1994.
[2] W. Andrag and C. Omlin, “Reinforcement learning for routing in communication networks,” Master’s thesis, University of Stellenbosch, 2003.
[3] D. Subramanian, P. Druschel, and J. Chen, “Ants and reinforcement
learning: A case study in routing in dynamic networks,” in IJCAI (2),
pp. 832–839, 1997.
[4] D. E. Comer, Internetworking with TCP/IP, vol. 1, ch. 16. Prentice Hall,
2000.
[5] G. D. Caro and M. Dorigo, “Ant colonies for adaptive routing in packetswitched communications networks,” Lecture Notes in Computer Science,
vol. 1498, pp. 673–??, 1998.
[6] T. White, “Routing with swarm intelligence,” 1997.
[7] R. E. James Kennedy, “A new optimizer using particles swarm theory,”
1995.
[8] I. Kassabalidis, M. El-Sharkawi, R. M. II, P. Arabshahi, and A. Gray,
“Swarm intelligence for routing in communication networks,” November
2001.