Download Business Intelligence: Optimization for Decision Making

Business Intelligence:
Optimization for Decision Making
A short introduction to Particle Swarm Optimization
Michael G. Epitropakis
Computational Heuristics, Operational Research and Decision Support CHORDS,
School of Natural Sciences,
Computing Science and Mathematics,
University of Stirling, UK
[email protected]
Stirling, 24 March 2015
Michael G. Epitropakis
Swarm Intelligence for Decision Making
1
Outline
1
Business Intelligence
Motivation
Definition
Business Intelligence Architecture
2
Global Optimization Problem
3
Swarm Intelligence
Particle Swarm Optimization (PSO)
Background, Origins.
The Original PSO model
PSO: Geometric Illustration
4
Applications
5
References
Michael G. Epitropakis
Swarm Intelligence for Decision Making
2
Business Intelligence
Motivation
Business Intelligence: Motivation
Amazon, Barclays, Facebook, Google, Lloyds, Microsoft,
Sainsbury’s, TESCO, ...
Data!
The answer to my problem is hidden in my data... but I cannot
dig it up!
Michael G. Epitropakis
Swarm Intelligence for Decision Making
3
Business Intelligence
Definition
Business Intelligence
The enterprises that are capable of transforming data into
information and knowledge can use them to make quicker
and more effective decisions and thus to achieve a competitive
advantage.
Business Intelligence:
Business intelligence may be defined as a set of
mathematical models and analysis methodologies that
exploit the available data to generate information and
knowledge useful for complex decision-making processes.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
4
Business Intelligence
Definition
Business Intelligence
Main purpose of business intelligence systems:
Is to provide decision makers with tools and methodologies that
allow them to make effective and timely decisions.
Effective: The application of rigorous analytical methods
allows decision makers to rely on information and
knowledge from data which are more dependable.
Make better decisions and devise action plans that allow
their objectives to be reached in a more effective way.
Timely: Enterprises operate in economic environments
characterized by growing levels of competition and high
dynamism.
Rapidly react to the actions of competitors and to new
market conditions is a critical factor in the success or even
the survival of a company.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
5
Business Intelligence
Definition
Business Intelligence
Michael G. Epitropakis
Swarm Intelligence for Decision Making
6
Business Intelligence
Definition
Data, information and knowledge
Data −→ Information −→ Knowledge
Data: are collected on a daily basis
in the form of bits, numbers,
symbols, and "objects".
Information: is "organized data",
which are preprocessed, cleaned,
arranged into structures, and
stripped of redundancy.
Knowledge: is "integrated
information", which includes facts
and relationships that have been
perceived, discovered, or learned.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
7
Business Intelligence
Business Intelligence Architecture
Business Intelligence Architecture
Michael G. Epitropakis
Swarm Intelligence for Decision Making
8
Global Optimization Problem
Global Optimization
Single objective minimization problem:
Given a real-valued objective function f : Dn ⊆ Rn → R, the aim is to
find an x? = (x1? , x2? , . . . , xn? )> ∈ Dn that
x? = arg min f (x).
x∈Dn
x? is a global minimizer and Dn is an n-dimensional scaled translation
of the unit hypercube.
An objective function (f )
A solution representation of x (here x ∈ Dn ⊆ Rn )
A search strategy – optimization algorithm.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
9
Swarm Intelligence
Swarm Intelligence
Swarm intelligence (SI) is the collective behavior of
decentralized, self-organized systems, natural or artificial.
(Wikipedia)
The system has abilities that are not present in the
individuals (is more intelligent)
“The whole is more than the sum of its parts”
Cooperation, co-evolution, competition, self-organisation
and communication
Examples of systems can be found in nature: ant colonies,
bird flocking, animal herding, bacteria molding and fish
schooling
Beni, G., Wang, J. Swarm Intelligence in Cellular Robotic Systems, Proceed. NATO Advanced Workshop on Robots
and Biological Systems, Tuscany, Italy, June 26-30 (1989)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
10
Swarm Intelligence
Swarm Intelligence
Michael G. Epitropakis
Swarm Intelligence for Decision Making
11
Swarm Intelligence
Swarm Intelligence Applications
Swarm-bots, an EU project led by Marco Dorigo, aimed to
study new approaches to the design and implementation
of self-organizing and self-assembling artifacts
(http://www.swarm-bots.org/).
Swarmanoid: Towards Humanoid Robotic Swarms, The
main scientific objective of this research project is the
design, implementation and control of a novel distributed
robotic system (http://www.swarmanoid.org)
Creation of complex interactive environments.
Disney’s The Lion King was the first movie to make use
of swarm technology (the stampede of the bisons scene).
The movie “Lord of the Rings” has also made use of
similar technology during battle scenes
Michael G. Epitropakis
Swarm Intelligence for Decision Making
12
Swarm Intelligence
Particle Swarm Optimization (PSO)
Particle Swarm Optimization
The inventors:
James Kennedy
(social psychologist)
Russell C. Eberhart
(electrical engineer)
J. Kennedy, and R. Eberhart, Particle swarm optimization, in
Proc. IEEE. Int. Conf. on Neural Networks, Piscataway, NJ, pp.
1942–1948, 1995.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
13
Swarm Intelligence
Particle Swarm Optimization (PSO)
Particle Swarm Optimization (PSO)
What is PSO:
a simple, computationally efficient optimization method
population-based, stochastic search method
direct search method, i.e. gradient free
individuals follow very simple behaviors:
emulate the success of neighboring individuals,
but also bias towards on experience of success
emergent behavior: discovery of optimal regions within a
high dimensional search space
“Particle swarm algorithm imitates human (or insects) social behavior. Individuals
interact with one another while learning from their own experience, and gradually the
population members move into better regions of the problem space” Eberhart &
Kennedy
Michael G. Epitropakis
Swarm Intelligence for Decision Making
14
Swarm Intelligence
Background, Origins.
Induce complex behavior from simple rules
Origins of PSO (precursors)
Reynolds (1987)’s simulation Boids: a simple flocking model
consists of three simple local rules: http://www.red3d.com/cwr/boids/
Separation: Avoid Collision with neighboring
agents (steer to avoid crowding local
flockmates)
Alignment: Match the velocity of neighboring
agents (steer towards the average heading of
local flockmates)
Cohesion: Stay near neighboring agents
(steer to move toward the average position of
local flockmates)
The work of Heppner and Grenander on using a “roost” as
attractor of all birds in the flock [HG90] (Seek roost)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
15
Swarm Intelligence
Background, Origins.
Towards a computational principle
Evaluate your current position
Compare it to your own experience (previous best) and to
the experience of your society (neighborhood best)
Imitate yourself and the others
Basic hypothesis:
There are two major sources of cognition:
own experience and
communication from others
Leon Festinger, 1954/1999, Social Communication and Cognition
Michael G. Epitropakis
Swarm Intelligence for Decision Making
16
Swarm Intelligence
Background, Origins.
The Original PSO model:
is a simplified social model of determining nearest neighbors
and velocity matching
Initial objective: to simulate the graceful, unpredictable
choreography of collision-proof birds in a flock
Randomly initializes positions of birds
At each iteration, each individual determines its nearest
neighbor and replaces its velocity with that of its neighbor
This resulted in synchronous movement of the flock, but
flock settled too quickly on the same, unchanging flying
direction
Michael G. Epitropakis
Swarm Intelligence for Decision Making
17
Swarm Intelligence
Background, Origins.
Random adjustments to velocities (referred to as
craziness) prevented individuals to settle too quickly on an
unchanging direction
To further expand the model, “roosts” were added as
attractors:
personal experience (personal best)
social experience (neighborhood best)
Introduction of the Particle Swarm Optimization method.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
18
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization (PSO)
Its main components:
What are the main components:
a swarm of particles (size usually fixed: NP)
each particle represents a candidate solution of the problem at
hand
the elements of a particle represent parameters to be optimized
The search process:
Position updates:
Xi (t + 1) = Xi (t) + Vi (t + 1),
xi,j (0) ∼ U(LBj , UBj )
Velocity updates:
denotes the amount of change (step size)
drives the optimization process
reflects the cognitive experience of a particle and the
socially exchanged information between particles.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
19
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The general PSO Algorithm
1: Initialize particles in the swarm
2: for each time step t do
3:
for each particle i in the swarm i ∈ {1, 2, . . . , NP} do
4:
Update cognitive knowledge/experience
5:
Update social knowledge/experience
6:
end for
7:
for each particle i in the swarm i ∈ {1, 2, . . . , NP} do
8:
Update Velocity of particle i
9:
Update Position of particle i
10:
end for
11: end for
Michael G. Epitropakis
Swarm Intelligence for Decision Making
20
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The global best (gbest) PSO
A simple PSO model: global best (gbest) PSO (Eberhart &
Kennedy, 1995)
It uses a full neighborhood topology (star social network).
Velocity update rule per dimension:
vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) − xi,j (t) + c2 r2 (t) pbest,j (t) − xi,j (t) ,
vi,j (0) = 0 (preferred)
c1 , c2 are positive acceleration coefficients
r1 (t), r2 (t) ∼ U(0, 1)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
21
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The global best (gbest) PSO
A simple PSO model: global best (gbest) PSO (Eberhart &
Kennedy, 1995)
It uses a full neighborhood topology (star social network).
Velocity update rule per dimension:
vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) − Xi,j (t) + c2 r2 (t) pbest,j (t) − xi,j (t) ,
| {z }
momentum
momentum:
inertia component
previous velocity term to carry the particle in the direction it
has traveled so far
prevents particle from drastically changing direction
Michael G. Epitropakis
Swarm Intelligence for Decision Making
21
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The global best (gbest) PSO
A simple PSO model: global best (gbest) PSO (Eberhart &
Kennedy, 1995)
It uses a full neighborhood topology (star social network).
Velocity update rule per dimension:
Vi (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) − xi,j (t) + c2 r2 (t) pbest,j (t) − xi,j (t) ,
|
{z
}
cognitive component
cognitive component:
Pi (t): personal best position vector
quantifies performance relative to past performances
tendency to return to the best position visited so far
(memory)
nostalgia
Michael G. Epitropakis
Swarm Intelligence for Decision Making
21
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The global best (gbest) PSO
A simple PSO model: global best (gbest) PSO (Eberhart &
Kennedy, 1995)
It uses a full neighborhood topology (star social network).
Velocity update rule per dimension:
vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) − xi,j (t) + c2 r2 (t) pbest,j (t) − xi,j (t) ,
{z
}
|
social component
social component:
Pbest (t): neighborhood best position vector (here: global
best position)
quantifies performance relative to neighbors
tendency to be attracted towards the best position found in
its neighborhood.
envy
Michael G. Epitropakis
Swarm Intelligence for Decision Making
21
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
Update experience
Pi (t) is the personal best position calculated as (assuming
minimization):
Pi (t)
if f (Xi (t + 1)) ≥ f (Pi (t))
Pi (t + 1) =
Xi (t + 1) if f (Xi (t + 1)) < f (Pi (t))
Pbest (t) is the global best position calculated as:
Pbest (t) = min{f (P0 (t)), f (P1 (t)), . . . , f (PNP (t))}
where NP is the number of particles in the swarm.
Michael G. Epitropakis
Swarm Intelligence for Decision Making
22
Swarm Intelligence
The Original PSO model
Particle Swarm Optimization
The gbest PSO Algorithm
1: Initialize particles in the swarm
2: for each time step t do
3:
for each particle i in the swarm i ∈ {1, 2, . . . , NP} do
4:
if f (Xi (t)) < f (Pi (t)) then
5:
Pi (t) = Xi (t)
6:
end if
7:
if f (Pi (t)) < f (Pbest (t)) then
8:
Pbest (t) = Pi (t)
9:
end if
10:
end for
11:
for each particle i in the swarm i ∈ {1, 2, . . . , NP} do
12:
Vi (t + 1) = Vi (t) + c1 r1 (t) Pi (t) − Xi (t) + c2 r2 (t) Pbest (t) − Xi (t) ,
13:
Xi (t + 1) = Xi (t) + Vi (t + 1)
14:
end for
15: end for
Michael G. Epitropakis
Swarm Intelligence for Decision Making
23
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
Vi(t)
Pbest(t)
Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
Vi(t)
Pbest(t)
Pbest(t) − Xi(t)
Xi(t)
Pi(t) − Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
Vi(t)
Pbest(t)
Pbest(t) − Xi(t)
Xi(t)
Pi(t) − Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
c1r1(t)(Pbest(t) − Xi(t))
Vi(t)
Pbest(t)
Pbest(t) − Xi(t)
Xi(t)
Pi(t) − Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
c1r1(t)(Pbest(t) − Xi(t))
c1r1(t)(Pi(t) − Xi(t))
Vi(t)
Pbest(t)
Pbest(t) − Xi(t)
Xi(t)
Pi(t) − Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Geometric Illustration
x2
c1r1(t)(Pbest(t) − Xi(t))
c1r1(t)(Pi(t) − Xi(t))
Xi(t + 1)
Vi(t)
Pbest(t)
Pbest(t) − Xi(t)
Xi(t)
Pi(t) − Xi(t)
Pi(t)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
x1
24
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
The local best (lbest) PSO
The local best (lbest) PSO uses a neighborhood topology (ring
social network).
Velocity update rule per dimension:
vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) − xi,j (t) + c2 r2 (t) pnbest,j (t) − xi,j (t) ,
|
{z
}
social component
Pnbest (t): is the neighborhood best, defined as:
Pnbest (t + 1) ∈ {x ∈ Ni | min{f (x), ∀x ∈ Ni }}
Ni = {pi−nNi , pi−nNi +1 , . . . , pi−1 , pi , pi+1 , . . . , pi+nNi }
where nNi is the neighborhood size
neighborhoods are based on particle indices, not spatial
information
neighborhoods overlap to facilitate information exchange
Michael G. Epitropakis
Swarm Intelligence for Decision Making
25
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Social Neighborhood topologies: Social Network Structures
Two most common models:
lbest: each particle is influenced only by particles in local
neighborhood
gbest: each particle is influenced by the best found from
the entire swarm
Ring Topology
Michael G. Epitropakis
Star/full topology
Swarm Intelligence for Decision Making
26
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Social Neighborhood topologies: Social Network Structures (2)
Von Neumann
Topology
Four Clusters Topology
Michael G. Epitropakis
Wheel Topology
Swarm Intelligence for Decision Making
27
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
Problem:
The velocity has a tendency to
explode to large values.
Solution: Velocity Clamping
vi,j (t + 1) =
vi,j (t + 1)
sgn(vi,j (t + 1))Vmax,j
if |vi,j (t + 1)| < Vmax,j
if |vi,j (t + 1)| ≥ Vmax,j
controlling the global exploration of the particles
it is problem-dependent
does not necessarily prevent particles from leaving the search
space, nor to converge.
it confines the step sizes, therefore restricting particles from
further divergence
Michael G. Epitropakis
Swarm Intelligence for Decision Making
28
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
x2
Velocity Update
Position Update
Xi(t)
x1
Michael G. Epitropakis
Swarm Intelligence for Decision Making
29
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1)
Xi(t)
vi(t + 1)
x1
Michael G. Epitropakis
Swarm Intelligence for Decision Making
29
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1)
Xi(t)
Xi(t + 1)
vi(t + 1)
x1
Michael G. Epitropakis
Swarm Intelligence for Decision Making
29
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1)
Xi(t + 1)
v̂2(t + 1)
Xi(t)
vi(t + 1)
x1
Michael G. Epitropakis
Swarm Intelligence for Decision Making
29
Swarm Intelligence
PSO: Geometric Illustration
Particle Swarm Optimization
Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1)
Xi(t + 1)
v̂2(t + 1)
X̂i(t + 1)
Xi(t)
vi(t + 1)
x1
Michael G. Epitropakis
Swarm Intelligence for Decision Making
29
Swarm Intelligence
PSO: Geometric Illustration
Some PSO Variants
Tribes (Clerc, 2006) – aims to adapt population size, so that it does not have to
be set by the users
FDR-PSO (Veeramachaneni, et al., 2003) – using nearest neighbour interactions
Cooperative PSO (van den Bergh and Engelbrecht, 2005) – a cooperative
approach
CLPSO (Liang, et al., 2006) – incorporate learning from more previous best
particles.
FIPS Fully Informed PSO (Mendes, Kennedy, 2004) – use several attractors in
the update rule
BBPSO Bare Bones PSO (Kennedy, 2003) – uses normal distribution around
personal/global best
UPSO Unified PSO (Parsopoulos, Vrahatis, 2004) – a unification of gbest and
lbest versions
PSODE (Epitropakis et al., 2012) – aims to combine various state-of-the-art DE
and PSO variants
Standard PSO 2006, 2007, 2011 – aims to define a standard PSO version for
comparisons
http://www.particleswarm.info/
Michael G. Epitropakis
Swarm Intelligence for Decision Making
30
Applications
Applications (1)
Optimization Problems:
Continuous, discrete, mixed search spaces, Optimization
Problems
Combinatorial Optimization Problems
Large Scale Optimization Problems
Multi-modal Optimization Problems
Multi-objective Optimization Problems
Problems in Dynamic and Uncertain environments
Constraint Optimization Problems
Michael G. Epitropakis
Swarm Intelligence for Decision Making
31
Applications
Applications (2)
Applications in:
Machine Learning (clustering, classification, parameter
tuning, feature selection)
Artificial Neural Networks (training, evolving structures)
Robotics (path planning, localization)
Bio-informatics and Medical Informatics (Medical diagnosis
and decision making)
Image processing (Image analysis, segmentation, pattern
recognition)
Industrial Applications (Job Scheduling, Vehicle Routing
Problem, Traveling Salesman Problem)
Michael G. Epitropakis
Swarm Intelligence for Decision Making
32
(-: Thank you very much for your attention :-)
Questions ???
Michael G. Epitropakis: [email protected]
www.epitropakis.co.uk
Michael G. Epitropakis
Swarm Intelligence for Decision Making
33
References
References: Business Intelligence (Incomplete)
1
Carlo Vercellis, Business Intelligence: Data Mining and Optimization for Decision
Making, John Wiley & Sons, 2009
2
Z. Michalewicz, M. Schmidt, M. Michalewicz, Adaptive Business Intelligence,
Springer, 2006
3
Foster Provost, Tom Fawcett, Data Science for Business: What you need to
know about data mining and data-analytic thinking, O’Reilly Media, 2013
4
A. Brabazon, M. O’Neill, I. Dempsey, An Introduction to Evolutionary
Computation in Finance, IEEE Computational Intelligence Magazine, 2008
5
W. Pedrycz, N. Ichalkaranje, G.P. Wren and L. Jain, Introduction to
Computational Intelligence for Decision Making, in Studies in Computational
Intelligence, 97, 79-96, Springer-Verlag, 2008
Michael G. Epitropakis
Swarm Intelligence for Decision Making
34
References
References: PSO (Incomplete)
1
2
3
4
5
6
7
8
9
10
Reynolds, C.W.: Flocks, herds and schools: a distributed behavioral model.
Computer Graphics, 21(4), p.25-34, 1987.
Heppner, F. and Grenander, U.: A stochastic nonlinear model for coordinated
bird flocks. In S.Krasner, Ed., The Ubiquity of Chaos. AAAS Publications,
Washington, DC, 1990.
Kennedy, J. and Eberhart, R.: Particle Swarm Optimization. In Proceedings of
the Fourth IEEE International Conference on Neural Networks, Perth, Australia.
IEEE Service Center(1995) 1942-1948.
Kennedy, J., Eberhart, R. C., and Shi, Y., Swarm intelligence, San Francisco:
Morgan Kaufmann Publishers, 2001.
J. Kennedy, Small Worlds and Mega-Minds: Effects of Neighborhood Topology
on Particle Swarm Performance, Proceedings of the IEEE Congress on
Evolutionary Computation, 1999, pp. 1931-1938.
Clerc, M.: Particle Swarm Optimization, ISTE Ltd, 2006.
Engelbrecht, A.P., Fundamentals of Computational Swarm Intelligence, Wiley,
2006.
Xiaodong Li, Advances in Particle Swarm Optimization, Tutorial, ACISS’09,
Melbourne
Engelbrecht, A.P., Particle Swarm Optimization Tutorial, IEEE Congress on
Evolutionary Computation, 2013
KE Parsopoulos, MN Vrahatis, Particle swarm optimization and intelligence:
advances and applications, Information Science Reference, 2010
Michael G. Epitropakis
Swarm Intelligence for Decision Making
35