Download chapter8B

Chapter 8
Geocomputation Part B:
Artificial Neural Networks (ANNs) &
Genetic Algorithms (GAs)
www.spatialanalysisonline.com
Geocomputation: ANNs
In this presentation on geocomputation:
ANNs discussed include
Multi-level perceptrons (MLPs)
Radial basis function neural networks (RBFNNs)
Self organising feature maps (SOFMs)
ANNs are particularly concerned with
Function approximation and interpolation
Image analysis and classification
Spatial interaction modelling
3rd edition
www.spatialanalysisonline.com
2
Geocomputation: Evolutionary computing
In this presentation on geocomputation:
EC elements discussed include
Genetic algorithms (GAs)
Genetic programming (GP)
EC is particularly concerned with
Complex problem solving using GAs
Model design using GP methods
3rd edition
www.spatialanalysisonline.com
3
Geocomputation
 Artificial Neural Networks (ANNs)
A computational model based on emulating
biological neural networks
A form of non-linear modelling tool
Often a 3-layer network structure is used:
input, hidden, output
The output layer of such structures are typically
modified weighted sums of intermediate layers,
which are modified weighted sums of the input
layer
3rd edition
www.spatialanalysisonline.com
4
Artificial Neural Networks
Hence at each output node (hidden or
final) a two-step process takes place:

3rd edition
g
www.spatialanalysisonline.com
5
Artificial Neural Networks
 Simple 3-layer feedforward ANN
 Fully inter-connected; each connection
is given a weight, w
 Known as a Multi-level perceptron
(MLP)
 In this case: 3 input nodes, 5 hidden
nodes, 2 output nodes and 2 bias
nodes (bias, B, is similar to the
constant term in regression models)
 At hidden node 1 we have:
B2
B1
1
2
1
1
3
2
2
3
4
h1*  w 01b1  w11x1  w 21x 2  w 31x 3
5
where the wij are weights to be
determined, b1=1, and the xi are the
observed input values
INPUT
n=3
3rd edition
www.spatialanalysisonline.com
HIDDEN
m=5
OUTPUT
p=2
6
Artificial Neural Networks
h1*  w 01b1  w11x1  w 21x 2  w 31x 3
is simply a linear weighted
sum of the inputs. To generate
a non-linear output it must be
modified by some (well
behaved) non-linear function,
g(), e.g. the logistic function:
g(h) 
Sample activation functions
1
0.5
1
1  e h
i.e.
0
-5
-4
h1  g w01b1  w11x1  w21x2  w31x3 
3rd edition
www.spatialanalysisonline.com
-3
-2
Logistic
-1
0
(1+Tanh)/2
1
2
Step
3
4
5
Linear
7
Artificial Neural Networks
We can now compute the output layer values as the
weighted sum
y1*  z01b2  z11h1  z21h2  z31h3  z41h4  z51h5 , and thus
 
y1  g y1*
Suppose we have known input values x1=1, x2=-3, x3=5,
and known outputs of 0 and 1. Can we select the weights
to ensure the inputs generate the known outputs?
Suggestion: <build your own worked example & program
here>
3rd edition
www.spatialanalysisonline.com
8
Artificial Neural Networks
 Learning
Supervised learning
 Split training/test data sets (control data)
 Known inputs and output (target) values for training data
 (Network output-Target output) = Error signal, e
 Systematically adjust weights to minimise sum of e2
 Adjustment typically based on backpropagation and gradient
descent
 Used in many classification/pattern recognition applications
and in function approximation
Unsupervised learning
 No training data
 Must create clusters by analysing dataset for patterns/clusters
3rd edition
www.spatialanalysisonline.com
9
Artificial Neural Networks
Some basic issues:
local vs global minimisation
Initialisation and selection
Data normalisation and coding
Momentum
Model design and over-fitting
Overtraining
Interpolation vs Extrapolation/Forecasting
3rd edition
www.spatialanalysisonline.com
10
Artificial Neural Networks
MLP: Example 1 function approximation
0.5
0.6
0.6
0.45
0.4
0.4
0.2
0.2
0
0
-0.2
-0.2
-0.4
-0.4
-0.6
-0.6
-0.8
-0.8
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
source data
3rd edition
0.4
0.6
0.8
1
-1
RMSE (Root mean squared error)
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
fitted solution curve
www.spatialanalysisonline.com
1
0
0
100
200
300
400
500
600
Epochs
700
800
900
1000
RMSE vs epochs
11
Artificial Neural Networks
 MLP Example 2: LCM
3rd edition
www.spatialanalysisonline.com
12
Artificial Neural Networks
 MLP Example 2: LCM
3rd edition
www.spatialanalysisonline.com
13
Artificial Neural Networks
 MLP Example 2: LCM
3rd edition
www.spatialanalysisonline.com
14
Artificial Neural Networks
 MLP Example 2: LCM
weights matrix
3rd edition
www.spatialanalysisonline.com
15
Artificial Neural Networks
 MLP Example 3: Spatial interaction model
Generalised model: Tij=f(Oi,Dj,dij)
Sample data format
(log transformed):
3rd edition
www.spatialanalysisonline.com
16
Artificial Neural Networks
 MLP Example 3: Spatial interaction model
3rd edition
www.spatialanalysisonline.com
17
Artificial Neural Networks
 Radial Basis Function Networks
Basic functional form:
n
yk 
 w ( x
ij
i
 c j ),
i 1
where
denotes the Euclidean norm.
Gaussian RBF:
( x ) 
3rd edition
1
2 2
e

 x 2 /2 2

www.spatialanalysisonline.com
18
Artificial Neural Networks
 Self organising function maps
SOM as an output space
Neighbourhood relations
Grid size, form and topology
3rd edition
www.spatialanalysisonline.com
19
Artificial Neural Networks
 Self organising function maps
Dimensional reductions
Mapped output – similar vectors (units) are
close to each other
Typically an unsupervised procedure
Spatial mapping of SOM can follow using
simple assignment to best matching unit (BMU)
3rd edition
www.spatialanalysisonline.com
20
Artificial Neural Networks
 Self organising function maps
Choose a grid size, form and topology
Train the network
Identify the best matching units
Modify the BMU and its neighbours (spatially biased
learning rule)
Map the trained network
3rd edition
www.spatialanalysisonline.com
21
Artificial Neural Networks
 Self organising function maps – some issues
Initialisation
Pre-processing
Normalisation
Missing data
Masking and weighting
Learning and tuning
Distance metrics
Neighbourhood functions (kernels)
Learning rate functions
3rd edition
www.spatialanalysisonline.com
22
Artificial Neural Networks
 Self organising function maps – Idrisi
3rd edition
www.spatialanalysisonline.com
23
Artificial Neural Networks
 Self organising function maps – Idrisi
3rd edition
www.spatialanalysisonline.com
24
Genetic Algorithms
 Solutions are represented as individuals
Individuals are modelled as chromosomes
Chromosomes are comprised of genes
Genes have values known as alleles
Chromosomes have a measurable fitness
New chromosomes (children) are created by
reproduction and mutation processes
The fittest individuals survive
The creation process is then iterated
3rd edition
www.spatialanalysisonline.com
25
Genetic Algorithms
GAs: Example 1 - TSP
allele=12 (ID of town in TSP problem set)
chromosome
genes
Each chromosome contains complete list of towns
• create a set of m randomly permuted strings and compute lengths, d
• evaluate the fitness of each string (e.g. 1/d)
• select random pairs of tours (biased by fitness)
• combine pairs by crossover operation
• evaluate fitness of offspring
• apply replacement rule (fittest retained) and iterate till stable
3rd edition
www.spatialanalysisonline.com
26
Genetic Algorithms
 GA components
 Encoding or representation – binary, list, tree etc
 Fitness function selection – use of rank transforms
 Population initialisation
 Selection: roulette, tournament, uniform random
 Reproduction
 Crossover e.g. A = [a b c d e f g h] B = [1 2 3 4 5 6 7 8]
and the crossover point is 3, the following children are generated:
child 1 = [a b c 4 5 6 7 8] child 2= [1 2 3 d e f g h]
 Mutation
 Local search
 Termination
3rd edition
www.spatialanalysisonline.com
27
Genetic Algorithms
 GAs: application areas
TSP (as above)
Clustering
Map labelling
Optimum location with capacity constraints
Concept can be extended to alleles that are
expressions or program elements rather than
numerical values  Genetic programming
3rd edition
www.spatialanalysisonline.com
28