Download 23 Sep - 04 Oct, 2002 WSC7 7

Canadian Weather Analysis
Using Connectionist Learning
Paradigms
Imran Maqsood*, Muhammad Riaz Khan, Ajith Abraham
*Environmental Systems Engineering Program, Faculty of Engineering, University of Regina,
Regina, Saskatchewan S4S 0A2, Canada, E-mail: [email protected]
Partner
Technologies Incorporated, 1155 Park Street, Regina, Saskatchewan S4N 4Y8,
Canada, E-mail:[email protected]
Faculty of Information Technology, School of Business Systems, Monash University,
Clayton 3800, Australia, E-mail: [email protected]
7th Online World Conference on Soft Computing in Industrial Applications (on WWW), September 23 - October 4, 2002
CONTENTS
•
•
•
•
•
Introduction
MLP, ERNN and RBFN Background
Experimental Setup of a Case Study
Test Results
Conclusions
23 Sep - 04 Oct, 2002
WSC7
2
1. INTRODUCTION
• Weather forecasts provide critical information
about future weather
• Weather forecasting remains a complex business,
due to its chaotic and unpredictable nature
• Combined with threat by the global warming and
green house gas effect, impact of extreme weather
phenomena on society is growing costly, causing
infrastructure damage, injury and the loss of life.
23 Sep - 04 Oct, 2002
WSC7
3
• Accurate weather forecast models are important to
the countries, where the entire agriculture depends
upon weather
• Previously, several artificial intelligence
techniques have been used in the past for
modeling chaotic behavior of weather
• However, several of them use simple feed-forward
neural network training methods using
backpropagation algorithm
23 Sep - 04 Oct, 2002
WSC7
4
Study Objectives
• To develop an accurate and reliable predictive
models for forecasting the weather of Vancouver,
BC, Canada.
• To compare performance of multi-layered
perception (MLP) neural networks, Elman
recurrent neural networks (ERNN) and radial basis
function network (RBFN) for the weather
analysis.
23 Sep - 04 Oct, 2002
WSC7
5
2. ANN BACKGROUND
INFORMATION
•
•
•
•
•
•
ANN Advantages
An ability to solve complex and non-linear
problems
Quick response
Self-organization
Real time operation
Fault tolerance via redundant information coding
Adaptability and generalization
23 Sep - 04 Oct, 2002
WSC7
6
(a) Multi-Layered Perceptron (MLP) Networks
• network is arranged in layers of
neurons
• every neuron in a layer computes
sum of its inputs and passes this
sum through a nonlinear function
as its output.
• Each neuron has only one output,
but this output is multiplied by a
weighting factor if it is to be used
as an input to another neuron (in a
next higher layer)
• There are no connections among
neurons in the same layer.
23 Sep - 04 Oct, 2002
WSC7
wjk
Wij
I1
O1
I2
O2



In




k



Om
Oi
Vj
Input
Layer
Hidden
Layer
Output
Layer
Figure: Architecture of 3-layered MLP network
for weather forecasting
7
(b) Elman Recurrent Neural Networks (ERNN)
• ERNN are a subclass of
recurrent networks
• They are multilayer perceptron
networks augmented with one
or more additional context
layers storing output values of
one of the layers delayed by one
step and used for activating this
or some other layer in the next
time step
• The Elman network can learn
sequences that cannot be
learned with other recurrent
neural network
23 Sep - 04 Oct, 2002
Feedback D-1
I1
I2
I3










O1
Om
In
Feedback D-1
Input Layer
Hidden Layer
Output Layer
Figure: Architecture of 3-layered ERNN
WSC7
8
(c) Radial Basis Function Network (RBFN)
• network consists of 3-layers:
input layer, hidden layer, and
output layer
• The neurons in hidden layer are
of local response to its input
and known as RBF neurons,
while the neurons of the output
layer only sum their inputs and
are called linear neurons
• network is inherently well
suited for weather prediction,
because it naturally uses
unsupervised learning to cluster
the input data.
23 Sep - 04 Oct, 2002
Output =  wi i(x)
Pure linear

W0
W1
Adjustable weights wi
w0 = bias
Wn
W2
1
I1
2



n
I2
Adjustable centers ci
Adjustable spreads i
Im
Input
Layer
Figure: Architecture of RBFN
WSC7
9
3. EXPERIMENTAL SETUP
– Minimum Temperature (oC)
– Maximum Temperature(oC)
– Wind-Speed (km/hr)
Min. temperature
20.0
10.0
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
Dec
-10.0
Nov
0.0
Oct
• Observed Parameters
(most important):
Max. temperature
Sep
Sep 2000 – Aug 2001
Daily Temperature (oC)
• Vancouver, BC, Canada
• 1-yr data:
30.0
1 3031 6061 9091 120
121 150
151 180
181 210
211 240
241 270
271 300
301 330
331 360
361
Time of the Year
100
Wind speed (km/hr)
Weather Data:
80
60
40
20
Time of the Year
23 Sep - 04 Oct, 2002
WSC7
10
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
Dec
Nov
Oct
Sep
0
Training and Testing Datasets
• Dataset 1: MLP and ERNN
– Testing dataset = 11-20 January 2001
– Training dataset = remaining data
• Dataset 2: RBFN, MLP and ERNN
– Testing dataset = 01-15 April 2001
– Training dataset = remaining data
• We used this above method to ensure that there is no bias
on the training and test datasets
23 Sep - 04 Oct, 2002
WSC7
11
Simulation System Used
• Pentium-III, 1GHz processor 256 MB RAM
• all the experiments were simulated using MATLAB
Steps taken before starting the training process:
• Error level was set to a value (10-4)
• The hidden neurons were varied (10-80) and the
optimal number for each network were then decided.
23 Sep - 04 Oct, 2002
WSC7
12
4. TEST RESULTS
Training Convergence of MLP and ERNN
1
1
10
10
0
0
10
10
-1
-1
10
Mean Square Error
Mean Square Error
10
-2
10
-3
10
Goal = 0.0004
-4
-5
-3
10
Goal = 0.0004
-4
10
10
-2
10
10
Levenberg-Marquardt
Iterations = 7
One-Step-Secant
Iterations = 1,015
-5
10
-6
Levenberg-Marquardt
Iterations = 11
One-Step-Secant
Iterations = 673
-6
10
10
Number of Iterations
Number of Iterations
Convergence of the LM and OSS training
algorithms using MLP network
23 Sep - 04 Oct, 2002
Convergence of the LM and OSS training
algorithms using ERNN
WSC7
13
Comparison of Actual vs. 10-day ahead Forecasting
using OSS and LM approaches
(a) Minimum Temperature (11-20 Jan 2001)
ERNN
MLP network
15
Minimum Temperature Forecasting
10
Actual
MLP-OSS
MLP-LM
5
0
-5
Temperature (oC)
Temperature (oC)
15
Minimum Temperature Forecasting
10
Actual
ERNN-OSS
ERNN-LM
5
0
-5
0
2
4
6
Days of the Month
Performance evaluation
parameters (min. temperature)
Mean absolute % error (MAPE)
Root mean square error (RMSE)
Mean absolute deviation (MAD)
Correlation coefficient
Training time (minutes)
23 Sep - 04 Number
Oct, 2002
of iterations (epochs)
8
10
0
2
4
6
Days of the Month
MLP Network
OSS
LM
0.0221
0.0202
OSS
0.0182
LM
0.0030
0.0199
0.7651
0.9657
0.3
WSC7
1015
0.0199
0.7231
0.9826
0.3
673
0.0031
0.1213
0.9998
7
11
0.0199
0.8411
0.9940
1
7
8
10
ERNN
14
(b) Maximum Temperature (11-20 Jan 2001)
ERNN
MLP network
15
Maximum Temperature Forecasting
Temperature (oC)
Temperature (oC)
15
10
5
0
Actual
MLP-OSS
Maximum Temperature Forecasting
10
5
0
Actual
MLP-LM
-5
ERNN-OSS
ERNN-LM
-5
0
2
4
6
Days of the Month
8
Performance evaluation
parameters
10
0
2
4
6
Days of the Month
8
Mean absolute % error (MAPE)
MLP Network
OSS
LM
0.0170
0.0087
ERNN
OSS
LM
0.0165
0.0048
Root mean square error (RMSE)
Mean absolute deviation (MAD)
Correlation coefficient
Training time (minutes)
Number of iterations (epochs)
0.0200
0.8175
0.964
0.4
850
0.0199
0.7944
0.945
1.8
1135
23 Sep - 04 Oct, 2002
WSC7
0.0099
0.4217
0.999
30
7
10
0.0067
0.2445
0.982
30
10
15
(c) Maximum Wind-Speed (11-20 Jan 2001)
ERNN
MLP network
50
Maximum Wind-Speed Forecasting
40
Actual
MLP-OSS
MLP-LM
30
20
10
0
Wind-Speed (km/h)
Wind-Speed (km/h)
50
Maximum Wind-Speed Forecasting
40
Actual
ERNN-OSS
ERNN-LM
30
20
10
0
0
2
4
6
Days of the Month
Performance evaluation
parameters (wind-speed)
8
10
0
2
4
6
Days of the Month
Mean absolute % error (MAPE)
MLP Network
OSS
LM
0.0896
0.0770
OSS
0.0873
LM
0.0333
Root mean square error (RMSE)
Mean absolute deviation (MAD)
Correlation coefficient
Training time (minutes)
Number of iterations (epochs)
0.1989
0.8297
0.9714
0.3
851
0.0199
0.7618
0.9886
0.5
1208
0.0074
0.3126
0.9995
8
12
23 Sep - 04 Oct, 2002
WSC7
0.0162
0.6754
0.9974
1
8
8
10
ERNN
16
Comparison of Relative Percentage Error
between Actual and Forecasted Parameters
MLP network
5
0
0
10
20
30
-5
Time (days)
-10
0
0
10
5
0
0
10
20
30
-5
Time (days)
5
0
0
10
40
40
0
0
23 Sep - 04 Oct, 2002
10
20
-20
Time (days)
WSC7
30
Relative Error (%)
Relative Error (%)
-10
20
20
30
-5
-10
-40
30
-5
Time (days)
Wind-Speed
20
10
Relative Error (%)
Relative Error (%)
5
-10
10
Maximum
Temperature
ERNN
10
Relative Error (%)
Minimum
Temperature
Relative Error (%)
10
Time (days)
20
0
0
10
20
30
-20
-40
Time (days)
17
Comparison of Training of Connectionist Models
Network model
Number of
hidden neurons
Number of
hidden layers
Activation function used
in hidden layer
Activation function used in
output layer
MLP
45
1
Log-sigmoid
Pure linear
ERNN
45
1
Tan-sigmoid
Pure linear
RBFN
180
2
Gaussian function
Pure linear
23 Sep - 04 Oct, 2002
WSC7
18
15
o
Temperature ( C)
Comparison among three Neural Networks
Techniques for 15-day ahead Forecasting (1-15 Apr 2001)
Maximum
Temperature
10
5
Actual value
RBFN
MLP
RNN
0
0
3
6
Minimum
Temperature
Temperature ( oC)
15
9
12
15
Days of the Month
10
5
Wind-Speed
Wind Speed (km/h)
0
60 0
3
6
9
12
15
Days of the Month
40
20
0
0
3
6
9
12
15
Days of the Month
23 Sep - 04 Oct, 2002
WSC7
19
Performance Evaluation of
RBFN, MLP and ERNN Techniques
Model
RBFN
Performance
Evaluation
Parameters
MAP
MAD
Correlation Coefficient
MLP
MAP
MAD
ERNN
MAP
MAD
23 Sep - 04 Oct, 2002
Correlation Coefficient
Correlation Coefficient
Maximum
Temperature
Minimum
Temperature
Wind Speed
3.821
0.420
0.987
3.622
1.220
0.947
4.135
0.880
0.978
6.782
1.851
0.943
6.048
1.898
0.978
6.298
1.291
0.972
5.802
0.920
0.946
5.518
0.464
0.965
5.658
0.613
0.979
WSC7
20
5. CONCLUSIONS
• In this paper, we developed and compared the
performance of multi-layered perceptron (MLP)
neural network, Elman recurrent neural network
(ERNN) and radial basis functions network
(RBFN).
• It can be inferred that ERNN could yield more
accurate results, if good data selection strategies,
training paradigms, and network input and output
representations are determined properly.
23 Sep - 04 Oct, 2002
WSC7
21
• Levenberg-Marquardt (LM) approach appears to
be the best learning algorithm. However, it
requires more memory and is computationally
complex while compared to one-step-secant (OSS)
algorithm.
• Empirical results clearly demonstrate that
compared to MLP neural network and ERNN,
RBFN are much faster and more reliable for the
weather forecasting problem considered.
• A comparison of the neurocomputing
techniques with other statistical techniques
would be another future research topic.
23 Sep - 04 Oct, 2002
WSC7
22
THANK YOU !
23 Sep - 04 Oct, 2002
WSC7
23