Download Neural Networks

ATSBA: Advanced Technologies Supporting Business Areas
Foundations of Optimization
6. Neural Networks
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
1
Content
1. Basic Terms and Definitions
2. McCulloch-Pitts Neuron Model
3. SNNS-Demo
4. Application Areas
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
2
1. Basic Terms and Definitions
What are Neural Networks?
– Neural Networks (NNs) are networks of neurons, for example, as found in real
(i.e. biological) brains.
– Artificial Neurons are crude approximations of the neurons found in brains.
They may be physical devices, or purely mathematical constructs.
– Artificial Neural Networks (ANNs) are networks of Artificial Neurons, and
hence constitute crude approximations to parts of real brains. They may be
physical devices, or simulated on conventional computers.
– From a practical point of view, an ANN is just a parallel computational system
consisting of many simple processing elements connected together in a specific
way in order to perform a particular task.
– One should never lose sight of how crude the approximations are, and how oversimplified our ANNs are compared to real brains.
[Source: http://www.cs.bham.ac.uk/~jxb/NN/l1.pdf, date 17 May 2011]
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
3
1. Basic Terms and Definitions - Biological Neurons
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
4
1. Basic Terms and Definitions
Why are Artificial Neural Networks worth studying?
– Extremely powerful computational devices, massive parallel
– They can learn and generalize from training and are particularly fault and noise
tolerant
What are Artificial Neural Networks used for?
– Brain Modelling: The scientific goal of building models of how real brains work.
(medicine, biology...)
– Artificial System Building: The engineering goal of building efficient systems
for real world applications. This may make machines more powerful, relieve
humans of tedious tasks, and may even improve upon human performance.
– [Extracts from source: http://www.cs.bham.ac.uk/~jxb/NN/l1.pdf, date 17 May 2011]
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
5
1. Basic Terms and Definitions
Some Current Artificial Neural Network Applications
– Brain modelling
– Models of human development – help children with developmental problems
– Simulations of adult performance – aid our understanding of how the brain works
– Neuropsychological models – suggest remedial actions for brain damaged patients
Real world applications
– Financial modelling – predicting stocks, shares, currency exchange rates
– Other time series prediction – climate, weather, airline marketing tactician
– Computer games – intelligent agents, backgammon, first person shooters
– Control systems – autonomous adaptable robots, microwave controllers
– Pattern recognition – speech recognition, hand-writing recognition, sonar signals
– Data analysis – data compression, data mining, PCA, GTM
– Noise reduction – function approximation, ECG noise reduction
– Bioinformatics – protein secondary structure, DNA sequencing
[Extracts from source: http://www.cs.bham.ac.uk/~jxb/NN/l1.pdf, date 17 May 2011]
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
6
1. Basic Terms and Definitions
Learning in Neural Networks
– There are many forms of neural networks. Most operate by
passing neural ‘activations’ through a network of connected
neurons.
– One of the most powerful features of neural networks is their
ability to learn and generalize from a set of training data.
– They adapt the strengths/weights of the connections between
neurons so that the final output activations are correct.
There are three broad types of learning:
1. Supervised Learning (i.e. learning with a teacher)
2. Reinforcement learning (i.e. learning with limited feedback)
3. Unsupervised learning (i.e. learning with no help)
[Extracts from source: http://www.cs.bham.ac.uk/~jxb/NN/l1.pdf, date 17 May 2011]
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
7
1. Basic Terms and Definitions
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
8
1. Basic Terms and Definitions
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
9
2. McCulloch-Pitts Neuron Model
The McCulloch-Pitts Neuron Model is a Threshold Logic Unit:
– A set of synapses (i.e. connections) brings in activations from other neurons.
– A processing unit sums the inputs, and then applies a non-linear activation function
(i.e. squashing/transfer/threshold function).
– An output line transmits the result to other neurons.
[Extracts from source: http://www.cs.bham.ac.uk/~jxb/NN/l2.pdf, date 17 May 2011]
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
10
2. McCulloch-Pitts Neuron Model
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
11
3. Stuttgart Neural Network Simulator (SNNS)
URL: http://www.ra.cs.unituebingen.de/SNNS/
JavaNNS:
http://www.ra.cs.unituebingen.de/downloads/JavaNNS/
java -jar JavaNNS.jar
JRE > 1.3 necessary
Select the zip-File and unzip it on temp.
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
12
SNNS-File extensions
.net: designed Neural Network
.pat: Pattern-File
.res: Result-file (trained net)
.cfg: Configuration-File
.readme: Info about the example
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
13
Step 1: Define the net topology
Define the layer of the Multi Layer Perceptron
Define the connection between the layers
Connections:
usually feed-forward
Result of step 1:
.net-File
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
14
Step 2: Define the training set
.pat-File
Text-file which is parsed by the SNNS
Sequence of input- and targeted
output patterns for training
(continuous or binary values)
# lines are ignored by the parser
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
15
Step 3: Training
Computing output values
of the pattern neurons
Initializing the network
Settings for Learning
Control panel (CTRL + C)
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
17.05.2011
16
Step 3: Training - Initializing
Randomize
weights
Min-Max values, usually
[-1, +1]
At the beginning the weights are initialized with randomized weights
Source: Rainer Telesko: Introduction into Neural
(= no knowledge is present).
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
17
Step 3: Training - Learning
Generalized Delta
Rule (GDR)
acceptable error
(difference between
target and actual output)
Learning rate
Training cycles
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
Training patterns are
selected randomly
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
17.05.2011
18
Step 4: Testing
Validating the net with previously
untrained examples
Saving the
Result (net +
weights) in
a .res-file
Here: Training pattern set =
Validation pattern set (Memorization)
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
19
Exercise – SNNS
Solve the following problem with a Neural Network.
– Based on input criteria (e.g. content of a module, level of difficulty of a module, preference for a
specific module, teacher etc.) you should calculate the chance to pass module exams in the Msc
BIS.
Specify the input neurons and the output neuron and the neuron values (binary / continuous
values)
Use one hidden layer in the MLP (This is sufficient.)
Consider training examples in the MSc BIS area (Try to cover a lot of different situations, then
the generalization is better.)
Set up the net and train it.
Test the net for a previously untrained case (e.g. ATSBA module, …).
Source: Rainer Telesko: Introduction into Neural
Networks; MSc BIS, Module ATSBA; University of
Applied Sciences Northwestern Switzerland; Winter
Term 2009/2010.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
20
Learning Targets
To be able to …
– explain the functionality of Artificial Neural Networks more in detail,
– designate the elements of Artificial Neuron,
– designate the elements of Artificial Neural Networks.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
21
6. Neural Networks
References:
– Rainer Telesko: Introduction into Neural Networks; MSc BIS, Module ATSBA;
University of Applied Sciences Northwestern Switzerland; Winter Term
2009/2010.
– Wikipedia: Neural Networks; http://en.wikipedia.org/wiki/Neural_networks; Date:
17 Mai 2011.
– John A. Bullinaria: Introduction to Neural Networks; 2nd Year UG, MSc in
Computer Science; http://www.cs.bham.ac.uk/~jxb/inn.html; The University of
Birmingham; 2004.
© Prof. Dr. Rolf Dornberger
-
ATSBA: Advanced Technologies Supporting Business Areas
17.05.2011
22