Download Neural Networks

Department of Computer and Information Science
Neural Nets for Data Mining
CISC 6930 Data Mining
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Outline
 Neural Networks: Background
 Neural Network Classifier
 ANN Architecture
 Strength and Weakness of ANN
 Applications
2
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Nearest Neighbor Classifier
o Data Mining
 Common data mining tasks






3
Classification [Predictive]
Clustering [Descriptive]
Association Rule Discovery [Descriptive]
Sequential Pattern Discovery [Descriptive]
Regression [Predictive]
Deviation Detection [Predictive]
Classifiers
•
•
•
•
•
•
•
•
•
•
•
•
Data Mining | Neural Network Classifier
Decision Trees
Rule Approaches
Logical statements (ILP)
Bayesian Classifiers
Nearest Neighbor Learning
Neural Networks
Discriminant Analysis
Support Vector Machines
Logistic regression
Artificial Neural Networks
Genetic Classifiers
...
Department of Computer and Information Science
Learning Objectives
o Learn the step-by-step process of how to use NN for
data mining
o Understand a variety of applications of NN, solving
problem types of
 Classification
 Regression
 Clustering
 Predicition
4
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Neural Networks: Background
The first learning algorithm came in 1959 (Rosenblatt)
who suggested that if a target output value is provided for
a single neuron with fixed inputs, one can incrementally
change weights to learn to produce these outputs using the
perceptron learning rule
5
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Neural Networks: Background
o What is NN?
 Biologically motivated approach to machine learning
o Similarity with biological network
 Indeed a great example of a good learning system
6
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Neural Networks: Background
o What is NN?
 Biologically motivated approach to machine learning
o Similarity with biological network
 Fundamental processing elements of a neural network is a
neuron
Dendrites
Synapse
Synapse
Axon
Soma
7
Dendrites
Data Mining | Neural Network Classifier
• A human brain
has 100 billion
neurons
• An ant brain has
Axon
250,000 neurons
• Synapses, the
Soma
basis of learning
and memory
Department of Computer and Information Science
Neural Networks: Background
 NNs is a set of connected INPUT/OUTPUT UNITS, where
each connection has a WEIGHT associated with it.
 NNs learning is also called CONNECTIONIST learning due
to the connections between units.
 It is a case of SUPERVISED, INDUCTIVE or
CLASSIFICATION learning.
Dendrites
Synapse
Synapse
Axon
Axon
Soma
8
Dendrites
Data Mining | Neural Network Classifier
Soma
Department of Computer and Information Science
Neural Networks: Background
Biology Analogy
9
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Outline
 Neural Networks: Background
 Neural Network Classifier
 ANN Architecture
 Strength and Weakness of ANN
 Applications
10
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Neural Network Classifier
o Input: Classification data
 It contains classification attribute
 Data is divided, as in any classification problem.
[Training data and Testing data]
o All data must be normalized
 i.e. all values of attributes in the database are changed to
contain values in the internal [0,1] or[-1,1]
Neural Network can work with data in the range of (0,1) or (-1,1)
o Basic normalization techniques for data
classification
 Max-Min normalization
 Decimal Scaling normalization
11
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Data Normalization
o Min-max normalization
x  min x
z
( New _ maxx  New _ minx )  New _ minx
maxx - minx
73600  56200
z
(1  0)  0  0.56
87000  56200
Consider employees income range
between $56200 to $87000. If
this range is normalized to [0, 1],
what is the B’s normalized salary?
12
Name
Gender
Salary
A
M
87000
B
F
73600
C
M
65000
D
M
76000
E
F
56200
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Data Normalization
o Decimal Scaling Normalization
 Normalization by decimal scaling normalizes by moving
the decimal point of values of attribute A.
v' 
v
10 j
Here j is the smallest integer such that max|v’|<1.
Example :
A – values range from -986 to 917.
Max |v| = 986.
v = -986 normalize to v’ = -986/1000 = -0.986
13
Data Mining | Neural Network Classifier
Department of Computer and Information Science
One Neuron as a Network
An artificial neuron is a mathematical function
conceived as a model of biological neurons. Artificial
neurons are the constitutive units in an artificial
neural network.

Here, x1 and x2 are normalized attribute value of data.
y is the output of the neuron , i.e the class label.

Value of x1 is multiplied by a weight w1 and values of x2 is multiplied by a weight w2.


Given that
•
•
14
w1 = 0.5 and w2 = 0.5
Say value of x1 is 0.3 and value of x2 is 0.8,
So, weighted sum is :
sum= w1 x x1 + w2 x x2 = 0.5 x 0.3 + 0.5 x 0.8 = 0.55
Data Mining | Neural Network Classifier
Department of Computer and Information Science
One Neuron as a Network
An artificial neuron is a mathematical
function conceived as a model of biological
neurons.
Artificial neurons are the constitutive weight units in an
artificial neural network (ANN).
15
Data Mining | Neural Network Classifier
Department of Computer and Information Science
One Neuron as a Network
•
The neuron receives the weighted sum as input and calculates the
output as a function of input as follows :
y = f(x) , where f(x) is defined as
f(x) = 0 { when x< 0.5 }
f(x) = 1 { when x >= 0.5 }
•
For our ex ample, x ( weighted sum ) is 0.55, so y = 1 ,
That means corresponding input attribute values are classified in class 1.
If for another input values , x = 0.45 , then f(x) = 0, so we could
conclude that input values are classified to class 0.

16
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Outline
 Neural Networks: Background
 Neural Network Classifier
 ANN Architecture
 Strength and Weakness of ANN
 Applications
17
Data Mining | Neural Network Classifier
Department of Computer and Information Science
ANN Architecture
o Formally, ANN is specified by:
 Neuron model
 ANN is a machine learning approach that models human brain and consists
of a number of artificial neurons.
 Each neuron in ANN receives a number of inputs.
 An architecture
 A set of neurons and links connecting neurons.


Each link has a weight
Neuron tends to have fewer connections than biological neurons.
 A learning algorithm
 It used for training the NN by modifying the weights in order


18
An activation function is applied to these inputs which results in activation level of
neuron (output value of the neuron).
Knowledge about the learning task is given in the form of examples called training
examples.
Data Mining | Neural Network Classifier
Department of Computer and Information Science
ANN Architecture
o Formally, ANN is specified by:
 Neuron model
 ANN is a machine learning approach that models human brain and consists
of a number of artificial neurons.
 Each neuron in ANN receives a number of inputs, e.g., x1, x2…xn
 An architecture
 A set of neurons and links connecting neurons.


m
u   wjxj
Each link has a weight
j 1
A set of links, describing the neuron inputs, with weights W1, W2, …, Wm

 A learning algorithm
 It used for training the NN by modifying the weights in order


19
An activation function is applied to these inputs which results in activation level of
neuron (output value of the neuron).
y   (u  b)
For limiting the amplitude of the neuron output. Here ‘b’ denotes bias.
Data Mining | Neural Network Classifier
Department of Computer and Information Science
How Does the ANN Learn?
o A neural network learns by determining the relation
between the inputs and outputs.
 By calculating the relative importance of the inputs and
outputs the system can determine such relationships.
 Through trial and error the system compares its results
with the expert provided results in the data until it has
reached an accuracy level defined by the user.
 With each trial the weight assigned to the inputs is
changed until the desired results are reached.
20
Data Mining | Neural Network Classifier
Department of Computer and Information Science
A Single Layer ANN
We need the bias value to be added to the weighted sum ∑wixi so that
we can transform it from the origin.
v = ∑wixi + b, here b is the bias

x0 = +1
x1
21
induced field of the neuron
W1

x2
w2
xm
wm

Input
Attribute
values
w0
v
 ( )
y

weights
Summing
function
Data Mining | Neural Network Classifier
Activation
function
Output
class
Department of Computer and Information Science
A Single Layer ANN
We need the bias value to be added to the weighted sum ∑wixi so that
we can transform it from the origin.
v = ∑wixi + b, here b is the bias
m

x0 = +1
x1
22
j 0
W1

x2
w2
xm
wm

Input
Attribute
values
v   w j x j  w0
w0
v
w0  b
 ( )
y

weights
Summing
function
Data Mining | Neural Network Classifier
Activation
function
Output
class
Department of Computer and Information Science
Multi-Layer Perceptron
Output Class
Output nodes
w jk
Hidden nodes
wij weights
Input nodes
Network is fully connected
Input Record :
23
xi
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Single Layer vs. Multi Layers
Output layer
Input layer
Output layer
Input layer
Hidden Layer
3-4-2 Network
24
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Network Training
o Backpropagation algorithm
o The ultimate objective of training
 Obtain a set of weights that makes almost all the tuples in
the training data classified correctly
o Steps
 Initialize weights with random values
 Feed the input tuples into the network one by one
 For each unit
 Compute the net input to the unit as a linear combination of all the inputs
to the unit
 Compute the output value using the activation function
 Compute the error
 Update the weights and the bias
25
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Network Pruning and Rule Extraction
o Network pruning
 Fully connected network will be hard to articulate
 N input nodes, h hidden nodes and m output nodes lead to
h(m+N) weights
 Pruning

26
Remove some of the links without affecting classification accuracy of the
network
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Outline
 Neural Networks: Background
 Neural Network Classifier
 ANN Architecture
 Strength and Weakness of ANN
 Applications
27
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Strength of ANN
o ANN has a high tolerance to noisy and incomplete
data
o Massive parallelism allowing for computational
efficiency
o Autonomous learning and generalization
o Able to deal with (identify/model) highly nonlinear
relationships
o Usually provides better results (prediction and/or
clustering) compared to its statistical counterparts
28
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Weakness of ANN
o Training may take a long time for large datasets;
which may require case sampling
o It is hard to find optimal values for large number of
network parameters
o Optimal design is still an art: requires expertise and
extensive experimentation
o It is hard to handle large number of variables
29
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Outline
 Neural Networks: Background
 Neural Network Classifier
 ANN Architecture
 Strength and Weakness of ANN
 Applications
30
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Application-I
o Handwritten Digit Recognition
o Face recognition
o Time series prediction
o Process identification
o Process control
o Optical character recognition
31
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Application-II
o Forecasting/Market Prediction: finance and
banking
o Manufacturing: quality control, fault diagnosis
o Medicine: analysis of electrocardiogram data, RNA
& DNA sequencing, drug development without
animal testing
o Control: process, robotics
32
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Data Mining Software Supporting ANN
o PASW (formerly SPSS Clementine)
o SAS Enterprise Miner
o Statistica Data Miner, … many more …
33
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Reference
o Chapter 7.5
o Professor Anita Wasilewska’s lecture note,
www.cs.vu.nl/~elena/slides03/nn_1light.ppt
o Xin Yao Evolving Artificial Neural Networks
o
o
o
o
o
http://www.cs.bham.ac.uk/~xin/papers/published_iproc_sep99.pdf
informatics.indiana.edu/larryy/talks/S4.MattI.EANN.ppt
www.cs.appstate.edu/~can/classes/5100/Presentations/DataMining1.ppt
www.comp.nus.edu.sg/~cs6211/slides/blondie24.ppt
www.public.asu.edu/~svadrevu/UMD/ThesisTalk.ppt
www.ctrl.cinvestav.mx/~yuw/file/afnn1_nnintro.PPT
34
Data Mining | Neural Network Classifier
Department of Computer and Information Science
Q&A
35
Data Mining | Neural Network Classifier