Download Neural Networks




Neural Networks
•
•
•
•
•
•
•
•
•

Introduction – What is a neural network and what can it
do for me?
Terminology, Design and Topology
Data Sets – When too much is not a good thing
The Back Propagation and other training methods
Network Pruning
Network Development - Hints and kinks, practical
application notes
Examples of Neural Networks
Closed loop control techniques
The Neural Network Laboratory
Introduction to DeltaV, HYSYS and the OLE connection
Develop a soft sensor application
Neural Network Course Objectives
●
Neural Network Fundamentals, Structure,
Training, Testing
●
How to use data to Train and Test the network
●
Develop DeltaV Neural Networks
●
Laboratory in DeltaV
●
Design Neural Networks in MATLAB and
EXCEL
Neural Networks, Heuristic
●
●
Heuristic vs. Deterministic Model Development,
NN are Heuristic Models, similar to statistics
HYSYS, ASPEN, etc are deterministic models,
using first principals
For DeltaV Neural Nets
●
●
●
Understand basic principals of the DeltaV Neural
Network
Construct a DeltaV NNet and Lab Entry function
blocks
Configure the operator interface for these
function blocks
DeltaV Neural
●
DeltaV Neural is a complete package
●
Network Development
●
Training, Testing
●
Operator Interface, both Neural Network and Lab
Entry
DeltaV Neural Applications
●
●
Critical process measurements available from
grab samples, paper properties, food properties,
etc.
Backup or cross check on a measurement by a
sampled or continuous analyzer, mass spec, stack
gas analyzer.
–
Hint consider the “cost” of a sample analysis, can the
neural network “fill in” for skipped samples?
DeltaV Neural Applications
●
●
●
Virtual sensors; neural network can be “trained”
to calculate the laboratory results, a frequent
application, a.k.a. “intelligent sensors” or “soft
sensors”, ISTK
This information is used by operators to
anticipate changes in plant operation
Can develop the network in much less time than
first principal methods
Example, paper mill physical property
VIRTUAL STFI SENSOR
Bot. Ply
Sec. Screen (1)
To Bottom
Ply Silo
#1 HD
Bottom
Hole (1)
Refiner
Base Raw
Stock Chest
Bottom
Refiner
Chest
Bottom
Tickler (1)
Refiner
Bottom
Machine
Chest
Bottom Ply
Stuffbox
Bot. Ply
Pri. Screen (1)
Mid
Ply
Scrn
Rej.
Tank
Sweetener Stock
to Saveall
To
Primary
Headbox
Rejects to #2 PM
Mach. WW Chest
Mid Ply Sec. Screen (1)
#2 HD
[Parallel]
Mid
Hole (1)
Refiner
Broke
Storage
#3 HD
Top Raw
Stock Chest
Top
Hole (1)
Refiner
Mid
Tickler (2)
Refiner
Mid
Refiner
Chest
Mid
Machine
Chest
OCC
Raw Stock
Chest
Top
Refiner
Chest
Mid Ply
Stuffbox
To
Primary
Headbox
Mid Ply Pri. Screen (3)
To
Mid Ply
Screen Rejects
Tank
Top
Tickler (1)
Refiner
To
Sec.
Headbox
Top Ply
Sutffbox
Pri.
CleanersSec.
To
Mid Ply
Screen Rejects
Tank
Cleaners Tertiary
Cleaners
Quat.
Cleaners
Predict the lab results
Just what is a neural network
anyway?
●
●
●
●
Neural networks are computer programs that
model the functionality of the brain.
Multi layered feed forward model, either single or
multiple outputs
Trained (in statistics called regression) by back
propagation or other algorithms
Non Linear Regression
The Neuron
●
●
There are estimates of as many as 100 billion neurons in the
human brain. They are often less than 100 microns in diameter and
have as many as 10,000 connections to other neurons.
Each neuron has an axon that acts as a wire for all connections to
the other cell's neurons. The neurons have input paths which are
called dendrites which gather information from these axons. The
connection between the dendrites and the axon is called a synapse.
The transmission of signals across the synapse is chemical in
nature and the magnitude of the signal depends on the amount of
chemicals (called neurotransmitter) released by the axon. Many
drugs work on the basis of changing those natural chemicals. The
synapse combined with the processing of information in the
neuron is how the brain's memory process functions.
The neuron is a nerve cell
Dendrites
Axon
Synapses
Brain – Neural Network Analogy
Brain
Neuron
Neural firing rate
Synaptic strength
Synapse
Neural Networks
Cell
Activation
Connection weight
Connection
Cell body that contains the nucleus
Dendrites or Inputs that connect impulses to the cell body
Axon conducts impulses away from the cell body
Synapses are gaps between neurons that act as outputs and are closely related
to the input to adjoining dendrites. This connection is chemical in nature and that
chemical concentration is how the weight is determined.
Within the cell the inputs are weighed, that is given weight to their strength. If the
weight is strong enough, the neuron “fires”, or an output is triggered.
Brain – Neural Network Analogy
Neural Network,
Under the hood
●
Layer approach
–
Input Layer
–
Hidden Layer (or Layers)
–
Output Layer
N1/N2/N3
Notation
DeltaV Neural
Three Layered Network
●
Only one “hidden” Layer
●
Only one output
●
●
With enough hidden layers can represent any
continuous non-linear function
Can track either a single continuous variable or a
sampled variable, Lab Analysis
Neural Networks – Hidden Layer
●
●
1 Hidden layer sufficient to model a continuous
function of several variables
Exception to the rule: Inverse action requires 2
layers
DeltaV Neural - Output
●
●
●
DeltaV Neural Network is designed for one
output
Why?
 of errors will not properly distribute with more
than one output



Network Structure

●
Inputs and Scaling
●
Synaptic Weights
●
Neuron, Summation and Transfer Function
●
Layer Concept, input, hidden and output
●
Output scaling and the Outputs
Network Structure – Input/Output
Scaling
●
●
PV ranges must be normalized so each variable
has the same input factor for presentation to the
network.
Scaling around zero
Scaled PV = (PV – mean)/s
●
s is standard deviation
The Network is a Collection of
Neurons and Weights
●
Multi-layered feed forward network (MFN)
●
Bias Neurons
–
Connected to each neuron except the input later
–
Provide a constant value or “bias” to the network
The Neuron
sigmoidal function, tanh
y=1-(2/(exp(2*x)+1))
1.0000
0.5000
-2
-1.5
-1
0.0000
-0.5
0
-0.5000
-1.0000
0.5
1
1.5
2
DeltaV - Building the Network
Data Collection – The process uses data to design
the network, good data is essential
Data Preprocessing – Remove outliers and missing
points. 3 sigma rule
Variable and time delay selection - Determines
which variable to use as well as the timing
Building the Network, continued
Network Training – Operation number of hidden
neurons, adjusts the weights on the conditioned
training set, learning the data
Network Verification – Checks how well the
network behaves against actual data
DeltaV - Training the Network
Divides the data in three sets, Training, Testing and
Verification
Presents Training data to the network
For each data set, presents the inputs, forward propagate
the training set through all layers and finally the output
Compares the output to the target value, adjust the weights
based on error, back-propagation
One training pass through all the data is called an epoch
Training the Network
Present the testing set data to the network after the
weights are adjusted in one epoch
Propagate the test signals through the network, to
the output
Compare the results, if small error exists, complete,
otherwise repeat the training process
Training the Network
●
●
●
Use “Balanced” Design, many points over the
total range of network inputs
Consider using techniques employed in design of
experiments, DOE
If most of the data is at one process point, it will
learn that point very well.
Gradient Descent Learning
●
Back propagation adjusts the weights to reduce
the error between the network and the training set
1
2
E   d pi  y pi 
2 p i
●
p is the pattern index i is the output nodes
indexes, d is the desired output and y is the
actual output.
Gradient Descent Learning
Process is:
a. Set weights to a random value
1. For each sample point in the training set, calculate
the output value and the error.
2. Calculate the derivative  E/  w
3. Adjust the weights to minimize the error
4. GOTO 1. until error decreases to the pre
determined value or the number of epochs exceeds
a pre determined value
Gradient Descent Learning
Two methods for weight update, batch mode
and on line mode.
Batch mode: The pattern partial error is
summed to obtain the total derivative:
E p
E

w
p wij
Gradient Descent Learning
●
On Line mode: Weights are updated based
on the partial of the error with respect to
weight based on one pattern or entry of test
values. This is implemented using a
momentum term. Momentum adds a portion
of the previous weight to the new change
E
wij (t )  awij (t  1)  
(t )
wij
0 < a < 0.9
Conjugate Gradient Method
●
DeltaV Neural uses Conjugate Gradient Method
●
Uses previous gradients
●
Adapts learning rate
●
No need to specify momentum or learning rate
factor
Training Criteria
●
●
Predict, not memorize the data presented
DeltaV neural training software cross validates
network against test set to locate lease test error,
will not over or under train
Verification of NN Accuracy
●
●
●
Compare predicted and actual values
Verification should be done on a set not used for
training and testing
It is very important that the data points in
verification data set is within the values
used to train and test the network. The
training set must contain the minimum
and maximum points!
DeltaV Neural ‘single-button’ Method
●
All the tools to develop a network are built in to
the DeltaV engineers workstation software
DeltaV Neural Network
●
Function blocks, similar to AI, PID, etc
●
Lab Entry block for entry of analytical data
DeltaV - Input Data for NN Training
●
●
●
Process data and lab analysis is collected when
the NN and Lab Entry blocks are downloaded
The NN application uses the data collected by the
DeltaV historian
Can input legacy data or data collected by another
system as a flat text file
DeltaV NN Block
●
Can Access Data anywhere within the control
system
●
Maximum of 20 references (30 in final release?)
●
3 Modes
–
Auto: Prediction of output based on the input
–
Manual: OUT can be set manually
–
Out of Service: OUT is set to a Bad status, no
calculations are executed
Neural Networks for Control
●
●
Using Neural Networks for feedfoward and
decoupling control interactions; an improvement
to conventional PID control
Using inverted networks for direct control, nonPID method