Download Application of recurrent network model on dynamic

Application of Recurrent Neural Network in Dynamic Modeling of Sensors
SHI MENG,TIAN SHEPING, JIANG PINGPING, YAN GUOZHENG
Department of Information Measurement Technology and Instruments
Shanghai Jiaotong University
Room 2201,Building 5,1915 Zhen Guang Rd.Shanghai,200333
CHINA
Abstract: - Dynamic modeling of sensors is an important aspect in the field of instrument technique. The recurrent neural
network is proposed for nonlinear dynamic modeling of sensors,as its architecture is determined only by the number of
nodes in the input, hidden and output layers. With the feedback behavior, the recurrent neural network can catch up with the
dynamic response of the system. A recursive prediction error algorithm, which converges fast, is applied to training the
recurrent neural network. Experimental results show that the the performance of the recurrent neural network model
conforms to the sensor to be modeled, proving the method is not only effective but of high precision.
Key-Words: - Recurrent neural network; Sensor; Dynamic modeling; Recursive prediction error algorithm; Dynamic
response; Mechanical sensor
1
Introduction
It is an important aspect to construct the model of a
sensor with the dynamic calibrating data in the field of
dynamic measurement[1]. Several methods have been
applied to dynamic modeling of sensors. A common
approach is to find the difference equation of a sensor
using the discrete domain calibrating data.
The
transfer function of the sensor can be gotten from
corresponding difference equation through bilinear
transform. Above method is suitable for linear dynamic
modeling of sensors and is useless when the sensors
display nonlinear characteristics.
Several artificial neural network paradigms and
neural learning schemes have been used in many
dynamic system identification problems, and many
promising results are reported. Most people made use of
the feedforward neural network, combined with tapped
delays, and the back propagation training algorithm to
solve the dynamical problems[2]; however, the
feedforward network is a static mapping and without the
aid of tapped delays it does not represent a dynamic
system mapping. On the other hand, the recurrent neural
networks have important capabilities that are not found
in feedforward networks, such as the ability to store
information for later use and higher predicting precision.
Thus the recurrent neural network is a dynamic mapping
and is better suited for dynamical systems than the
feedforward network.
This paper discusses the application of the recurrent
neural networks in nonlinear dynamic modeling of
sensors. With the feedback behavior, the recurrent
neural network can capture the dynamic response of the
system. A recursive prediction error algorithm, which
converges fast, is applied to training the recurrent neural
network. Experimental results show that the dynamic
modeling method is effective.
2
Dynamic modeling of sensors based
on recurrent neural network model
2.1 Recurrent neural network model
The fully connected recurrent neural network,
however, where all neurons are coupled to one another,
is difficult to train and to make it converge in a short
time. With the requirement of fewer weights and a
shorter training time for the neural network model, a
simplified recurrent neural network is proposed. The
architecture of the simplified version is a modified
model of the fully connected recurrent neural network.
It normally has an input layer, an output layer, and one
hidden layer. The hidden layer is comprised of
self-recurrent neurons, each feeding its output only into
itself and not to other neurons in the hidden layer. On
the other hand, the neurons in the output layer are linear
type neurons, and do not have feedback weights. It was
shown that with a proper choice of input and output
weights, the simplified recurrent neural network
paradigm can be made equivalent to a fully connected
recurrent neural network. As shown in Fig.1, the
recurrent neural network’s characteristics are
determined by the connected weights of neighboring
layers.
W ijI
W jD
W jO
I
ij
and nodes in hidden layer respectively. For a recurrent
network with single output, it has (p+2) q weights. g(.)
is the activation function of neurons which is usually
chosen as the sigmoid function:
g ( z) 
I i (t )W W O j (t )
(3)
If the input of the recurrent network is denoted as
{u (t-1), y’ (t-1)}, output is y’ (t), the input –output
mapping of the recurrent neural network can be
represented by:
y' (t )  f [u (l ), y' (l ), l  t  1]
(4)
Where f (.) is a nonlinear function. So f (.) is a nonlinear
dynamic mapping.
Fig. 2 shows the framework of dynamic modeling
of sensors based on recurrent neural network model, in
which y’ (t) is the output of the recurrent neural network
model. The determination of weights can be achieved by
applying certain training algorithm.
u(t)
O
j
1
1  ez
y(t)
sensor
+
O (t )
-
…
…
I i (t )
y’(t)
-1
z
Fig. 1 A recurrent neural network model
Each neuron in the hidden layer is a recurrent one
Fig. 2
Dynamic modeling of sensors based on
recurrent neural network model
with a nonlinear activation function. I i (t ) is the ith
input; Sj (t), Xj (t) are the sum of input and output of the
jth neuron respectively; O (t) represents the output of
the whole network. The weighting vectors of the input
layer, hidden layer and output layer compose the vector
W={wI, wD, wO}. So the relation between the input and
output can be described as:
q
O(t )  W jO X j (t )
X j (t )  g[ S j (t )]
j 1
(1)
p
S j (t )  W jD X j (t  1)  WijI I i (t )
(2)
i 1
Where p and q refer to the numbers of network inputs
2.2 Training algorithm of the recurrent neural
network
Several methods such as back propagation
algorithm can be used to estimate the weights of a
recurrent network model. In this paper we utilize the
recursive prediction error algorithm (RPE) to train the
recurrent network. Prediction error method achieves the
parameters’ estimates by minimizing the prediction error
criterion. First, let’s define the prediction error as:
 t,W   yt   y' t,W 
(5)
After N data have been recorded, a criterion function
can be expressed by the following sum of squared
prediction errors:
J (W ) 
N
1
2N

T
(t , w)(t , w)
the Gauss-Newton search direction of J (W ) to make
J→min . The basic equation is:
W t   W t  1  st W t  1 (7)
Where s (t) is the step size, µ (w) represents the
Gauss-Newton search direction.
Where
J W 
1 N
   t , W t , W  (9)
W
N t 1
 dy ' t , W  
t , W   
 dW 
pt  

N
 t ,W  t ,W 
T
(11)
t 1
(12a)



1
1
pt  1  pt  1t  t I   T t  pt  1t   T t  pt  1
t 
W t   W t  1  pt t t 
y' (t )
 W jO Pj (t )
W jD
(14b)
y ' (t )
 W jO Qij (t )
I
Wij
(14c)
Pj (t ) 
X j (t )
W j
D
, Q ij (t ) 
X j (t )
Wij
I
and meet the following conditions:
D

Pj (t )  X j (t  1)  W j Pj (t  1), Pj (0)  0

D

Qij (t )  I i (t  1)  W j Qij (t  1), Qij (0)  0
(15a)
(15b)
(10)
The RPE algorithm based on above rules is
described by following equations[3,4]:
t   yt   y' t 
(14a)
T
H (w) can be calculated from the expression as follows:
1
H W  
N
y' (t )
 X j (t )
W jO
In which Pj (t), Qij (t) are defined as:
(8)
Here the gradient of J(w) towards w is denoted as ▽
J(w), H(w) is the second order derivative of J(w),
namely the Hessian matrix of J(w).
It can be easily derived out that:
J W  
(13)
ψ(t) can be obtained by equation :
The unknown weighing vectors are updated along
1
 t   0  t  1  1  0 
(6)
i 1
W   H W  J W 
the above requirements:
(12b)
(12c)
p (t) is called the middle matrix, representing the
covariance matrix of parameters when t→∞, whose
initial value p (0) is usually chosen from the range of
104I to 105I, where I is the identity matrix.λ(t)is called
the forgetting factor. It’s desirable to setλ(t)<1 at the
initial stage so that rapid adaptation takes place and then
to letλ(t)→1 as t→∞. Following equation can meet
3
Experimental results
From above analysis, the process of applying
recursive network to dynamic modeling of a sensor can
be summarized as follows:
1. According to the performance of the sensor,
determine the node number of the hidden layer and
relevant initial values of them.
2. Acquire experimental data, apply RPE algorithm to
training recurrent network until its convergence.
For example: to model the system whose
input-output function is:
y(t ) 
0.8
1  exp  0.5 y(t  1)  0.6u (t  1)  0.9
Apparently, above system is highly nonlinear.
In the simulation, normally distributed random
noise with mean value of 0 and standard variance of 0.1
is added to the experimental data. The input data for
training the recurrent neural network model are pseudo
random binary sequences whose amplitude is ±1 and
length is 64. The number of nodes in hidden layer is
taken as q=5. The weighting vectors converge in about
800 iterations. Fig. 3 shows the simulation testing
results when the trained model is imposed by the input
signal which is taken as u (t )  sin( t / 50)  e(t )
where e(t) is uncorrelated random noise with mean
value of 0 and standard variance of 0.05. In Fig.3, curve
one represents the output of the system and curve two
represents the output of the recurrent neural network
model. The results indicate that the two outputs coincide
very well.
Fig. 3 The simulation test results
To verify the validity of above method, it is carried
out to construct the model of a mechanical sensor. Fig.4
gives the modeling results of a mechanical sensor. The
training data for the recurrent neural network are taken
as the data of the impulse response of the sensor. The
number of nodes in hidden layer is taken as q=8. In
Fig.4, curve one represents the step response of the
mechanical sensor while curve two represents the output
of the recurrent neural network model.
Fig.4 Dynamic modeling of mechanical sensor
4 Conclusion
According to above analysis, the recurrent neural
network possesses the ability of nonlinear dynamic
mapping. Its architecture is determined only by the
number of nodes in the input, hidden and output layers,
so applying the recurrent neural network to dynamic
modeling of sensors is a valid approach and especially
significant for modeling highly non-linear sensors. From
the experimental results, we learn that the performance
of the recurrent neural network model conforms to the
sensor to be modeled, proving the method is not only
effective but of high precision. But to ensure the
correctness of the dynamic modeling, the training data
chosen must be typical. Increasing the number of nodes
in the hidden layer is also helpful to improve the
precision of dynamic modeling, but will be at the cost of
training time.
References:
[1] Xu kejun, Chen rongbao, Zhang chongwei, Common
Techniques in Automatic Measurement and
Instrument, Tsinghua University Press, 2000
[2] K.
S.
Narendra
and
K.
Parthasarathy,
Identificationand Control of Dynamical Systems
using Neural Networks, IEEE Trans. on Neural
Networks, Vol.1,No.1,1990,pp.4~27
[3] Ku C C and Lee K Y, Diagonal Recurrent Neural
Network for Dynamic Systems Control, IEEE Trans
on NN,Vol.6,No.1,1995,pp.144-155
[4] L.ljiung and T.Soderstrom, Theory and Practice of
Recursive Identification ,The MIT Press, 1983
[5] Draye,J.,Pavisc,D.,Libert,G., Dynamic Recurrent
Neural Networks: a Dynamical Analysis,IEEE Trans.
On Systems Man and Cybernetics, Vol.26, No.5, 1996,
pp.692-706
[6] Parlos,A.,Chong,K.,Atiya,A., Application of the
Recurrent Multilayer Perceptron in Modelling
Complex Process Dynamics,IEEE Trans.On Neural
Networks,VOL.5, No.2 ,1994, pp.255-2
[7] Kosmatopoulos,E., Polycarpou,M., Iannou,A.,
High-order neural network structures for
identification of dynamical systems, IEEE Trans.
On Neural Networks,vol6,no.2,1995,pp.422-431