Download Evolution of Neural Computation :Naturalization of Intelligence

Brics
“Although neural computation has been successful in many applications,
naturalization of intelligence is a missing feature in such computational model.
Naturalization of intelligence is possible only if more rigorous studies of conscious
attributes are done in the scientific domain. The collective response model using wave
equation is being inspired from the study of attributes of consciousness. The belief is
that such an attempt will augment neural computation to the next level of intelligent
computing..”
Evolution of Neural Computation :Naturalization
of Intelligence
By Laxmidhar Behera
Information processing in the brain is
mediated by the dynamics of large, highly
interconnected neuronal populations. The
activity patterns exhibited by the brain are
extremely rich; they include stochastic weakly
correlated local firing, synchronized
oscillations and bursts, as well as propagating
waves of activity. Perception, emotion etc. are
supposed to be emergent properties of such a
complex nonlinear neural circuit. Although,
revolution in computing power in the last
century has reached a stage beyond our
imagination, such super computing machines
trail way behind when compared with human
brain processing tasks such as pattern
recognition and language understanding. The
structure of natural intelligence is still a
mystery for us and this mystery has to be
unraveled if neural computation has to go to
the next stage of revolution.
Natural biological systems starting from plant
to human beings exhibit variegated level of
intelligence. It seems that intelligence is an
attribute in all those species which possess
consciousness. Can studies in consciousness
help one to improve the understanding of
intelligence better?
Fig. 1. An intelligent system should have
abilities to understand, perceive, reason, solve
problems and moreover learn from past
experiences. The systemic understanding of
cognitive processes consists of the
for mulation and solution of three
fundamental problems in the design of
intelligent machines that 'intelligently'
observe, predict and interact with the objects
of the physical world. These problems are
known as the system identification problem,
the stochastic filtering problem, and the
adaptive control problem.
II.Intelligent Computing: Some challenges
How intelligent an artificial machine is?
Consider a case of intelligent control system
where such an artificial machine is employed
as a controller. One can pose a following
question:
· Can we construct a Control System that
hypothesizes its own control law ?
I.Intelligence - Something still Mysterious
Real Intelligence is what inspires a normal
thought process of a human. Artificial
Intelligence is a property of machines which
gives it ability to mimic the human thought
process. The foundational framework to
intelligent computing lies in our proper
understanding of mental processes. Though
the term intelligence is still not completely
defined, according to Turing, it is a
combination of five components as shown in
Fig 1 : What is intelligence ?
Let the same machine is put to use for a
denoising application. Can one ask the
following question?
55
· Can the machine estimate any signal
embedded in a noise without assuming any
signal or noise behavior?
Here are some examples from control systems:
1. Consider a second order non-linear system
(1)
To study the stability of the system around the
origin
the generic form of
the Lyapunov function is
(2)
Can we construct an intelligent machine that
can hypothesize the above Lyaponov function?
2. Consider a dynamical system in state space:
(3)
A generic structure of a control law for this
system is given as:
(4)
where parameters and are computed using
principle of optimality and stability. How does
a control engineer prescribe such a structure
for the controller? Possibly, he does so through
his experience, intuition and some basic
foundation in control systems that he has
learnt.
shown in Fig. 2. Signal analysis can either be on
a case-by-case basis or through a Generic model.
The Signal is often embedded in Noise, the
nature of which has an important role in Signal
Estimation. This noise can be Gaussian or nonGaussian as shown in figure 3.
Can a machine denoise a signal without any assumption
about the signal type and noise type?
III. Neural Computation : A Brief Review
Real neuron are highly complex units, with the
information contained in spikes. Plasticity is
one of aspects of real neural networks at an
advanced level. Simultaneously information
processing is done through complex processes
and not through simple aggregation.
In contrast artificial neurons as shown in figure
4 are elementary units - can be digital, analog
and even spike models. These neurons perform
basic Computation such as summation,
multiplication (in higher order neurons) and
some advanced processes, but have little
biological correlation. Any advanced structure
of neural computation involves aggregation or
integrate-and-fire model.
Fig 4 : McCulloch – pitts Neuron Model
Fig 2 : Various types of signals
Fig 3 : Broad classification of noise
Can one allow an artificial machine to do everything
from the scratch until a stable controller is designed as a
control engineer would do?
Another application is signal denoising. A
signal can belong to one of the three types as
56
Historically, neural computation is just another
method for machine learning. This field has
given rise to many popular learning sechmes
which are widely used in many applications.
Some of these techniques are enumerated as
follows:
1. Error Back-Propagation [Rumelhart:86a,
Rumelhart:86b,Hornik:89,Werbos:90,Werbos:
92]: Neural computation became popular with
error-backpropagation learning scheme. This
learning scheme is still widely used algorithm in
all categories of feed-forward networks.
2. Recurrent Networks [Pineda:87, Pineda:88,
Pearlmutter:89,Williams:89,Williams:90,Willia
ms:95]: Such feedback networks are very good
for time series modeling and modeling of
natural feedback system. However, learning
algorithms for such networks do not have good
convergence properties.
3. Kohonen's Self-Organizing Feature Map
[Kohonen:88,Kohonen:93,Kohonen:96,Koho
nen:97]: Kohonen SOM is widely used for
clustering and has been applied to visual motor
coordination, character and patter n
recognition.
4. Hopfield Network [Hopfield:82]: Hopfield
network models an associative memory and
has been applied to optimization problems as
well.
5. Support Vector Machines [Vapnik:98,
Vapnik:71]: These networks are developed
using Vapnik's statistical learning theory. The
basic idea is to map the data into a higherdimensional space where nonlinear map
between input and output data is translated to a
linear map.
IV. Visual Motor Coordination - An Example
Since we have been talking about naturalization
of intelligence, visual motor coordination that
mimics hand-eye coordination is an apt
example to investigate. The problem is
described in figure 5 where outputs from two
CCD cameras are used as stimulus to a
Kohonen map. A combined scheme of
supervised and unsupervised learning is
employed to extract a map from cartisian space
to joint space. Cluster points in a joint inputoutput space are shown in figure 6. Tracking of
a circle is shown in figure 7. Although there are
many interesting works in this regard
[Behera:99,Behera:04], such mimicing lacks the
rigour involved in a soccer field as one traps a
pass.
Fig 6 : Plot of the reference centers of different neurons and
corresponding joint angles obtained using QC-based method.
Fig 7 : Plot of the tracked circle using 164 neurons
The Big Question
Fig 5 : Schematic Diagram of the Visuo – Motor Setup.
Inspite of many ideal applications of neural
computation in control, pattern recognition,
image processing and speech synthesis,
learning in neural computation is still far from
being natural. All these methods make strong
assumption about the space around. They
cannot work in a generalized condition. It is
still a long way to go before we can ask
57
Recently we have proposed wave equation
model of neural computation
[Behera:04b,behera:05,Behera:05b] called as
recurrent quantum neural network (RQNN)
that can easily model probability density
function in stochastic data. The prime
motivation comes from the study of attributes
of consciousness. As a conscious person, we
always have a holistic experience although the
biological body consists of many individual
parts or agents. We identify that collective response
behavior is a key feature in conscious beings.
In this context, nonlinear Schroedinger wave
equation has been used to model collective
response behavior. It is shown that such a
paradigm can naturally make a model more
intelligent. This aspect has been demonstrated
through an application intelligent filtering where complex signals are denoised without
any a priori knowledge about the signal and
noise as well, whereas the classical filtering
methods such as Kalman filter for DC signals,
FFT filter or wavelet filter for sinusoids, make
prior assumptions about the signal and noise.
In essence, we claim that consciousness based
model of matter may reveal nature in a more
aesthetic manner. The filtering result of a DC
signal of strength, embedded in 0dB Gaussian
noise is presented here. The exact nature of
trajectory tracking is shown in Fig. 8(top). In
this figure, the noise envelope is shown, and
obviously its size is large due to a high noise
content in the signal. The result is compared
with the performance of a Kalman filter. The
other interesting aspect of the results is the
movement of wave packet which is square of
the solution of Schroedinger wave equation. It
is observed that these wave packets move in
discrete steps i.e., the movement is not
continuous. In Figure 8 (bottom), snapshots of
wave packets are plotted at different instances
corresponding to marker points as shown
58
8
6
ya (t)
VI. Naturalizing Intelligence in neural
Computation
along the tracking trajectory. It can be noticed
that a very flat initial Gaussian wave packet first
moves to the left, and then proceeds toward the
right until the mean of the wave packet exactly
matches the actual spatial position. To better
appreciate the tracking performance, an error
plot is also shown in Fig. 9 on page 5. Similar
results were observed for a DC signal of same
strength embedded in 6dB and 20dB Gaussian
noises. It should be noted that the Kalman
filter has the a priori knowledge that the
embedded signal is a dc signal whereas the
RQNN is not provided with this knowledge.
The Kalman filter also makes use of the fact
that the noise is Gaussian and estimates the
variance of the noise based on this assumption.
Thus it is expected that the performance of the
Kalman filter will degrade as the noise becomes
non-Gaussian.
10
a
b
c
d
4
3
2
2
0
1
-2
-4
-6
0
1
t
2
3
4
10
8
f(x)
following questions:
· Can a learning machine hypothesize a theory?
· Can a Learning Theory prove a Theorem?
· Can a machine propose a theory similar to the
special theory of relativity?
6
4
1
2
3
2
Initial
0
-3
-2
-1
0
x
1
2
3
Fig. 8. (top) Eye tracking of a fixed target in a noisy
environment of 0 dB SNR: 'a' respresents fixed target, 'b'
represents target tracking using RQNN model and 'c'
represents target tracking using a Kalman filter. The noise
envelope is represented by the curve 'd'; (bottom) The
snapshots of the wave packets at different instances
corresponding to the marker points (1,2,3) as shown in the
top figure. The solid line represent the initial wave packet
assigned to the Schroedinger wave equation.
2.5
tracking error
2
1.5
1
0.5
0
-0.5
0
1
2
3
4
t
Fig. 9. The continuous line represents tracking error using
RQNN model while the broken line represents tracking
error using Kalman filter.
movement are given in figure10 for the case of
20dB SNR. The performance of the RQNN is
as good as the wavelet filter and is better than
the FFT filter. Readers should take note of the
evolution of the wave packets in case of nonGaussian noise as shown in figure 10. The non
gaussian noise has a pdf that is split into two
symmetrical triangular lobes that meet at the
origin. Although the initial state of the
Schroedinger wave equation was a Gaussian
pdf, the wave packet finally split into two halves
approaching the shape of the actual nonGaussian pdf envelope. This split shape was
maintained as the wave packet tracks the
sinusoid signal through the positive and
negative extremum.
5
1
In contrast, the RQNN model does not make
any assumption about the noise. The
comparative performance in terms of rms
error for all the noise levels is shown in Table I.
It is easily seen from Table I that the rms
tracking error of RQNN is much less than that
of the Kalman filter. Moreover, RQNN
performs equally well for all the three
categories of noise levels, whereas the
performance of the Kalman filter degrades
with the increase in noise level. In this sense we
can say that our model performs the tracking
with a greater efficiency compared to the
Kalman filter.
4
3
2
3
1
a
b
c
0
-1
4.5
4.6
4.7
t
4.8
4.9
2
5
0.4
Initial Wave Packet
Table 1. Performance comparison between
Kalman filter and RQNN for various levels
of Gaussian noise
f(x)
0.3
0.2
1
2
3
0.1
0
-20
Next, the filtering result of a DC-shifted
sinusoid, embedded in 20dB non-Gaussian
noise is presented. Non-Gaussian noise
samples are generated from a noise pdf that is a
normalized sum of two triangular noise pdf.
The plots for the tracking and wavepacket
-10
0
x
10
20
Fig. 10. (top) Filtering of a sinusoid signal with a dc shift
1a(t)=2+2sin(2p10t) with SNR 20dB embedded in non
Gaussian noise : 'a' represents actual signal, 'b' represents
estimation using RQNN, and 'c' represents estimation
using wavelet filter; (bottom) Snapshots of three wave
packets corresponsing to marker points shown in top plot.
These marker points correspond to t = 4.925, 4.975 and t = 5
sec respectively.
59
VII. Concluding Remarks
Although neural computation has been
successful in many applications, naturalization
of intelligence is a missing feature in such
computational model. Naturalization of
intelligence is possible only if more rigorous
studies of conscious attributes are done in the
scientific domain. In our recent works,
complex signals such as amplitude modulated
signal and speech signal are denoised without
making any assumption about the signal and
noise as well. In contrast, all prevalent
techniques use apriori information about the
signal and noise before signals can be denoised.
The collective response model using wave
equation used in our works is being inspired
from the study of attributes of consciousness.
Finally we believe that such an attempt will
augment neural computation to the next level
of intelligent computing. Foundational
research in consciousness will help us to closely
understand neural information processing
dynamics. Another very interesting application
of such understanding will find in health care
system to help mentally sick persons.
References
1.
D. E. Rumelhart, G. E. Hanton and R.J. Williams, “Learning representations of backpropagation errors”, Nature (London), vol. 323, pp. 533—536, 1986.
2.
D. E. Rumelhart, G. E. Hanton, and R.J. Williams,“Learning internal representations by
error propagation”, vol.1, chapter 8, MIT Press, 1986.
K. Hornik and M. Stinchcombe and H. White,“ Multilayer feedforward networks are
universal approximators”, vol.2, Neural Networks, pp. 359--366, 1989.
P. J. Werbos, “Backpropagation through time: What it does and how to do it”,
Proceedings of the IEEE, vol.78, pp.1550—1560, 1990.
P. J. Werbos, “Neural Networks and the human mind: New mathematics fits humanistic
insight”, in Proceedings of IEEE International Conference on Systems, Man and
Cybernatics, 1992, pp. 78-83.
F. J. Pineda,“Generalization of back-propagation to recurrent neural networks”, Physical
Review Letters, vol.59, pp. 2229-2232, 1987.
F. J. Pineda, “Dynamics and architecure in neural computation”, Journal of Complexity,
vol.4, pp. 216-245, 1988.
B. A. Pearlmutter, “Learning state-space trajectories in recurrent neural networks”,
Neural Computation, vol. 1, pp. 263--269, 1989.
R. J. Williams and D. Zisper, “A learning algorithm for continually running fully
recurrent neural networks”, Neural Computation, vol. 1, pp. 270--280, 1989.
R. J. Williams and J. Peng, “An efficient gradient-based algorithm for online training of
recurrent network trajectories”, Neural Computation, vol. 2, pp. 490--501, 1990.
R. J. Williams and D. Zisper, “Backpropagation: Theory, Architectures and
Applications”, chapter Gradient-based learning algorithms for recurrent networks and
their computational complexity, pp. 433--486, Hillsdale, NJ:Lawrence Erlbaum, 1995.
T. Kohonen, “Self-Organization and Associative Memory”, Springer-Verlag, New York,
3rd edition, 1988.
T. Kohonen, “Physiological interpretation of the self-organizing map algorithm”, Neural
Networks, vol. 6, pp. 895--905, 1993.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14. T. Kohonen and E. Oja and O. Simula and A. Visa and J. Kangas, “Engineering
applications of the self-organizing map”, Proceedings of the IEEE, vol. 84, pp. 13581384, 1996.
60
15. T. Kohonen, “Self-Organizing Maps”, Springer-Verlag, Berlin, 2nd edition, 1997.
16. J. J. Hopfield, “Nerual Networks and physical systems with emergent collective
computational abilities”, Proceedings of the National Academy of Sciences of the USA,
vol. 79, pp. 2554-2558, 1982.
17. V. N. Vapnik, “Statistical Learning Theory”, John Wiley and Sons, Inc., New York, 1998.
18. V. N. Vapnik and A. Y. Chervonenkis, “On the uniform convergence of relative
frequencies of events to their probabilities”, Theoretical Probability and Its Applications,
vol. 17, pp. 264--280, 1971.
19. L. Behera and Nandagopal K., “A hybrid neural control scheme for visuo-motor
coordination”, IEEE Control System Magazine, vol. 146, No. 5, pp. 484--491, 1999.
20. N. Kumar and L. Behera, “Visual Motor Coordination using a quantum clustering based
neural control scheme”, Neural Processing Letters, vol. 20, pp. 11--22, 2004.
21. L. Behera and B.Sundaram, “Stochastic filtering and speech enhancement using a
recurrent quantum neural network”, in Proceeding, International Conference on
Intelligent Sensors and Information Processing, 2004, pp. 165--170.
22. L. Behera, I. Kar and A. C. Elitzur, “A recurrent quantum neural network model to
describe eye tracking of moving targets”, Foundations of Physics Letters, Vol 18, No. 4,
pp. 357 – 370, 2005.
23. L. Behera, I. Kar and A. C. Elitzur, “The Emerging Physics of Consciousness”, chapter
Recurrent Quantum Neural Network and its Applications, Springer-Verlag, 2005, To be
published.
About the author : Prof. L. Behera is an Associate Professor in the Department of
Electrical Engineering, IIT Kanpur. He received his Ph.D degree from IIT Delhi.
His research interests are Intelligent Control System , Quantum Learning ,
Cognitive Modeling and Soft Computing.
61