* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Evolution of Neural Computation :Naturalization of Intelligence
Neuroscience and intelligence wikipedia , lookup
Biological neuron model wikipedia , lookup
Neuroethology wikipedia , lookup
Binding problem wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Intelligence wikipedia , lookup
Artificial consciousness wikipedia , lookup
Catastrophic interference wikipedia , lookup
Neurocomputational speech processing wikipedia , lookup
Ethics of artificial intelligence wikipedia , lookup
Perceptual control theory wikipedia , lookup
Synaptic noise wikipedia , lookup
Neural correlates of consciousness wikipedia , lookup
Convolutional neural network wikipedia , lookup
Artificial general intelligence wikipedia , lookup
Neural modeling fields wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Nervous system network models wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Artificial intelligence wikipedia , lookup
Neural binding wikipedia , lookup
Artificial neural network wikipedia , lookup
Development of the nervous system wikipedia , lookup
Neural engineering wikipedia , lookup
Metastability in the brain wikipedia , lookup
Brics “Although neural computation has been successful in many applications, naturalization of intelligence is a missing feature in such computational model. Naturalization of intelligence is possible only if more rigorous studies of conscious attributes are done in the scientific domain. The collective response model using wave equation is being inspired from the study of attributes of consciousness. The belief is that such an attempt will augment neural computation to the next level of intelligent computing..” Evolution of Neural Computation :Naturalization of Intelligence By Laxmidhar Behera Information processing in the brain is mediated by the dynamics of large, highly interconnected neuronal populations. The activity patterns exhibited by the brain are extremely rich; they include stochastic weakly correlated local firing, synchronized oscillations and bursts, as well as propagating waves of activity. Perception, emotion etc. are supposed to be emergent properties of such a complex nonlinear neural circuit. Although, revolution in computing power in the last century has reached a stage beyond our imagination, such super computing machines trail way behind when compared with human brain processing tasks such as pattern recognition and language understanding. The structure of natural intelligence is still a mystery for us and this mystery has to be unraveled if neural computation has to go to the next stage of revolution. Natural biological systems starting from plant to human beings exhibit variegated level of intelligence. It seems that intelligence is an attribute in all those species which possess consciousness. Can studies in consciousness help one to improve the understanding of intelligence better? Fig. 1. An intelligent system should have abilities to understand, perceive, reason, solve problems and moreover learn from past experiences. The systemic understanding of cognitive processes consists of the for mulation and solution of three fundamental problems in the design of intelligent machines that 'intelligently' observe, predict and interact with the objects of the physical world. These problems are known as the system identification problem, the stochastic filtering problem, and the adaptive control problem. II.Intelligent Computing: Some challenges How intelligent an artificial machine is? Consider a case of intelligent control system where such an artificial machine is employed as a controller. One can pose a following question: · Can we construct a Control System that hypothesizes its own control law ? I.Intelligence - Something still Mysterious Real Intelligence is what inspires a normal thought process of a human. Artificial Intelligence is a property of machines which gives it ability to mimic the human thought process. The foundational framework to intelligent computing lies in our proper understanding of mental processes. Though the term intelligence is still not completely defined, according to Turing, it is a combination of five components as shown in Fig 1 : What is intelligence ? Let the same machine is put to use for a denoising application. Can one ask the following question? 55 · Can the machine estimate any signal embedded in a noise without assuming any signal or noise behavior? Here are some examples from control systems: 1. Consider a second order non-linear system (1) To study the stability of the system around the origin the generic form of the Lyapunov function is (2) Can we construct an intelligent machine that can hypothesize the above Lyaponov function? 2. Consider a dynamical system in state space: (3) A generic structure of a control law for this system is given as: (4) where parameters and are computed using principle of optimality and stability. How does a control engineer prescribe such a structure for the controller? Possibly, he does so through his experience, intuition and some basic foundation in control systems that he has learnt. shown in Fig. 2. Signal analysis can either be on a case-by-case basis or through a Generic model. The Signal is often embedded in Noise, the nature of which has an important role in Signal Estimation. This noise can be Gaussian or nonGaussian as shown in figure 3. Can a machine denoise a signal without any assumption about the signal type and noise type? III. Neural Computation : A Brief Review Real neuron are highly complex units, with the information contained in spikes. Plasticity is one of aspects of real neural networks at an advanced level. Simultaneously information processing is done through complex processes and not through simple aggregation. In contrast artificial neurons as shown in figure 4 are elementary units - can be digital, analog and even spike models. These neurons perform basic Computation such as summation, multiplication (in higher order neurons) and some advanced processes, but have little biological correlation. Any advanced structure of neural computation involves aggregation or integrate-and-fire model. Fig 4 : McCulloch – pitts Neuron Model Fig 2 : Various types of signals Fig 3 : Broad classification of noise Can one allow an artificial machine to do everything from the scratch until a stable controller is designed as a control engineer would do? Another application is signal denoising. A signal can belong to one of the three types as 56 Historically, neural computation is just another method for machine learning. This field has given rise to many popular learning sechmes which are widely used in many applications. Some of these techniques are enumerated as follows: 1. Error Back-Propagation [Rumelhart:86a, Rumelhart:86b,Hornik:89,Werbos:90,Werbos: 92]: Neural computation became popular with error-backpropagation learning scheme. This learning scheme is still widely used algorithm in all categories of feed-forward networks. 2. Recurrent Networks [Pineda:87, Pineda:88, Pearlmutter:89,Williams:89,Williams:90,Willia ms:95]: Such feedback networks are very good for time series modeling and modeling of natural feedback system. However, learning algorithms for such networks do not have good convergence properties. 3. Kohonen's Self-Organizing Feature Map [Kohonen:88,Kohonen:93,Kohonen:96,Koho nen:97]: Kohonen SOM is widely used for clustering and has been applied to visual motor coordination, character and patter n recognition. 4. Hopfield Network [Hopfield:82]: Hopfield network models an associative memory and has been applied to optimization problems as well. 5. Support Vector Machines [Vapnik:98, Vapnik:71]: These networks are developed using Vapnik's statistical learning theory. The basic idea is to map the data into a higherdimensional space where nonlinear map between input and output data is translated to a linear map. IV. Visual Motor Coordination - An Example Since we have been talking about naturalization of intelligence, visual motor coordination that mimics hand-eye coordination is an apt example to investigate. The problem is described in figure 5 where outputs from two CCD cameras are used as stimulus to a Kohonen map. A combined scheme of supervised and unsupervised learning is employed to extract a map from cartisian space to joint space. Cluster points in a joint inputoutput space are shown in figure 6. Tracking of a circle is shown in figure 7. Although there are many interesting works in this regard [Behera:99,Behera:04], such mimicing lacks the rigour involved in a soccer field as one traps a pass. Fig 6 : Plot of the reference centers of different neurons and corresponding joint angles obtained using QC-based method. Fig 7 : Plot of the tracked circle using 164 neurons The Big Question Fig 5 : Schematic Diagram of the Visuo – Motor Setup. Inspite of many ideal applications of neural computation in control, pattern recognition, image processing and speech synthesis, learning in neural computation is still far from being natural. All these methods make strong assumption about the space around. They cannot work in a generalized condition. It is still a long way to go before we can ask 57 Recently we have proposed wave equation model of neural computation [Behera:04b,behera:05,Behera:05b] called as recurrent quantum neural network (RQNN) that can easily model probability density function in stochastic data. The prime motivation comes from the study of attributes of consciousness. As a conscious person, we always have a holistic experience although the biological body consists of many individual parts or agents. We identify that collective response behavior is a key feature in conscious beings. In this context, nonlinear Schroedinger wave equation has been used to model collective response behavior. It is shown that such a paradigm can naturally make a model more intelligent. This aspect has been demonstrated through an application intelligent filtering where complex signals are denoised without any a priori knowledge about the signal and noise as well, whereas the classical filtering methods such as Kalman filter for DC signals, FFT filter or wavelet filter for sinusoids, make prior assumptions about the signal and noise. In essence, we claim that consciousness based model of matter may reveal nature in a more aesthetic manner. The filtering result of a DC signal of strength, embedded in 0dB Gaussian noise is presented here. The exact nature of trajectory tracking is shown in Fig. 8(top). In this figure, the noise envelope is shown, and obviously its size is large due to a high noise content in the signal. The result is compared with the performance of a Kalman filter. The other interesting aspect of the results is the movement of wave packet which is square of the solution of Schroedinger wave equation. It is observed that these wave packets move in discrete steps i.e., the movement is not continuous. In Figure 8 (bottom), snapshots of wave packets are plotted at different instances corresponding to marker points as shown 58 8 6 ya (t) VI. Naturalizing Intelligence in neural Computation along the tracking trajectory. It can be noticed that a very flat initial Gaussian wave packet first moves to the left, and then proceeds toward the right until the mean of the wave packet exactly matches the actual spatial position. To better appreciate the tracking performance, an error plot is also shown in Fig. 9 on page 5. Similar results were observed for a DC signal of same strength embedded in 6dB and 20dB Gaussian noises. It should be noted that the Kalman filter has the a priori knowledge that the embedded signal is a dc signal whereas the RQNN is not provided with this knowledge. The Kalman filter also makes use of the fact that the noise is Gaussian and estimates the variance of the noise based on this assumption. Thus it is expected that the performance of the Kalman filter will degrade as the noise becomes non-Gaussian. 10 a b c d 4 3 2 2 0 1 -2 -4 -6 0 1 t 2 3 4 10 8 f(x) following questions: · Can a learning machine hypothesize a theory? · Can a Learning Theory prove a Theorem? · Can a machine propose a theory similar to the special theory of relativity? 6 4 1 2 3 2 Initial 0 -3 -2 -1 0 x 1 2 3 Fig. 8. (top) Eye tracking of a fixed target in a noisy environment of 0 dB SNR: 'a' respresents fixed target, 'b' represents target tracking using RQNN model and 'c' represents target tracking using a Kalman filter. The noise envelope is represented by the curve 'd'; (bottom) The snapshots of the wave packets at different instances corresponding to the marker points (1,2,3) as shown in the top figure. The solid line represent the initial wave packet assigned to the Schroedinger wave equation. 2.5 tracking error 2 1.5 1 0.5 0 -0.5 0 1 2 3 4 t Fig. 9. The continuous line represents tracking error using RQNN model while the broken line represents tracking error using Kalman filter. movement are given in figure10 for the case of 20dB SNR. The performance of the RQNN is as good as the wavelet filter and is better than the FFT filter. Readers should take note of the evolution of the wave packets in case of nonGaussian noise as shown in figure 10. The non gaussian noise has a pdf that is split into two symmetrical triangular lobes that meet at the origin. Although the initial state of the Schroedinger wave equation was a Gaussian pdf, the wave packet finally split into two halves approaching the shape of the actual nonGaussian pdf envelope. This split shape was maintained as the wave packet tracks the sinusoid signal through the positive and negative extremum. 5 1 In contrast, the RQNN model does not make any assumption about the noise. The comparative performance in terms of rms error for all the noise levels is shown in Table I. It is easily seen from Table I that the rms tracking error of RQNN is much less than that of the Kalman filter. Moreover, RQNN performs equally well for all the three categories of noise levels, whereas the performance of the Kalman filter degrades with the increase in noise level. In this sense we can say that our model performs the tracking with a greater efficiency compared to the Kalman filter. 4 3 2 3 1 a b c 0 -1 4.5 4.6 4.7 t 4.8 4.9 2 5 0.4 Initial Wave Packet Table 1. Performance comparison between Kalman filter and RQNN for various levels of Gaussian noise f(x) 0.3 0.2 1 2 3 0.1 0 -20 Next, the filtering result of a DC-shifted sinusoid, embedded in 20dB non-Gaussian noise is presented. Non-Gaussian noise samples are generated from a noise pdf that is a normalized sum of two triangular noise pdf. The plots for the tracking and wavepacket -10 0 x 10 20 Fig. 10. (top) Filtering of a sinusoid signal with a dc shift 1a(t)=2+2sin(2p10t) with SNR 20dB embedded in non Gaussian noise : 'a' represents actual signal, 'b' represents estimation using RQNN, and 'c' represents estimation using wavelet filter; (bottom) Snapshots of three wave packets corresponsing to marker points shown in top plot. These marker points correspond to t = 4.925, 4.975 and t = 5 sec respectively. 59 VII. Concluding Remarks Although neural computation has been successful in many applications, naturalization of intelligence is a missing feature in such computational model. Naturalization of intelligence is possible only if more rigorous studies of conscious attributes are done in the scientific domain. In our recent works, complex signals such as amplitude modulated signal and speech signal are denoised without making any assumption about the signal and noise as well. In contrast, all prevalent techniques use apriori information about the signal and noise before signals can be denoised. The collective response model using wave equation used in our works is being inspired from the study of attributes of consciousness. Finally we believe that such an attempt will augment neural computation to the next level of intelligent computing. Foundational research in consciousness will help us to closely understand neural information processing dynamics. Another very interesting application of such understanding will find in health care system to help mentally sick persons. References 1. D. E. Rumelhart, G. E. Hanton and R.J. Williams, “Learning representations of backpropagation errors”, Nature (London), vol. 323, pp. 533—536, 1986. 2. D. E. Rumelhart, G. E. Hanton, and R.J. Williams,“Learning internal representations by error propagation”, vol.1, chapter 8, MIT Press, 1986. K. Hornik and M. Stinchcombe and H. White,“ Multilayer feedforward networks are universal approximators”, vol.2, Neural Networks, pp. 359--366, 1989. P. J. Werbos, “Backpropagation through time: What it does and how to do it”, Proceedings of the IEEE, vol.78, pp.1550—1560, 1990. P. J. Werbos, “Neural Networks and the human mind: New mathematics fits humanistic insight”, in Proceedings of IEEE International Conference on Systems, Man and Cybernatics, 1992, pp. 78-83. F. J. Pineda,“Generalization of back-propagation to recurrent neural networks”, Physical Review Letters, vol.59, pp. 2229-2232, 1987. F. J. Pineda, “Dynamics and architecure in neural computation”, Journal of Complexity, vol.4, pp. 216-245, 1988. B. A. Pearlmutter, “Learning state-space trajectories in recurrent neural networks”, Neural Computation, vol. 1, pp. 263--269, 1989. R. J. Williams and D. Zisper, “A learning algorithm for continually running fully recurrent neural networks”, Neural Computation, vol. 1, pp. 270--280, 1989. R. J. Williams and J. Peng, “An efficient gradient-based algorithm for online training of recurrent network trajectories”, Neural Computation, vol. 2, pp. 490--501, 1990. R. J. Williams and D. Zisper, “Backpropagation: Theory, Architectures and Applications”, chapter Gradient-based learning algorithms for recurrent networks and their computational complexity, pp. 433--486, Hillsdale, NJ:Lawrence Erlbaum, 1995. T. Kohonen, “Self-Organization and Associative Memory”, Springer-Verlag, New York, 3rd edition, 1988. T. Kohonen, “Physiological interpretation of the self-organizing map algorithm”, Neural Networks, vol. 6, pp. 895--905, 1993. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. T. Kohonen and E. Oja and O. Simula and A. Visa and J. Kangas, “Engineering applications of the self-organizing map”, Proceedings of the IEEE, vol. 84, pp. 13581384, 1996. 60 15. T. Kohonen, “Self-Organizing Maps”, Springer-Verlag, Berlin, 2nd edition, 1997. 16. J. J. Hopfield, “Nerual Networks and physical systems with emergent collective computational abilities”, Proceedings of the National Academy of Sciences of the USA, vol. 79, pp. 2554-2558, 1982. 17. V. N. Vapnik, “Statistical Learning Theory”, John Wiley and Sons, Inc., New York, 1998. 18. V. N. Vapnik and A. Y. Chervonenkis, “On the uniform convergence of relative frequencies of events to their probabilities”, Theoretical Probability and Its Applications, vol. 17, pp. 264--280, 1971. 19. L. Behera and Nandagopal K., “A hybrid neural control scheme for visuo-motor coordination”, IEEE Control System Magazine, vol. 146, No. 5, pp. 484--491, 1999. 20. N. Kumar and L. Behera, “Visual Motor Coordination using a quantum clustering based neural control scheme”, Neural Processing Letters, vol. 20, pp. 11--22, 2004. 21. L. Behera and B.Sundaram, “Stochastic filtering and speech enhancement using a recurrent quantum neural network”, in Proceeding, International Conference on Intelligent Sensors and Information Processing, 2004, pp. 165--170. 22. L. Behera, I. Kar and A. C. Elitzur, “A recurrent quantum neural network model to describe eye tracking of moving targets”, Foundations of Physics Letters, Vol 18, No. 4, pp. 357 – 370, 2005. 23. L. Behera, I. Kar and A. C. Elitzur, “The Emerging Physics of Consciousness”, chapter Recurrent Quantum Neural Network and its Applications, Springer-Verlag, 2005, To be published. About the author : Prof. L. Behera is an Associate Professor in the Department of Electrical Engineering, IIT Kanpur. He received his Ph.D degree from IIT Delhi. His research interests are Intelligent Control System , Quantum Learning , Cognitive Modeling and Soft Computing. 61