* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download chapter_1
Endocannabinoid system wikipedia , lookup
Neuroeconomics wikipedia , lookup
Feature detection (nervous system) wikipedia , lookup
Apical dendrite wikipedia , lookup
Neurocomputational speech processing wikipedia , lookup
Neuroethology wikipedia , lookup
State-dependent memory wikipedia , lookup
Neuroinformatics wikipedia , lookup
Neural oscillation wikipedia , lookup
Cognitive neuroscience wikipedia , lookup
Artificial intelligence wikipedia , lookup
Neuroanatomy wikipedia , lookup
Optogenetics wikipedia , lookup
End-plate potential wikipedia , lookup
Caridoid escape reaction wikipedia , lookup
Multielectrode array wikipedia , lookup
Donald O. Hebb wikipedia , lookup
Neuromuscular junction wikipedia , lookup
Mirror neuron wikipedia , lookup
Activity-dependent plasticity wikipedia , lookup
Central pattern generator wikipedia , lookup
Synaptogenesis wikipedia , lookup
Neural coding wikipedia , lookup
Molecular neuroscience wikipedia , lookup
Catastrophic interference wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Single-unit recording wikipedia , lookup
Metastability in the brain wikipedia , lookup
Neural engineering wikipedia , lookup
Neural modeling fields wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Stimulus (physiology) wikipedia , lookup
Sparse distributed memory wikipedia , lookup
Artificial neural network wikipedia , lookup
Neurotransmitter wikipedia , lookup
Development of the nervous system wikipedia , lookup
Nonsynaptic plasticity wikipedia , lookup
Convolutional neural network wikipedia , lookup
Chemical synapse wikipedia , lookup
Biological neuron model wikipedia , lookup
Synaptic gating wikipedia , lookup
Recurrent neural network wikipedia , lookup
Chapter 1 Introduction to Neurocomputing 國立雲林科技大學 資訊工程研究所 張傳育(Chuan-Yu Chang ) 博士 Office: ES 709 TEL: 05-5342601 ext. 4337 E-mail: [email protected] What is Neurocomputing? Neurocomputing approach Involves a learning process within an ANN 一旦類神經網路訓練完成,該類神經網路即可進行特定的工 作。例如pattern recognition Associated (聯想) Why can we (human) perform certain tasks much better than a digital computer? Our brain is organized 大腦神經(nerve)的傳導速度比電子的速度慢(106)倍,但腦神經 具有大量平行的計算能力。(約1011個neuron) 大腦是一個adaptive, nonlinear, parallel的計算機。 What is Neurocomputing? (cont.) 類神經網路的主要能力: 由範例學習(learn by example) 歸納(generalize) The NN can classify input patterns to an acceptable level of accuracy even if they were never used during the training process. 大部分的類神經網路具有類似的特徵: Parallel computational architecture Highly interconnected Nonlinearity output What is Neurocomputing? (cont.) Applications of neural networks: Image processing Prediction and forecasting Associative memory Clustering Speech recognition Combinatorial optimization Feature extraction … What is Neurocomputing? (cont.) 使用neurocomputing解決特定問題的優點: Fault tolerance Nonlinear Adaptive Course Introduction Text book Fredric M. Ham, Principles of Neurocomputing for Science & Engineering, McGRAW-HILL, 2001. Simon Haykin, Neural Networks-A comprehensive foundation, 2nd Edition. Prentice Hall, 1999 Contents Introduction to Neurocomputing Fundamental Neurocomputing Concepts Activation Function Adaline and Madaline Perceptron Multilayer perceptron Leaning Rules Mapping Networks Associative memory Backpropagation Learning Algorithm Counterpropagation Radial Basis Function Neural Networks Contents (con’t) Self-Organizing Networks Kohonen Self-Organizing Map Learning Vector Quantization Adaptive Resonance Theory Neural network Recurrent Networks and Temporal feedforward Networks Hopfield Associative Memory Simulated Annealing Boltzmann Machine Time-Delay Neural Networks Contents (con’t) Statistical Methods Using Neural Networks Principal Component Analysis Independent Component Analysis Course Requirements Homeworks Final Project Supervised learning: Digits Recognition Unsupervised learning: Traveling Salesperson Problem (TSP) Presentation Implementation and Demonstration Final Examination 請思考這門課對各位的研 究是否有幫助? Does this course helpful to your research? Historical notes McCulloch and Pitts (1943) Hebb (1949) No training to neurons. Act as certain logic functions. Based on a neurobiological viewpoint, describes a learning process. Hebb stated that information is stored in the connections of neurons and postulated a learning strategy for adjustment of the connection weight. Rosenblatt (1958) Proposed the concept of the perceptron Historical notes (cont.) Widrow and Hoff (1960) Minsky and Papert (1969) Slowed down neural network research in 1969. Perceptrons have limited capabilities, the XOR problem. Kohonen and Anderson (1972) The Adaline (adaptive linear element) trained by the LMS learning rule. Content-addressable associative memories. Werbos (1974) The first description of the backpropagation algorithm for training multilayer feed-forward perceptrons. Historical notes (cont.) Little and Shaw (1975) Lee (1975) Presented the fuzzy McCulloch-Pitts neuron mode. Hopfield (1982) Use a probabilistic model of a neuron instead of a deterministic one. Proposed a recurrent neural network The network can store information in a dynamically stable storage and retrieval. Kohonen (1982) Presented the self-organizating feature map. It is an unsupervised, competitive learning, clustering network in which only one neuron is “on” at a time. Historical notes (cont.) Oja (1982) Presented a single linear neuron trained by a normalized Hebbian learning rule that acts as a principal-component analyzer. The neuron is capable of adaptive extracting the 1st principal eignvector from the input data. Carpenter and Grossberg (1987) Developed self-organizing neural networks based adaptive resonance theory (ART) Sivilotti, Mahowald, and Mead (1987) The first VLSI realization of neural networks. Broomhead and Lowe (1988) First exploitation of radial basis function in designing neural networks. McCulloch-Pitts neuron The McCulloch-Pitts neuron is a two-state device. (on/off, binary) The output of a particular neuron cannot coalesce with output of another neuron; it can branch to another neuron and terminate as an input to that neuron.(個別神經元的輸出無法與其他神經元 的輸出合併,但可當成其他神經元的輸入。) Two types of terminations Excitatory input Inhibitory input There can be any number of inputs to a neuron. .(一個神經元可 有任意多個輸入) Whether the neuron will be on (firing) or off depends on the threshold of the neuron.(但其輸出是由threshold所決定。) McCulloch-Pitts neuron Physical assumptions:(基本假設) The neuron activity is an all-or-nothing process, ie., the activation of the neuron is binary. A certain fixed number of synapses (>1) must be excited within a period of latent addition for a neuron to be excited. The only significant delay within the nervous system is synaptic delay. The activity of any nonzero inhibitory synapse absolutely prevents excitation of the neuron at that time. The structure of the network does not change with time. McCulloch-Pitts neuron Architecture of a McCulloch-Pitts neuron N excitatory input M inhibitory input 1個由threshold決定的輸出y。 1 y 0 for u for u 為滿足前面的假設4, nw c u= S xi w McCulloch-Pitts neuron Example 1 The AND logic function can be realized using the McCulloch-Pitts neuron having two synaptic connections with equal weights of 1 and a threshold =2. McCulloch-Pitts neuron Example 2: The OR logic function can be realized using the McCullochPitts neuron having two synaptic connections with equal weights of 2 and a threshold =2. McCulloch-Pitts neuron Example 3: The XOR logic function can be realized using three McCulloch-Pitts neuron having two synaptic connections with equal weights of 2 and a threshold =2. Neurocomputing and Neuroscience Biological neural networks 神經系統(nervous system)是一個巨大且複雜的神經網路 (neural network),大腦是神經系統的中樞,神經系統和各感 官器官相連結,傳遞感官訊息給大腦,並且傳送行為命令給 反應器官(effectors) 。 大腦是由大約1011個神經元所組成,而這些神經元相互連接 成子網路,組成所謂的細胞核(nuclei) ,細胞核是由一連串 的神經元所組成,具有特定的功能。 在大腦中的感官系統(sensory system)和它的子網路,擅於 將複雜的感官資訊分解(decompose)成知覺的基本特性。對 於不同的知覺會有不同的分解。 Neurocomputing and Neuroscience Fausett 更有效的建構生物的(biological)神經元(neuron)成 類神經元(artificial neurons)。大腦和神經系統的其他部分是 由許多不同種類的神經元組成,有不同的電子特性、數量、 大小、及連接方式。 Small cell 樹狀突 軸突 Large cell Neurocomputing and Neuroscience 一個生物神經元包含有三個 部分: 樹狀突(dendrites) :收取來 自其他神經元的訊號。 Cell body:用來彙總來自樹 狀突及其他神經鍵的訊號。 當收到的刺激大於threshold 時,神經元將產生一個電能 (potential) ,也就是fire,並 將此電能沿著axon傳給其他 神經元或是目的細胞,例如 肌肉。 軸突(axon) :藉由神經鍵 (synapses)和其他神經元的 軸突(axon)相連結。神經鍵 (synapses)的數目可能從數 百到上萬。 (體幹) Neurocomputing and Neuroscience A biological neuron consists of three main components: Dendrites :receive signals from other neurons。 Cell body (soma):sums the incoming signals from the dendrites and sums the signals from the numerous synapses on its surface. Axon :the axons of other neurons connect to the dendrite and cell body surfaces by means of connectors called synapses。The number of synaptic connection from other neurons may range from a few hundred to 10,000. (體幹) Neurocomputing and Neuroscience Simplified drawing of the synapses Neurocomputing and Neuroscience 由於神經元的外表有神經薄膜,因此訊號到達樹狀突時會因 時間和距離而迅速衰減。而喪失刺激神經元的能力,除非它 們收到來自附近神經元的訊號強化了衰減的訊號。 Neurocomputing and Neuroscience 如果神經元的輸入無法達到threshold,則輸入訊號將快速衰 減,而且不會產生行動電能(potential)。因此,action potential的產生原則是all-or-nothing。 輸入強度是表現在每秒產生的action potential數量,而不是 action potential的大小。較強的輸入信號會比較弱的輸入信 號產生較多的action potential。 Neurocomputing and Neuroscience Synapses are the points of contact that connect the axon terminals to their targets. Synapse由下列所組成: Nerve terminal Synaptic cleft or gap Postsynaptic membrane Classification of neural networks Supervised learning vs. unsupervised learning