* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download ALGORITHMICS - Universitatea de Vest din Timisoara
Microneurography wikipedia , lookup
Neural coding wikipedia , lookup
Biological neuron model wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Optogenetics wikipedia , lookup
Neuroeconomics wikipedia , lookup
Neural modeling fields wikipedia , lookup
Cortical cooling wikipedia , lookup
Computational creativity wikipedia , lookup
Neural oscillation wikipedia , lookup
Neurocomputational speech processing wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Neuroesthetics wikipedia , lookup
Neuroethology wikipedia , lookup
Artificial general intelligence wikipedia , lookup
Binding problem wikipedia , lookup
Neural correlates of consciousness wikipedia , lookup
Central pattern generator wikipedia , lookup
Pattern recognition wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Catastrophic interference wikipedia , lookup
Nervous system network models wikipedia , lookup
Artificial intelligence wikipedia , lookup
Metastability in the brain wikipedia , lookup
Neural binding wikipedia , lookup
Convolutional neural network wikipedia , lookup
Development of the nervous system wikipedia , lookup
Neural engineering wikipedia , lookup
Artificial neural network wikipedia , lookup
Introduction in Computational Intelligence What is this course about ? Neural Computing Evolutionary Computing Fuzzy Computing Course structure and other stuff Neural Networks - Lecture 1 1 What is this course about ? About solving ill-posed problems Well posed problem: - It exists a formal model associated to the problem - It exists a solving algorithm Example: classifying the employees in two classes based on their income (based on a known threshold) Ill posed problem: - It is not easy to formalize it - There exist only some solving examples - The classical methods can not be applied Example: classifying the employees in two classes based on their credibility concerning a bank credit Neural Networks - Lecture 1 2 What is this course about ? Well posed problems Input data Algorithm = description of a method Results Class1 or Class2 Income value If income> threshold then Class1 else Class2 Neural Networks - Lecture 1 3 What is this course about ? Ill posed problems Input data Income value Examples: (900, Class1), (500, Class1),(5000,Class2), (1100, Class2) Adaptive computational structure (incorporates a learning/evolution module Results Class1 or Class2 The model is extracted by the adaptive computational structure Neural Networks - Lecture 1 4 What is this course about ? The solving methods for ill posed problems should be characterized by: Ability to extract patterns (models) from examples Adaptability to dynamic problems Robustness to errors or noise in input data Ability to give a result (even if only an approximate one) in a reasonable amount of time The field dealing with the development of these techniques is called “computational intelligence” or “soft computing” Neural Networks - Lecture 1 5 What is this course about ? Computational Intelligence “is a branch of the study of artificial intelligence; it aims to use learning, adaptive, or evolutionary algorithms to create programs that are, in some sense, intelligent. “ [Wikipedia] Soft Computing “is a collection of new techniques in computer science, especially in artificial intelligence; unlike hard computing, it is tolerant of imprecision, uncertainty and partial truth. In effect, the role model for soft computing is the human mind. The guiding principle of soft computing is: exploit the tolerance for imprecision, uncertainty and partial truth to achieve tractability, robustness and low solution cost.” [Wikipedia] Neural Networks - Lecture 1 6 What is this course about ? Directions Tools Starting point Neural Computing Artificial Neural Network Human brain Evolutionary Computing Evolutionary Algorithm Natural evolution Fuzzy Computing Human reasoning and natural language Fuzzy System Neural Networks - Lecture 1 7 What is this course about ? Natural Computing = problem solving methods inspired by nature Neural Computing DNA Computing Membrane Computing Quantum Computing Evolutionary Computing Swarm Intelligence Immunocomputing Fuzzy Computing Neural Networks - Lecture 1 8 Neural Computing - overview Basic principles The biological model Artificial neural networks Applications History Neural Networks - Lecture 1 9 Basic principles A functional point of view: neural network = adaptive system which give answers to a problem after it has been trained for similar problems Examples Input data Neural network = learning system Neural Networks - Lecture 1 Answer 10 Basic principles A structural point of view: Artificial neural network (ANN)= set of highly interconnected simply processing units (also called neurons) Processing unit = simplified model of a neuron Artificial neural network = (very) simplified model of the brain Neural Networks - Lecture 1 11 Basic principles The complex behavior emerges from simple rules which interact and are applied in parallel This bottom-up approach is opposite to the top down approach particular to classical artificial intelligence The learning ability derives from the adaptability of some parameters associated with the processing units behaviour (the parameters are modified by learning) The processing is mainly numerical unlike the symbolical approach of AI Neural Networks - Lecture 1 12 Neural Computing - overview Basic principles The biological model Artificial neural networks Applications History Neural Networks - Lecture 1 13 The biological model Human brain around 1011 neurons, and 1015 interconnections (synapses) Neural Networks - Lecture 1 14 The biological model How a neuron works ? • It receives input signals through its dendrites • The input signals generate difference of electrical potential on the cell membrane • The difference is propagated to the axon hillock • A train of electrical impulses is generated along the axon • The impulses on the axon generate the release in the synaptic space of some neurotransmitters Neural Networks - Lecture 1 15 The biological model Von Neumann machine Human Brain •One or a few high speed (ns) processors with considerable computing power • Large number (1011) of low speed processors (ms) with limited computing power •One or a few shared high speed buses for communication • Large number (1015) of low speed connections •Sequential memory access by address • Content addressable recall (CAM) • Problem-solving knowledge resides in the connectivity of neurons • Adaptation by changing the connectivity •Problem-solving knowledge is separated from the computing component •Hard to be adaptive Neural Networks - Lecture 1 16 Neural Computing - overview Basic principles The biological model Artificial neural networks Applications History Neural Networks - Lecture 1 17 Artificial Neural Networks ANN = interconnected processing units with adaptive parameters Processing unit: many inputs, one output inputs Operations on a processing unit (ex.) n output y f ( w j x j w0 ) j 1 f (u ) sgn( u ) Numerical weights attached to connections f (u ) H (u ) Neural Networks - Lecture 1 f (u ) tanh( u ) 1 f (u ) 1 exp( u ) 18 Artificial Neural Networks Example: feedforward architecture Structure of a ANN - Architecture (directed graph) - Functioning (closed form functions or recurrence relations) - Learning (algorithm to adjust the adaptive parameters) Output computation N1 2 N 0 1 yi f wik f wkj x j , i 1, N 2 k 0 j 0 Neural Networks - Lecture 1 19 Artificial Neural Networks Examples of architectures Neural Networks - Lecture 1 20 Artificial Neural Networks Learning = extracting the model corresponding to the problem starting from examples = finding the network parameters Learning strategies: • Supervised (i.e. learning with a teacher) • Unsupervised (i.e. learning without a teacher) • Reinforcement (i.e. based on reward and penalization) Neural Networks - Lecture 1 21 Neural Computing - overview Basic principles The biological model Artificial neural networks Applications History Neural Networks - Lecture 1 22 Applications Classification and pattern recognition Input data: set of features of an object Ex: information about the employee Output: Class to which the object belongs Ex: eligible/non eligible for a bank credit Other examples: character recognition, image/texture classification, speech recognition, signal classification (EEG, EKG), land type classification Particularity: supervised learning (based on examples of correctly classified data) Neural Networks - Lecture 1 23 Applications Data clustering (the unsupervised version of classification – the data are grouped based on their similarities) Input data: set of features of an object Ex: information concerning a user session in a e-commerce system Output: class to which the object belongs Ex: group of users with similar behavior Other examples: image segmentation Neural Networks - Lecture 1 24 Applications • Example: image segmentation = find homogeneous regions in the image (facilitate the analysis of the image) Neural Networks - Lecture 1 25 Applications Approximation / estimation = finding the relationship between two variables Input data: values of some independent variables Ex: values obtained by measurements Output: value corresponding to the dependent variable Ex: estimated value for the measure which depends on the independent variables Neural Networks - Lecture 1 26 Applications Prediction = estimating the future value(s) in a time series Input data: sequence of values Ex: currency exchange rate for the last month Output: estimation of the next value Ex: estimation of the currency exchange rate for tomorrow Other examples: stock prediction, prediction in meteorology Neural Networks - Lecture 1 27 Applications Optimization = finding an approximation of the optimum of a difficult problem Associative memories = the recall process is based on stored content not on an address Adaptive control = find a control signal which ensure a given output of a system Neural Networks - Lecture 1 28 Neural Computing - overview Basic principles The biological model Artificial neural networks Applications History Neural Networks - Lecture 1 29 ANN History Pitts & McCulloch (1943) First mathematical model of biological neurons All Boolean operations can be implemented by these neuron-like nodes (with different threshold and excitatory/inhibitory connections). Competitor to Von Neumann model for general purpose computing device Origin of automata theory. Hebb (1949) Hebbian rule of learning: increase the connection strength between neurons i and j whenever both i and j are activated. Or increase the connection strength between nodes i and j whenever both nodes are simultaneously ON or OFF. Neural Networks - Lecture 1 30 ANN History Early booming (50’s – early 60’s) – Rosenblatt (1958) • Perceptron: network of threshold nodes for pattern classification. Perceptron learning rule – first learning algorithm • Perceptron convergence theorem: everything that can be represented by a perceptron can be learned – Widow and Hoff (1960, 19062) • Learning rule based on minimization methods – Minsky’s attempt to build a general purpose machine with Pitts/McCullock units Neural Networks - Lecture 1 31 ANN History • The setback (mid 60’s – late 70’s) – Serious problems with perceptron model (Minsky’s book 1969) • Single layer perceoptrons cannot represent (learn) simple functions such as XOR • Multi-layer of non-linear units may have greater power but there was no learning rule for such nets • Scaling problem: connection weights may grow infinitely – The first two problems overcame by latter effort in 80’s, but the scaling problem persists Neural Networks - Lecture 1 32 ANN History • Renewed enthusiasm and flourish (80’s – present) – New techniques • Backpropagation learning for multi-layer feed forward nets (with non-linear, differentiable node functions) • Physics inspired models (Hopfield net, Boltzmann machine, etc.) • Unsupervised learning – Impressive application (character recognition, speech recognition, text-to-speech transformation, process control, associative memory, etc.) – Traditional approaches face difficult challenges – Caution: • Don’t underestimate difficulties and limitations • Poses more problems than solutions Neural Networks - Lecture 1 33 Relationship with other topics Computer Science Biology Neurology Artificial Intelligence Data mining Cognitive psichology ANN Engineering Signal processing Adaptive control Mathematics Linear Algebra Numerical Methods Economics Statistics Differential equations (estimation, regression) Neural Networks - Lecture 1 Stock prediction Risk analysis 34 Evolutionary Computing Basic principles Structure of an Evolutionary Algorithm (EA) Variants of EAs Applications Neural Networks - Lecture 1 35 Evolutionary Computing: basic principles EC is inspired by the natural evolution processes based on the heredity principles and the darwinian principal of survival of the fittest The solution of the problem is found by exploring the space of possible solutions; a populations of individuals (agents) is used to explore the space The population elements are coded based on the particularities of the problem (bit strings, real value vectors, trees, graphs etc.) Neural Networks - Lecture 1 36 Evolutionary Computing: basic principles Parallel between the natural evolution and problem solving Natural Evolution Environment Individual Adequacy degree to the environment Problem solving Problem Potential solution Solution quality Neural Networks - Lecture 1 37 Structure of an EA solution Population initialization Stopping condition Evaluation Selection Recombination mutation Neural Networks - Lecture 1 EA = iterative process based on iterative application of - recombination - mutation - selection on a randomly initialized population 38 Variants of EAs Genetic Algorithms: Binary codification Primary operator: crossing over Secondary operator: mutation Target: combinatorial optimization problems Evolution strategies: Real codification Primary operator: mutation Secondary operator: recombination Target: real optimization problems Neural Networks - Lecture 1 39 Variants of EAs Genetic programming: Structured individuals (trees, expressions, programs etc) Target: Generate computing structures through evolution processes Evolutionary programming: Real codification Only mutation Target: optimization on continuous domains, neural networks architectures evolution Neural Networks - Lecture 1 40 EAs applications Prediction Scheduling (automated timetabling generation, tasks scheduling). Image processing (image filters design) Neural networks design Structures identification (bioinformatics) Evolutionary design & art Neural Networks - Lecture 1 41 Fuzzy Computing Based on approximate reasoning which uses fuzzy values instead of crisp values Fuzzy set: each element has a membership function Fuzzy rule: rule involving fuzzy variables Cold Warm Hot Applications: control devices (washing machines, refrigerators, elevators etc) Neural Networks - Lecture 1 42 Structure of the course ANN design One layer perceptrons. Linear separable classification problems. Widrow-Hoff and Delta learning rules. Multilayer perceptrons. BackProgation learning algorithm and variants. Radial basis function networks Competitive learning. Kohonen maps. Adaptive resonance theory Correlative learning. Principal component analysis Recurrent ANN (Hopfield, Elman, Cellular NN) Evolutionary design of NN Neural Networks - Lecture 1 43 Structure of the lab Aim: Simulating basic ANNs Tools: MatLab (SciLab,Weka, JavaSNNS) Lab assignments 1. Simple perceptron (e.g. character recognition) 2. Backpropagation network (e.g. nonliner mapping, classification) 3. RBF network (e.g. nonliner mapping, classification) 4. Unsupervised learning (e.g. clustering) Neural Networks - Lecture 1 44 Grading Lab assignments: 40% (1 point / assignment) Final test: 60% Optional project: 60%-80% Any combination is allowed ! Neural Networks - Lecture 1 45 Course materials http://web.info.uvt.ro/~dzaharie/ • C. Bishop – Neural Networks for Pattern Recognition, Clarendon Press, 1995 • L. Chua, T. Roska – Cellular neural networks and visual computing , Cambridge.University Press, 2004. • A. Engelbrecht – Computational Intelligence. An Introduction, 2007 • M.Hagan, H. Demuth, M.Beale – Neural Networks Design, PWS Publishing 1996 • Hertz,J.;Krogh,A.;Palmer,R. - Introduction to the Theory of Neural Computation, Addison-Wesley P.C., 1991. • Ripley, B.D. - Pattern Recognition and Neural Networks, Cambridge University Press, 1996. Neural Networks - Lecture 1 46