Download CSE 5290: Artificial Intelligence

Artificial Intelligence and Decision Making
Session13: Learning in Neural and Belief Networks
19.1 How the Brain Works
19.1.1 Comparing brains with digital computers
19.2 Neural Networks
19.2.1 Notation
19.2.2 Simple computing elements
19.2.3 Network structures
19.2.4 Optimal network structure
19.3 Perceptrons
19.3.1 What perceptrons can represent
19.3.2 Learning linearly separable functions
19.4 Multilayer Feed-Forward Networks
19.4.1 Back – Propagation Learning
19.4.2 Back – propagation as gradient descent search
19.4.3 Discussion
19.5 Applications of Neural Networks
19.5.1 Pronunciation
19.5.2 Handwritten character recognition
19.5.3 Driving
19.6 Bayesian Methods for Learning Belief Networks
19.6.1 Bayesian learning
19.6.2 Belief network learning problems
19.6.3 Learning networks with fixed structure
19.6.4 A comparison of belief networks and neural networks
19.7 Summary
NOTE: Limitations and Future Trends in Neural Computation. A NATO Advanced
Research Workshop. Siena, 22-24 October 2001.
Web Sites
http://www.boeing.com/nosearch/ijcai/tutorial_program.htm#MA5
http://www.inns.org/
http://www.aist.go.jp/NIBH/~b0616/Lab/Links.html
http://www.ewh.ieee.org/tc/nnc/
http://www.gc.ssr.upm.es/inves/neural/ann1/anntutorial.html
http://www.mathworks.com/products/neuralnet/
http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html
From: http://www.emsl.pnl.gov:2080/proj/neuron/neural/what.html
What is a Neural Network and How does it work?
Also referred to as connectionist architectures, parallel distributed
processing, and neuromorphic systems, an artificial neural network (ANN)
is an information-processing model inspired by the way the densely
interconnected, parallel structure of the mammalian brain processes
information. Artificial neural networks are collections of mathematical
models that emulate some of the observed properties of biological nervous
systems and draw on the analogies of adaptive biological learning. The
key element of the ANN paradigm is the novel structure of the information
processing system. It is composed of a large number of highly
interconnected processing elements that are similar to neurons and are tied
together with weighted connections that are analogous to synapses.
Learning in biological systems involves adjustments to the synaptic
connections that exist between the neurons. This is true of ANNs as well.
Learning typically occurs by example through training, or exposure to a
trusted set of input/output data where the training algorithm iteratively
adjusts the connection weights (synapses). These connection weights store
the knowledge necessary to solve specific problems.
Although ANNs have been around since the late 1950's, it wasn't until the
mid-1980's that algorithms became sophisticated enough for general
applications. Today ANNs are being applied to an increasing number of
real- world problems of considerable complexity. They are good pattern
recognition engines and robust classifiers, with the ability to generalize in
making decisions about inexact input data. They offer ideal solutions to a
variety of classification problems such as speech, character and signal
recognition, as well as functional prediction and system modeling where
the physical processes are not understood or are highly complex. ANNs
may also be applied to control problems, where the input variables are
measurements used to drive an output actuator, and the network learns the
control function. The advantage of ANNs lies in their resilience against
distortions in the input data and their capability of learning. They are often
good at solving problems that are too complex for conventional
technologies (e.g., problems that do not have an algorithmic solution that
are sometimes called NP complete problems or for which an algorithmic
solution is too complex to be found) and are often well suited to problems
that people are good at solving, but for which traditional methods are not.
There are multitudes of different types of ANNs. Some of the more
popular include the multilayer perceptron which is generally trained with
the backpropagation of error algorithm, learning vector quantization,
radial basis function, Hopfield, and Kohonen, to name a few. Some ANNs
are classified as feedforward while others are recurrent (i.e., implement
feedback) depending on how data is processed through the network.
Another way of classifying ANN types is by their method of learning (or
training), as some ANNs employ supervised training while others are
referred to as unsupervised or self-organizing. Supervised training is
analogous to a student guided by an instructor. Unsupervised algorithms
essentially perform clustering of the data into similar groups based on the
measured attributes or features serving as inputs to the algorithms. This is
analogous to a student who derives the lesson totally on his or her own.
ANNs can be implemented in software or in specialized hardware.
Links:
represents a working link, represents a broken link, represents a verified
dead link, and represents a closed link, as determined during the last link check
on 1 June 1997 or by reported problems.
represents a link added in the last 90
days (i.e., since 26 October 2000) Please help us keep this service useful by
letting us know when you change your URLs.
Sources:












Introduction: Neural Network Computing: The Sudden Rise of
Neurocomputing (Accel Infotech S Pte Ltd.)
Introduction: NeuroFuzzy Computing (Accel Infotech S Pte Ltd.)
Introduction: Genetic Algorithms and Genetically Evolved Neural
Networks (Accel Infotech S Pte Ltd.)
Introduction: NeuroGenetic Computing (Accel Infotech S Pte Ltd.)
What are Artificial Neural Networks? (Accurate Automation
Corporation - AAC)
Neural Networks: An Overview (Advanced Technology Transfer
Group)
What are Neural Networks? (AI Intelligence)
Artificial Neural Networks (AiMaze)
A neural model of fear may lead to a better understanding of other
emotions (American Psychological Association - APA)
Network aids psychologists with their business needs (American
Psychological Association - APA)
An introduction to neural networks by Ben Kröse (University of
Amsterdam - UvA)
What Is A Neural Network ? (Attrasoft)































Réseaux de neurones artificiels (AVENTI - Agence pour la
Valorisation et l脱nseignement du Neuro-Traitement de
l棚nformation)
What are neural networks? (BioComp Systems Inc.)
Neural Network Background (BitStar International)
Úvod do umelých neuronových sítí (Technical University of Brno)
Neural Nets by Tom D. Grove (Technical University of Brno)
Mutual Fund Net Asset Value Forecasting Using Neural Networks
(Central Missouri State University)
Software engineering glossary: Neural Networks (CERN)
Making BrainWaves (CIO Magazine )
An Introduction to Neural Networks (David Clark)
Adaptive Logic Networks Technical Overview (Dendronic
Decisions Limited)
Artificial Neural Systems (Wesley R. Elsberry)
Genetic Algorithms & Artificial Neural Networks (Wesley R.
Elsberry)
On-line definition of neural networks (Encyclopaedia Britannica)
Neural networks in financial economics: A breif tutorial
(Athanasios Episcopos)
Introduction to Neural Networks (Generation 5)
Introduction to the Self-Organizing Map by Teuvo Kohonen
(Helsinki University of Technology)
A Basic Introduction To Neural Networks (University of Illinois at
Urbana-Champaign - UIUC)
Neural Networks by Christos Stergiou and Dimitrios Siganos
(Imperial College)
Why Neural Networks by Dimitrios Siganos (Imperial College)
Neural Network Questions and Answers by Dimitrios Siganos and
Christos Stergiou (Imperial College)
Neural Networks, the Human Brain and Learning (Imperial
College)
What is a Neural Network by Chris Stergiou (Imperial College)
Neural Networks Introduction by Vikram Pudi (Indian Institute of
Science - IIS)
Neural Networks at Your Fingertips (Karsten Kutza )
Neural Networks (LBS Capital Management Inc.)
How long before superintelligence? (London School of
Economics)
What is Soft Computing? (Michigan State University - MSU)
Introducing Neural Nets (Neural Computer Sciences - NCS)
Fuzzy / Neurofuzzy Logic (Neural Computer Sciences - NCS)
Neural Computing Theory (Neural Technologies Limited)
What is an Artificial Neural Network? (NeuroDimension Inc.)

















Introduction to Neural Networks (Neusciences)
Glossary of Neural Network Terms (PC AI Magazine)
Definition of Neural Network (PC Webopaedia)
Overivew of Neural Networks (PMSI)
Guide du concepteur de r駸eaux de neurones (PMSI)
Neural net overview by Jochen Fröhlich (Fachhochschule
Regensburg )
Neural FTP Archive (SAS Institute)
Neural Nets by Kevin Gurney (University of Sheffield)
Artificial neural networks: a developing science (Society of PhotoOptical Instrumentation Engineers - SPIE)
An Introduction to Neural Networks by Leslie Smith (Stirling
University)
Backpropagator's Review (Donald R. Tveter)
The Basis of AI (Donald R. Tveter)
The brain as a Darwin Machine (University of Washington)
Neural Network Definition from the Grolier Electronic
Encyclopedia (University of Washington)
Artificial Neural Networks (West Virginia University - WVU)
What is a Neural Network? (whatis?.com)
An Introduction to Neural Networks (Z Solutions)
Additional Introductory Material on Neural
Networks and Connectionism
Links:
represents a working link, represents a broken link, represents a verified
dead link, and represents a closed link, as determined during the last link check
on 1 June 1997 or by reported problems.
represents a link added in the last 90
days (i.e., since 26 October 2000) Please help us keep this service useful by
letting us know when you change your URLs.
Sources:




Introduction: Neural Network Computing: The Sudden Rise of
Neurocomputing (Accel Infotech S Pte Ltd.)
Introduction: NeuroFuzzy Computing (Accel Infotech S Pte Ltd.)
Introduction: Genetic Algorithms and Genetically Evolved Neural
Networks (Accel Infotech S Pte Ltd.)
Introduction: NeuroGenetic Computing (Accel Infotech S Pte Ltd.)




























What are Artificial Neural Networks? (Accurate Automation
Corporation - AAC)
Neural Networks: An Overview (Advanced Technology Transfer
Group)
What are Neural Networks? (AI Intelligence)
Artificial Neural Networks (AiMaze)
A neural model of fear may lead to a better understanding of other
emotions (American Psychological Association - APA)
Network aids psychologists with their business needs (American
Psychological Association - APA)
An introduction to neural networks by Ben Kröse (University of
Amsterdam - UvA)
What Is A Neural Network ? (Attrasoft)
Réseaux de neurones artificiels (AVENTI - Agence pour la
Valorisation et l脱nseignement du Neuro-Traitement de
l棚nformation)
What are neural networks? (BioComp Systems Inc.)
Neural Network Background (BitStar International)
Úvod do umelých neuronových sítí (Technical University of Brno)
Neural Nets by Tom D. Grove (Technical University of Brno)
Mutual Fund Net Asset Value Forecasting Using Neural Networks
(Central Missouri State University)
Software engineering glossary: Neural Networks (CERN)
Making BrainWaves (CIO Magazine )
An Introduction to Neural Networks (David Clark)
Adaptive Logic Networks Technical Overview (Dendronic
Decisions Limited)
Artificial Neural Systems (Wesley R. Elsberry)
Genetic Algorithms & Artificial Neural Networks (Wesley R.
Elsberry)
On-line definition of neural networks (Encyclopaedia Britannica)
Neural networks in financial economics: A breif tutorial
(Athanasios Episcopos)
Introduction to Neural Networks (Generation 5)
Introduction to the Self-Organizing Map by Teuvo Kohonen
(Helsinki University of Technology)
A Basic Introduction To Neural Networks (University of Illinois at
Urbana-Champaign - UIUC)
Neural Networks by Christos Stergiou and Dimitrios Siganos
(Imperial College)
Why Neural Networks by Dimitrios Siganos (Imperial College)
Neural Network Questions and Answers by Dimitrios Siganos and
Christos Stergiou (Imperial College)




























Neural Networks, the Human Brain and Learning (Imperial
College)
What is a Neural Network by Chris Stergiou (Imperial College)
Neural Networks Introduction by Vikram Pudi (Indian Institute of
Science - IIS)
Neural Networks at Your Fingertips (Karsten Kutza )
Neural Networks (LBS Capital Management Inc.)
How long before superintelligence? (London School of
Economics)
What is Soft Computing? (Michigan State University - MSU)
Introducing Neural Nets (Neural Computer Sciences - NCS)
Fuzzy / Neurofuzzy Logic (Neural Computer Sciences - NCS)
Neural Computing Theory (Neural Technologies Limited)
What is an Artificial Neural Network? (NeuroDimension Inc.)
Introduction to Neural Networks (Neusciences)
Glossary of Neural Network Terms (PC AI Magazine)
Definition of Neural Network (PC Webopaedia)
Overivew of Neural Networks (PMSI)
Guide du concepteur de r駸eaux de neurones (PMSI)
Neural net overview by Jochen Fröhlich (Fachhochschule
Regensburg )
Neural FTP Archive (SAS Institute)
Neural Nets by Kevin Gurney (University of Sheffield)
Artificial neural networks: a developing science (Society of PhotoOptical Instrumentation Engineers - SPIE)
An Introduction to Neural Networks by Leslie Smith (Stirling
University)
Backpropagator's Review (Donald R. Tveter)
The Basis of AI (Donald R. Tveter)
The brain as a Darwin Machine (University of Washington)
Neural Network Definition from the Grolier Electronic
Encyclopedia (University of Washington)
Artificial Neural Networks (West Virginia University - WVU)
What is a Neural Network? (whatis?.com)
An Introduction to Neural Networks (Z Solutions)
Neural Network Demos
From: http://www.emsl.pnl.gov:2080/proj/neuron/neural/
Description:
This page contains links to web sites with demos on the topic of Neural Networks
and Connectionism. These demos are generally web based (e.g., Java and CGI
scripts). If you experience problems with a demo or are unable to run a demo,
please contact the author of the demo or the administrator of the associated web
site.
URL:
The URL of this page is
http://www.emsl.pnl.gov:2080/proj/neuron/neural/demos.html
Additions:
We always welcome additions! To add a link or make a correction, please use the
Add Demo Web Page form or send e-mail to the address given at the bottom of
this page.
Links:
represents a working link, represents a broken link, represents a verified
dead link, and represents a closed link, as determined during the last link check
on 1 June 1997 or by reported problems.
represents a link added in the last 90
days (i.e., since 26 October 2000) Please help us keep this service useful by
letting us know when you change your URLs.
Demos:










EightPuzzle Applet in A* (AI Engineering)
o Description:
PCNN Net Demo (Alabama A & M)
o Description: A demo Pulse Coupled Neural Network Java applet.
SOM Search of the Web (University of Arizona)
o Description: An adaptive 2-D Kohonen-based Self-organizing Map has
been developed for use with Alta Vista searching. This SOM allows user
customization of the level of categorization the tool provides.
Fisheye and Fractal SOM (University of Arizona)
o Description: Generated in 1997 on a set of 200 electronic brainstorming
comments, this version of the self-organizing map uses a 3-D
representation to demonstrate relationships between concepts.
Artificial Neural Networks Overview (Battelle Memorial Institute)
Artificial Neural Networks: A Quick Introduction (Bradley University)
o Description:
Motion Planner Applet using A* (University of California at Berkeley)
o Description: A demo of the A* algorithm.
Backprop_XOR (Caltech - California Institute of Technology)
o Description: This Java simulation implements the backpropagation error
learning algorithm to train a network with two synaptic layers with two
inputs and a single output. The network can be trained to emulate the
functions of XOR, AND, OR, etc.
Neural Network Demo (Cambridge University)
o Description:
Dynamic Associative Neural Memory Simulator (David Clark)
o Description: The Dynamic Associative Neural Memory Simulator
performs the following learning algorithms: Hopfield, Optimal Linear















Associative Memory (OLAM), Semi-Adaptive OLAM, Adaptive OLAM,
Perceptron, Adaline, and the learning algorithm presented in David Clark's
thesis
Hopfield Network JAVA Demo (Dublin City University - DCU)
Self-Organizing Map Demo by Simon Lucas (University of Essex)
o Description: This is a simple demonstration of using self-organising
feature maps to project a high-dimensional space onto a lowerdimensional space - though in this case the mapping is from a 3-d rgb
colour to a 2-d grid.
The HTML Neural Net Consulter (hav.Software)
o Description: Several example Neural Net Consultation apps are provided
including 2-way XOR and 3D Kohonen.
Binary Hopfield Applet (Matt Hill)
o Description:
KnowMan on the Internet (Intellix A/S)
o Description: Demonstrations of our intractive knowledge mapping
technology and experience how it works on the Internet.
Black Jack and Reinforcement Learning (École Polytechnique Fédérale de
Lausanne - EPFL)
o Description: A Java applet uses a reinforcement learning algorithm to
play a simplified version of the game of Black Jack. One or two players
can play against the dealer (i.e., the casino).
IHearYourPitch 1.0 (Monowave Corp.)
o Description: Transcription of pitch in speech or music
Artificial Neural Network Lab on the Web (National Institute of Bioscience and
Human-Technology )
o Description: Demonstrations on artificial neural networks using javaapplets and GIF-animations
Bayesian Self-Organizing Map Simulation (National Institute of Bioscience and
Human-Technology )
o Description: The Bayesian self-organizing map (BSOM) is a method for
estimating a probability distribution generating data points on the basis of
a Bayesian stochastic model. It is also regarded as a learning method for a
kind of neural network.
Calculate Weld Pool Shape of Pulsed Laser Welding Process via Neural Network
(Oak Ridge National Laboratory - ORNL)
o Description:
Neural Network Ferrite Number Predictions for Stainless Steel Welds (Oak Ridge
National Laboratory - ORNL)
o Description:
Traveling Salesman Applet with Kohonen SOM (Patol)
XOR Applet with Backprop (Patol)
o Description: XOR applet is a little example that implements a 3-input
XOR gate with a 3 layered neuronal network.
SOM Applet (Patol)
The Boltzmann Machine: Necker Cube Example (University of Queensland)
o











Description:
Neural Networks with Java by Jochen Fröhlich (Fachhochschule Regensburg )
o Description: Examples of backprogation and Kohonen self-organizing
feature maps with full documentation and description of the JAVA code.
DemoGNG (Ruhr Universität Bochum)
o Description: DemoGNG, a Java applet, implements several methods
related to competitive learning. It is possible to experiment with the
methods using various data distributions and observe the learning process.
A common terminology is used to make it easy to compare one method to
the other.
Real Estate Appraisal Demo (VirtualMind Pty Ltd)
o Description: The neural network behind this demo has had lots of
experience, in fact it has been trained with the sales data of 1700 homes.
From this data the neural network has found general patterns, not rules,
that enable it to appraise the value of homes.
Soybean Disease Diagnosis (VirtualMind Pty Ltd)
o Description: This demonstration illustrates the type of service that could
be offered to agriculturists. It is able to diagnose 15 different diseases
common to Soybean crops.
Heart Disease Diagnosis (VirtualMind Pty Ltd)
o Description: Neural computing gained publicity in a Wall Street Journal
when a neural network was able to diagnose heart attacks with better
accuracy than physicians. This demonstration is of a heart disease
diagnostic service.
Mushroom Edible/Poisonous (VirtualMind Pty Ltd)
o Description: There are thousands of different species of mushrooms some
are edible and others poisonous, even deadly. The neural network behind
this demo is able to identify mushrooms as being edible or poisonous.
Predicting Politics (VirtualMind Pty Ltd)
o Description: Here we have two demonstrations both based on the 1984
United States congressional voting records.
Iris Classifying (VirtualMind Pty Ltd)
o Description: This demonstration is able to classify three types of Iris from
only the size of the sepal and petal.
Animated Perceptron Learning Rule (Wayne State University)
o Description: This program applies the pereceptrone learning rule to draw
a separating surface between to classes of points X & O.
Backprop Tool for Function Approximation and Classification (Wayne State
University)
o Description: This applet can be used to experiment with backprop
learning for function approximation problems. You can choose an
underlying function to be approximated, then choose a number of training
samples, network size, and learning rate.
Neural Nets for Control: The Ball Balancing Problem (Wayne State University)
o




Description: This problem is a classic regulator-type control problem and
is precisely posed as: Given any initial condition , what is an appropriate
control signal , which can produce the desired final state ? A neural net
can be trained to learn such a control by observing the actions of a skilled
human operator.
Image Compression using Backprop (Wayne State University)
o Description: Computer images are extremely data intensive and hence
require large amounts of memory for storage. As a result, the transmission
of an image from one machine to another can be very time consuming. By
using data compression techniques, it is possible to remove some of the
redundant information contained in images, requiring less storage space
and less time to transmit. Neural nets can be used for the purpose of image
compression.
Generalizations of the Hamming Associative Memory (Wayne State University)
o Description:
2D Time Dynamical System Java Program (Wayne State University)
o Description: The program plots the state trajectory of this system from
the starting point (you selected). You can change the value of delta T.
Localizing Algorithm with WebSim (Wright Laboratory)
o Description: This is an on-line demo of Radial Basis Function networks
and 2-layer sigmoidal networks.
Explore Other Related On-Line Web Demos:
EMSL Web Site
Cognitive Systems
Summary of Cognitive Systems On-Line Web Demos
Computational Intelligence
Summary of Computational Intelligence On-Line Web Demos
Neural Networks On-Line Web Demos
Fuzzy Logic On-Line Web Demos
Evolutionary Computation On-Line Web Demos
Classical Artificial Intelligence
Summary of Classical Artificial Intelligence On-Line Web Demos
Knowledge-Based Systems On-Line Web Demos
Biological/Natural Intelligence
Summary of Biological/Natural Intelligence On-Line Web Demos
Cognitive Science On-Line Web Demos