Download PPT

Computational Intelligence
John Sum
Institute of Technology Management
National Chung Hsing University
Taichung, Taiwan ROC
OUTLINE




Historical Background
Computational Intelligence
Example Problems
Methodology





Model Structure
Model Parameters
Parametric Estimation
Discussion
Conclusion
John Sum
Computational Intelligence
2
HISTORY
John Sum
Computational Intelligence
3
HISTORY





1940 – First computing machine
1957 – Perceptron (First NN model)
1965 – Fuzzy Logic (Rules)
1960s – Genetic Algorithm
1970s – Evolutionary Computing
John Sum
Computational Intelligence
4
HISTORY

1980s



Neural Computing
Swarm Intelligence
1990s (Hybrid)


John Sum
Fuzzy Neural Networks
NFG, FGN, GNF, etc
Computational Intelligence
5
HISTORY

Beyond 1990s: Research areas converge




Computational Intelligence
Softcomputing
Intelligent Systems
Covering





John Sum
Adaptive Systems
Fuzzy Systems
Neural Networks
Evolutionary Computing
Data Mining
Computational Intelligence
6
COMPUTATIONAL INTELLIGENCE

Computational Intelligence

Heuristic algorithms (or models) such as in fuzzy
systems, neural networks and evolutionary
computation.

Techniques that use Simulated annealing, Swarm
intelligence, Fractals and Chaos Theory, Artificial
immune systems, Wavelets, etc.
John Sum
Computational Intelligence
7
COMPUTATIONAL INTELLIGENCE

Goal: Problem Solving








John Sum
Financial forecast
Customer segmentation (CRM)
Supply chain design (SCM)
Business process re-engineering
System control
Pattern recognition
Image compression
Homeland security
Computational Intelligence
8
COMPUTATIONAL INTELLIGENCE

Underlying structure of the model is unknown, or the
model is known but it is too complicated
Example: DJI versus HIS (Time Series)

Define system structure




NL model (NN, ODE, etc.)
Rule-based system
Parametric estimation


John Sum
Deterministic search (Gradient descent or Newton’s
method)
Stochastic search (SA or MCMC)
Computational Intelligence
9
COMPUTATIONAL INTELLIGENCE

Underlying model structure is known
Example: Manufacturing process (SCM)

Define the objective to be maximized



Examples: Completion time, Cost, Profit
Optimization



John Sum
Linear programming, ILP, NLP
Deterministic search (Gradient descent or Newton’s
method)
Stochastic search (SA or MCMC)
Computational Intelligence
10
EG1: Nonlinear Dynamic System
Unknown
system
x
John Sum
Noise
g (x )
Computational Intelligence
y
11
EG2: Nonlinear Function
Unknown
system
x
John Sum
Noise
g (x )
Computational Intelligence
y
12
EG3: Car Price

Predict the price of a car based on





John Sum
Specification of an auto in terms of various
characteristics
Assigned insurance risk rating
Normalized losses in use as compared to other
cars
Number of attributes: 25
Missing values: Yes!
Computational Intelligence
13
EG3: Car Price
John Sum
Computational Intelligence
14
EG4: Purchasing Preference
John Sum
Computational Intelligence
15
EG5: Financial Time Series
14000
13000
12000
11000
10000
9000
8000
7000
1
John Sum
159 317 475 633 791 949 1107 1265 1423 1581 1739
Computational Intelligence
16
EG5: Financial Time Series

What would happen in the next trading day?
(Time series prediction problem)




Closing value
Open value
UP or DOWN
Time series prediction + trading rules


John Sum
What should I do tomorrow? HOLD, SELL or BUY
When should I BUY and SELL?
Computational Intelligence
17
Remarks on EG1 ~ EG5
System
Structure
Data Types
Model
Dynamic
System
Unknown
Continuous
RNN, Fuzzy NN
Nonlinear
Function
Unknown
Continuous
BPN, RBF, Fuzzy NN
Car Price
Unknown
Continuous
Discrete
BPN, RBF, Fuzzy NN
Purchasing
Preference
Known (SEM)
Discrete
SEM
Bayesian Net
Financial Time
Series
Unknown
Continuous
BPN, RBF, Fuzzy NN
John Sum
Computational Intelligence
18
COMPUTATIONAL INTELLIGENCE
Statement of Problem

Given a set of data collected (or measured) from a
system (probably an unknown system), devise a
model (by whatever structure, technique, method in
CI) that mimics the behavior of that system as ‘good’
as possible.

Making use of the devised model to




John Sum
(1) interpret the behavior of the system,
(2) predict the future behavior of the system,
(3) control the behavior of the system,
(4) make money.
Computational Intelligence
19
METHODOLOGY

Step 1: Data Collection





Experiments or measurements
Questionnaire
Magazine
Public data sets
Step 2: Model Structure Assumption


John Sum
IF it is known, SKIP this step.
ELSE, DEFINE a model structure
Computational Intelligence
20
METHODOLOGY

Step 3: Parametric Estimation








John Sum
Gradient descent
Newton’s method
Exhaustive search
Genetic algorithms (*)
Evolutionary algorithms (*)
Swarm intelligence
Simulated annealing (*)
Markov Chain Monte Carlo (*)
Computational Intelligence
21
METHODOLOGY

Step 4: Model Validation (is it a reasonable
good model)




Hypothesis test
Validation/Testing set
Leave one out validation
Step 5: Model Reduction (would there be a
simpler model that is also reasonable good)


John Sum
AIC, BIC, MDL
Pruning (using testing set)
Computational Intelligence
22
METHODOLOGY

Beyond Model Reduction




John Sum
Any redundant input
Any redundant sample (or outlier)
Any better structure (alternative)
How do we determine a ‘good’ model
Computational Intelligence
23
NN MODEL STRUCTURES


















Perceptron
Multilayer Perceptron (MLP or BPN)
Adaptive Resonance Theory Model (ART)
Competitive Learning (CL)
Hopfield Network, Associative Network
Bidirectional Associative Model (BAM)
Recurrent Neural Network (RNN)
Boltzmann Machine
Brain-State-In-A-Box (BSB)
Radial Basis Function Network (RBF Net)
Bayesian Networks
Self Organizing Map (SOM or Kohonen Map)
Learning Vector Quantization (LVQ)
Support Vector Machine (SVM)
Support Vector Regression (SVR)
PCA, ICA, MCA
Winner-Take-All Network (WTA)
Spike neural networks
John Sum
Computational Intelligence

Remarks



Not all of them is able to learn,
eg BSB, WTA
Might need to combine two
structures to solve a single
problem
Multiple definitions on the
‘neuron’
24
NN MODEL STRUCTURES

Supply Chain Management (Optimization Problem)


Customer Segmentation (Clustering Problem)


RNN, Recurrent RBF
Car Price/NL Function (Function Approximation)


CL, SOM, LVQ, ART
Dynamic Systems Modeling


Hopfield Network
MLP, RBF Net, Bayesian Net, SVR, +SOM/LVQ
Financial TS (FA or Time Series Prediction)

John Sum
RNN, SVR, MLP, RBF Net, + SOM/LVQ
Computational Intelligence
25
FUZZY MODEL STRUCTURE
John Sum
Computational Intelligence
26
FUZZY MODEL STRUCTURE
John Sum
Computational Intelligence
27
NN MODEL PARAMETERS

MLP




Input Weights
Output Weights
Neuron model
RNN




Input Weights
Output Weights
Recurrent Weights
Neuron model
John Sum
Computational Intelligence
28
NN MODEL PARAMETERS
John Sum
Computational Intelligence
29
NN MODEL PARAMETERS
John Sum
Computational Intelligence
30
NN MODEL PARAMETERS
John Sum
Computational Intelligence
31
FUZZY MODEL PARAMETERS
John Sum
Computational Intelligence
32
PARAMETRIC ESTIMATION
John Sum
Computational Intelligence
33
PARAMETRIC ESTIMATION
Gradient Descent
John Sum
Computational Intelligence
34
PARAMERTIC ESTIMATION
Genetic Algorithm
John Sum
Computational Intelligence
35
PARAMERTIC ESTIMATION
Genetic Algorithm
John Sum
Computational Intelligence
36
PARAMERTIC ESTIMATION
Genetic Algorithm
John Sum
Computational Intelligence
37
DISCUSSIONS

CI is not the only method (or structure) to
solve a problem.

Even it can solve, its performance might not
be better than other methods.

Should compare with other well-known or
existing methods
John Sum
Computational Intelligence
38
DISCUSSIONS

SCM Problem





LP, LIP, NLP
Lagrangian Relaxation
Cutting Plane
CPLEX
Function Approximation



John Sum
Polynomial Series
Trigonometric Series
B-Spline
Computational Intelligence
39
CONCLUSIONS

IF




The problem to be solved has been well
formulated
The structure has been selected
The objective function to evaluation the goodness
of a parametric vector has been defined
THEN

John Sum
Every problem is just an optimization problem
Computational Intelligence
40
JOHN SUM ([email protected])










Taiwan HK-Chinese, PhD (98) and MPhil (95) from CUHK, BEng (92)
from PolyU HK.
Taught in HK Baptist University (98-00), OUHK (00) and PolyU HK
(00-04), Chung Shan Medical University (05-07)
Adj. Associate Prof., Institute of Software, CAS Beijing (99-02)
Short visit: CityU HK, Griffith University in Australia, FAU, Boca
Raton FL US, CAS in Beijing, Ching Mai University in Thailand.
Assist. Prof., IEC (07-09), Asso. Prof., ITM (09-) NCHU Taiwan
2000 Marquis Who's Who in the World.
Senior Member of IEEE, CI Society, SMC Society (05-)
GB Member, Asia Pacific Neural Network Assembly (09-)
Associate Editor of the IJCA (05-09)
Research Interests include NN, FS, SEM, EC, TM
John Sum
Computational Intelligence
41