Download 1 =A T

A TSK-Type Neuro-fuzzy Network
Approach to System Modeling
Problems
Chen-Sen Ouyang Wan-Jui Lee
Shie-Jue Lee
Presented by: Pujan Ziaie
1
Authors (1)
• Chen-Sen Ouyang
– born in Kin-Men, Taiwan
– Received Ph.D. degree from the National Sun
Yat-Sen University, Kaohsiung, Taiwan, in
2004
– Research interests:
• Soft computing, data mining, pattern recognition, video
processing
– member of the Taiwanese Association of
Artificial Intelligence
Hirota lab
2/45
Authors (2)
• Wan-Jui Lee
– born in Tainan, Taiwan
– Received B.S. degree from the National Sun
Yat-Sen University, Kaohsiung, Taiwan, in
2000
– Research interests:
• data mining, fuzzy set theory, neural networks,
and support vector learning
Hirota lab
3/45
Authors (3) -Professor
• Shie-Jue Lee
– born in Kin-Men, Taiwan
– Received Ph.D. degree from the University of
North Carolina , Chapel Hill, in 1990
– Research interests:
• machine intelligence, data mining, soft computing,
multimedia communications, and chip design
– Received the Excellent Teaching Award of
National Sun Yat-Sen University
– Chairman of the Electrical Engineering
Department from 2000
Hirota lab
4/45
Universities (Taiwan)
• I-Shou (C-S. Ouyang)- Kaohsiung
•National Sun Yat-Sen
University (W.-J. Lee
and S.-J. Lee)
Hirota lab
5/45
Paper info
• Manuscript received June 18, 2004
• revised November 18, 2004
• supported by the National Science Council
• recommended by Editor H.-X. Li
• IEEE TRANSACTIONS ON SYSTEMS, MAN,
AND CYBERNETICS—PART B:
CYBERNETICS, VOL. 35, NO. 4, AUGUST
2005
Hirota lab
6/45
Outline
• Rudiments of Fuzzy control & Neural Networks
• Paper Introduction
• Rule extraction: Merged-based fuzzy
clustering:
• Rule Refinement: Neural Networks
• Experimental results
• Conclusion
Hirota lab
7/45
µAc(t) = 1 - µA(t)
What isFuzzy logic

Proposed by professor Lotfizadeh-1964
Mathematical idea – (worst way to explain)

Applicative explanation

• Crisp logic: 0 or 1 >> Fuzzy logic [0..1]
• Use a membership function instead of 0,1
• A way of describing the word by linguistic,
•
inexact, fuzzy variables
Explain the behavior of a system through
linguistic variables
Hirota lab
8/45
Membership functions



Defining fuzzy variables by using
membership functions
Common functions:
Example: “youngness”
Youngness grade
97%
100%
1
10Natori-san35
Hirota lab
9/45
age
Fuzzy control concept

Using if-then rules with linguistic
variables instead of differential equations
Example: Riding Unicycle
Classic way of stabilization:
•large number of non-linearity
•unknown and variables such as friction and
total mass
Hirota lab
10/45
Fuzzy control example

How much tip according to service and food quality?
max
Defuzzification
Centroid Average (CA)
Maximum Center Average (MCA)
Mean of Maximum (MOM)
Smallest of Maximum (SOM)
Largest of Maximum (LOM)
inputs Hirota lab
output
11/45
TSK-type fuzzy rule

Proposed by: Takagi, Sugeno, and Kang
fuzzy inputs but crisp outputs (constant or a function)
If X is X1 and Y is Y1 then z = f1(x,y)
Example:

If pressure is low and temperature is medium then valve opening is 5*p + 3*t

If X is X1 and Y is Y1 then z = f1(x,y)
If X is X2 and Y is Y2 then z = f2(x,y)
...
If X is Xn and Y is Yn then z = fn(x,y)
Defuzzification:
(wi) being the degree of matching (product of µ(xi))
(w1*f1(x,y) + w2*f2(x,y) + … + wn*fn(x,y)) / (w1 + w2 + …
+ wn) = ∑wn*fn(x,y)/ ∑wn






Hirota lab
12/45
Neural Networks

Perform learning to approximate a
desired function
w1
I1
w2
I2
w3
I3


f(I,W, T)
wn
Useful when

Activation
vector
processor
In
lots of examples of the behaviour
we can't formulate an algorithmic solution
Hirota lab
13/45
Linear Neural Networks(1)

Model the existing in-out data by the
simple function: y = w2 x + w1
Hirota lab
14/45
Linear Neural Networks(2)


1I1
Consider the simple N-N
Adjust the weights to map the function
by minimizing the loss
W1
Adder and
Activation function
O
Inputs
Σ g (x)
Output
xI2
W2
weight
Hirota lab
15/45
Gradient descent Algorithm
I1

Computing the gradient
W1
Adder and
Activation function
O
Inputs
Σ g (x)
Output
I2
W2
weight
Learning rate
Hirota lab
16/45
Multi-layer N-N

Add a hidden layer
Input layer
Hidden layer
Neurones
Out layer layer
Output = w0+w1*f(x1) +.. +wn*f(xn)
Hirota lab
17/45
Multi-layer N-N learning

Forward activation


Calculating output error


passing (or feeding) the signals
through the network
Mean squared error (MSE)
Error backpropagation
Hirota lab
18/45
Introduction
• Modeling
– System is unknown
– Measured input-output data available
• Neuro-fuzzy systems
– Fuzzy rules: describe the behavior (structure)
– Neural Networks: adjust control parameters
and improvement of the system (refinement)
Hirota lab
19/45
Introduction
• Modeling process
Hybrid learning
algorithm
Merged-based
fuzzy clustering
In-Out data Self-Constructing
Rule Generation Fuzzy rules
Structure Identification
Neural Network Result
Final
Fuzzy
Rules
Parameter identification
Hirota lab
20/45
Controller Structure
• Fuzzy rules from Data
– If x is Mx1 Then y is Mz1
– If x is Mx2 Then y is Mz2
X
– 2 inputs
– If x is Mx2 and y is My1
Then y is Mz2
Mx2
Mx1
Mz1
Mz2
z
y
Hirota lab
21/45
[38],[42]
Our approach
• Gaussian function for membership
functions
• J final clusters of data
• TSK fuzzy rule:
– If x1 is µ1j(x1) AND x2 is µ2j(x2) AND … AND
xn is µnj(xn) THEN
y = b0j +b1jx1+..+bnjxn
b0j > m0j
Deviation vector
Mean vector
(m)
b1j..bnj > temporarily 0 (can not be deduced)
Hirota lab
22/45
Clustering
• To define input
membership functions
regions
– Data partitioning
• Sensitive to input order
• Might be redundant
– Clusters merge
• Merging similar clusters
Hirota lab
•n inputs> x1…xn, one output
•Cj > fuzzy cluster
23/45
Data partitioning
• N: number of training patterns
• tv: pattern (1≤v≤N)
• Sj: Size of cluster Cj
• comb > operator to combine
Cj and tv
Hirota lab
24/45
Combination
• Combining: Changing Gaussian functions
of cluster
Initial deviations
User-defined
constant
Hirota lab
25/45
Partitioning process
• tv> New training instance
• Calculate Ij(p v) > compare with ρ
(threshold)
• Calculate O'j(y)=comb_y(Oj(y),qv) and
compare ơ'oj with η (threshold)
•If both failed > new cluster k
•If not> add to the most appropriate Cluster
Using the combination method
•Largest input-similarity
•With minimum output-variance
Hirota lab
26/45
Cluster merge
• Partitions > input order dependant
• Merging > creating efficient clusters
Hirota lab
27/45
Merging conditions
• Calculating input and output similarity
measure rIij , rOij for every two clusters
(Ci,Cj)
• If rIij ≥ ρ , rOij ≥ ε (threshold) Then put them
into the same candidate class X
Hirota lab
28/45
Merging process
• K Clusters combination:
• Clusters are combined if ơ'oj ≤ η
Hirota lab
29/45
Merging process (full procedure)
Merging
clusters
Increasing
ρ,ε
Merged clusters
To
(1+θ)* (ρ , ε )
No
clusters
Input-output
similarity
measurement
Every candidate class
has only one cluster
Finished
Hirota lab
30/45
Rule Refinement

Final clusters >> Final rules (TSK
type)
• C1,C2,…Cj >> R={R1,R2,..Rj}

If x1 IS µ1j(x1) AND If x2 IS µ2j(x2)
AND … AND If xn IS µnj(xn)
THEN yj IS fj(x)=b0j+b1jx1+…+bnjxn
• µij(x1) = exp[-( (xi-mij)/ơij)2]
• b0j = m0j, b1j..bnj=0 (will be learned later)
Hirota lab
31/45
TSK-type fuzzy rule - Reminding





fuzzy inputs but crisp outputs (constant or a
function)
If X is X1 and Y is Y1 then z = f1(x,y)
If X is X2 and Y is Y2 then z = f2(x,y)
...
If X is Xn and Y is Yn then z = fn(x,y)
Defuzzification:
(wi) being the degree of matching (product of
µ(xi))
Y= (w1*f1(x,y) + w2*f2(x,y) + … +
wn*fn(x,y)) / (w1 + w2 + … + wn)
Hirota lab
32/45
Rules’ Calculations(1)
Rule j strength



αj(x )=µ1j(x1)*µ2j(x2)*…*µnj(xn)
=Ij(x )
fj(x)=b0j+b1jx1+…+bnjxn
Final output y:
•µij(x1) = exp[-( (xi-mij)/ ij)2]
•b0j = m0j, b1j..bnj=0
Hirota lab
33/45
Rules’ Calculations(2)

µnj(xn) > αj(x )=∏µij > dj(x ) >
sj(x ) > y
Hirota lab
34/45
Network structure
2
Layer 1:
2: µ
3:
4:
5:
αjnj
d
s
y
(x
(x
=
(x
∑
)=exp[-(
)
)=∏µ
)
s
=
=
(x
d
α
(x
(x
)
(x
)*f
)/∑α
-m
(x
(x
)/ơ
)
)
)
j
n j
jjij
i j jij
ij ]

µ11
x1
µi1
.
.
µn1
xi
(mij,ơij)
x
∏
µ1j
µij
.
.
.
µ1J
xn
µiJ
∏
N
αj(x )
N
s1
dj(x )
sj
y
∑
µnj
µnJ
∏
N
Hirota lab
sJ
35/45
Network refinement
µ11

Hybrid Learning Algorithm
Combination of:
x1
µi1
.
.
µn1
xi
• Recursive SVD-based leastsquare estimator > refine bij
• Gradient decent method >
refine (mij,ơij)
µij
µ1J
xn
µiJ
N
∏
µ1j
.
.
.
Gradient
decent
algorithm
Hirota lab
(mij,ơij)
x
∏
αj(x)
N
s1
dj(x)
sj
µnj
∏
N
sJ
µnJ
SVDbased
LSE
36/45
∑
y
Singular Value Decomposition(1)

N training patterns: tV=
trace(D)=∑n1di
i
||D|| =
(trace(DTD))1/2
Hirota lab
37/45
Singular Value Decomposition(2)

Suppose aij as constant > Finding X*:
the optimal solution to minimize E
DTD=I<>A1=AT

SVD ensures: A=U∑VT
• U: NхN orthonormal matrix
• V: JхJ orthonormal matrix
• ∑: NхJ diagonal matrix with ei:
eigenvalues of ATA Hirota lab
38/45
SVD calculations
Calculating X*: (X the only variable)
U: orthonormal
E(X)=||Q-U∑VTX||
E(X)=||UTQ-∑VTX|| > VTX=Y
TQ= Q’
∑= ∑’
U
0
Q’’

E(X)=
Q’
Q’’
-
∑’ Y
0
=
Q’- ∑’Y
Q’’
Hirota lab
minimized
Q’-∑’Y*=0
X*=VY*
39/45
Gradient Decent Method

To refine (mij,ơij)
Hirota lab
40/45
Summary of process
µ11
x1
µi1
.
.
µn1
xi
(mij,ơij)
x
µ1j
µij
.
.
.
µ1J
xn
µiJ
N
∏
∏
αj(x)
N
s1
dj(x)
sj
∑
y
µnj
∏
N
sJ
µnJ
Mergedbased fuzzy
clustering
Hybrid learning
algorithm
Self-Constructing Fuzzy rules
Rule Generation
Neural Network
Hirota lab
Final
Result Fuzzy
Rules
41/45
Experimental results

Modeling the function
Refinement through Neural Network
µ11
x1
7 Rules:
4
clusters
>4
7 Rules
s
∏
N
Final Fuzzy
µ
1
i1
µn1
.
.
xi
x
(mij,ơij)
µ1j
µij
.
.
.
µ1J
xn
µiJ
∏
αj(x)
N
dj(x)
sj
∑
µnj
∏
N
sJ
µnJ
Hirota lab
42/45
y
Comparison of methods

Modeling function:
Yen’s System
Juang’s System
Hirota lab
This paper’s System
43/45
Conclusion


Using differential equations is useless
for many control problems
Advantages of this approach
• flexibility
• Simplicity

Disadvantages
• No explanation on methods used
• Data-set needed
Hirota lab
44/45
Thank you for listening

Any questions?
Hirota lab
45/45
Thank you for listening

Any questions?
Hirota lab
46/45
µ11
x1
µi1
.
.
µn1
xi
(mij,ơij)
x
∏
µ1j
µij
.
.
.
µ1J
xn
µiJ
∏
N
αj(x )
N
s1
dj(x )
sj
∑
µnj
µnJ
∏
N
Hirota lab
sJ
47/45