Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A TSK-Type Neuro-fuzzy Network Approach to System Modeling Problems Chen-Sen Ouyang Wan-Jui Lee Shie-Jue Lee Presented by: Pujan Ziaie 1 Authors (1) • Chen-Sen Ouyang – born in Kin-Men, Taiwan – Received Ph.D. degree from the National Sun Yat-Sen University, Kaohsiung, Taiwan, in 2004 – Research interests: • Soft computing, data mining, pattern recognition, video processing – member of the Taiwanese Association of Artificial Intelligence Hirota lab 2/45 Authors (2) • Wan-Jui Lee – born in Tainan, Taiwan – Received B.S. degree from the National Sun Yat-Sen University, Kaohsiung, Taiwan, in 2000 – Research interests: • data mining, fuzzy set theory, neural networks, and support vector learning Hirota lab 3/45 Authors (3) -Professor • Shie-Jue Lee – born in Kin-Men, Taiwan – Received Ph.D. degree from the University of North Carolina , Chapel Hill, in 1990 – Research interests: • machine intelligence, data mining, soft computing, multimedia communications, and chip design – Received the Excellent Teaching Award of National Sun Yat-Sen University – Chairman of the Electrical Engineering Department from 2000 Hirota lab 4/45 Universities (Taiwan) • I-Shou (C-S. Ouyang)- Kaohsiung •National Sun Yat-Sen University (W.-J. Lee and S.-J. Lee) Hirota lab 5/45 Paper info • Manuscript received June 18, 2004 • revised November 18, 2004 • supported by the National Science Council • recommended by Editor H.-X. Li • IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART B: CYBERNETICS, VOL. 35, NO. 4, AUGUST 2005 Hirota lab 6/45 Outline • Rudiments of Fuzzy control & Neural Networks • Paper Introduction • Rule extraction: Merged-based fuzzy clustering: • Rule Refinement: Neural Networks • Experimental results • Conclusion Hirota lab 7/45 µAc(t) = 1 - µA(t) What isFuzzy logic Proposed by professor Lotfizadeh-1964 Mathematical idea – (worst way to explain) Applicative explanation • Crisp logic: 0 or 1 >> Fuzzy logic [0..1] • Use a membership function instead of 0,1 • A way of describing the word by linguistic, • inexact, fuzzy variables Explain the behavior of a system through linguistic variables Hirota lab 8/45 Membership functions Defining fuzzy variables by using membership functions Common functions: Example: “youngness” Youngness grade 97% 100% 1 10Natori-san35 Hirota lab 9/45 age Fuzzy control concept Using if-then rules with linguistic variables instead of differential equations Example: Riding Unicycle Classic way of stabilization: •large number of non-linearity •unknown and variables such as friction and total mass Hirota lab 10/45 Fuzzy control example How much tip according to service and food quality? max Defuzzification Centroid Average (CA) Maximum Center Average (MCA) Mean of Maximum (MOM) Smallest of Maximum (SOM) Largest of Maximum (LOM) inputs Hirota lab output 11/45 TSK-type fuzzy rule Proposed by: Takagi, Sugeno, and Kang fuzzy inputs but crisp outputs (constant or a function) If X is X1 and Y is Y1 then z = f1(x,y) Example: If pressure is low and temperature is medium then valve opening is 5*p + 3*t If X is X1 and Y is Y1 then z = f1(x,y) If X is X2 and Y is Y2 then z = f2(x,y) ... If X is Xn and Y is Yn then z = fn(x,y) Defuzzification: (wi) being the degree of matching (product of µ(xi)) (w1*f1(x,y) + w2*f2(x,y) + … + wn*fn(x,y)) / (w1 + w2 + … + wn) = ∑wn*fn(x,y)/ ∑wn Hirota lab 12/45 Neural Networks Perform learning to approximate a desired function w1 I1 w2 I2 w3 I3 f(I,W, T) wn Useful when Activation vector processor In lots of examples of the behaviour we can't formulate an algorithmic solution Hirota lab 13/45 Linear Neural Networks(1) Model the existing in-out data by the simple function: y = w2 x + w1 Hirota lab 14/45 Linear Neural Networks(2) 1I1 Consider the simple N-N Adjust the weights to map the function by minimizing the loss W1 Adder and Activation function O Inputs Σ g (x) Output xI2 W2 weight Hirota lab 15/45 Gradient descent Algorithm I1 Computing the gradient W1 Adder and Activation function O Inputs Σ g (x) Output I2 W2 weight Learning rate Hirota lab 16/45 Multi-layer N-N Add a hidden layer Input layer Hidden layer Neurones Out layer layer Output = w0+w1*f(x1) +.. +wn*f(xn) Hirota lab 17/45 Multi-layer N-N learning Forward activation Calculating output error passing (or feeding) the signals through the network Mean squared error (MSE) Error backpropagation Hirota lab 18/45 Introduction • Modeling – System is unknown – Measured input-output data available • Neuro-fuzzy systems – Fuzzy rules: describe the behavior (structure) – Neural Networks: adjust control parameters and improvement of the system (refinement) Hirota lab 19/45 Introduction • Modeling process Hybrid learning algorithm Merged-based fuzzy clustering In-Out data Self-Constructing Rule Generation Fuzzy rules Structure Identification Neural Network Result Final Fuzzy Rules Parameter identification Hirota lab 20/45 Controller Structure • Fuzzy rules from Data – If x is Mx1 Then y is Mz1 – If x is Mx2 Then y is Mz2 X – 2 inputs – If x is Mx2 and y is My1 Then y is Mz2 Mx2 Mx1 Mz1 Mz2 z y Hirota lab 21/45 [38],[42] Our approach • Gaussian function for membership functions • J final clusters of data • TSK fuzzy rule: – If x1 is µ1j(x1) AND x2 is µ2j(x2) AND … AND xn is µnj(xn) THEN y = b0j +b1jx1+..+bnjxn b0j > m0j Deviation vector Mean vector (m) b1j..bnj > temporarily 0 (can not be deduced) Hirota lab 22/45 Clustering • To define input membership functions regions – Data partitioning • Sensitive to input order • Might be redundant – Clusters merge • Merging similar clusters Hirota lab •n inputs> x1…xn, one output •Cj > fuzzy cluster 23/45 Data partitioning • N: number of training patterns • tv: pattern (1≤v≤N) • Sj: Size of cluster Cj • comb > operator to combine Cj and tv Hirota lab 24/45 Combination • Combining: Changing Gaussian functions of cluster Initial deviations User-defined constant Hirota lab 25/45 Partitioning process • tv> New training instance • Calculate Ij(p v) > compare with ρ (threshold) • Calculate O'j(y)=comb_y(Oj(y),qv) and compare ơ'oj with η (threshold) •If both failed > new cluster k •If not> add to the most appropriate Cluster Using the combination method •Largest input-similarity •With minimum output-variance Hirota lab 26/45 Cluster merge • Partitions > input order dependant • Merging > creating efficient clusters Hirota lab 27/45 Merging conditions • Calculating input and output similarity measure rIij , rOij for every two clusters (Ci,Cj) • If rIij ≥ ρ , rOij ≥ ε (threshold) Then put them into the same candidate class X Hirota lab 28/45 Merging process • K Clusters combination: • Clusters are combined if ơ'oj ≤ η Hirota lab 29/45 Merging process (full procedure) Merging clusters Increasing ρ,ε Merged clusters To (1+θ)* (ρ , ε ) No clusters Input-output similarity measurement Every candidate class has only one cluster Finished Hirota lab 30/45 Rule Refinement Final clusters >> Final rules (TSK type) • C1,C2,…Cj >> R={R1,R2,..Rj} If x1 IS µ1j(x1) AND If x2 IS µ2j(x2) AND … AND If xn IS µnj(xn) THEN yj IS fj(x)=b0j+b1jx1+…+bnjxn • µij(x1) = exp[-( (xi-mij)/ơij)2] • b0j = m0j, b1j..bnj=0 (will be learned later) Hirota lab 31/45 TSK-type fuzzy rule - Reminding fuzzy inputs but crisp outputs (constant or a function) If X is X1 and Y is Y1 then z = f1(x,y) If X is X2 and Y is Y2 then z = f2(x,y) ... If X is Xn and Y is Yn then z = fn(x,y) Defuzzification: (wi) being the degree of matching (product of µ(xi)) Y= (w1*f1(x,y) + w2*f2(x,y) + … + wn*fn(x,y)) / (w1 + w2 + … + wn) Hirota lab 32/45 Rules’ Calculations(1) Rule j strength αj(x )=µ1j(x1)*µ2j(x2)*…*µnj(xn) =Ij(x ) fj(x)=b0j+b1jx1+…+bnjxn Final output y: •µij(x1) = exp[-( (xi-mij)/ ij)2] •b0j = m0j, b1j..bnj=0 Hirota lab 33/45 Rules’ Calculations(2) µnj(xn) > αj(x )=∏µij > dj(x ) > sj(x ) > y Hirota lab 34/45 Network structure 2 Layer 1: 2: µ 3: 4: 5: αjnj d s y (x (x = (x ∑ )=exp[-( ) )=∏µ ) s = = (x d α (x (x ) (x )*f )/∑α -m (x (x )/ơ ) ) ) j n j jjij i j jij ij ] µ11 x1 µi1 . . µn1 xi (mij,ơij) x ∏ µ1j µij . . . µ1J xn µiJ ∏ N αj(x ) N s1 dj(x ) sj y ∑ µnj µnJ ∏ N Hirota lab sJ 35/45 Network refinement µ11 Hybrid Learning Algorithm Combination of: x1 µi1 . . µn1 xi • Recursive SVD-based leastsquare estimator > refine bij • Gradient decent method > refine (mij,ơij) µij µ1J xn µiJ N ∏ µ1j . . . Gradient decent algorithm Hirota lab (mij,ơij) x ∏ αj(x) N s1 dj(x) sj µnj ∏ N sJ µnJ SVDbased LSE 36/45 ∑ y Singular Value Decomposition(1) N training patterns: tV= trace(D)=∑n1di i ||D|| = (trace(DTD))1/2 Hirota lab 37/45 Singular Value Decomposition(2) Suppose aij as constant > Finding X*: the optimal solution to minimize E DTD=I<>A1=AT SVD ensures: A=U∑VT • U: NхN orthonormal matrix • V: JхJ orthonormal matrix • ∑: NхJ diagonal matrix with ei: eigenvalues of ATA Hirota lab 38/45 SVD calculations Calculating X*: (X the only variable) U: orthonormal E(X)=||Q-U∑VTX|| E(X)=||UTQ-∑VTX|| > VTX=Y TQ= Q’ ∑= ∑’ U 0 Q’’ E(X)= Q’ Q’’ - ∑’ Y 0 = Q’- ∑’Y Q’’ Hirota lab minimized Q’-∑’Y*=0 X*=VY* 39/45 Gradient Decent Method To refine (mij,ơij) Hirota lab 40/45 Summary of process µ11 x1 µi1 . . µn1 xi (mij,ơij) x µ1j µij . . . µ1J xn µiJ N ∏ ∏ αj(x) N s1 dj(x) sj ∑ y µnj ∏ N sJ µnJ Mergedbased fuzzy clustering Hybrid learning algorithm Self-Constructing Fuzzy rules Rule Generation Neural Network Hirota lab Final Result Fuzzy Rules 41/45 Experimental results Modeling the function Refinement through Neural Network µ11 x1 7 Rules: 4 clusters >4 7 Rules s ∏ N Final Fuzzy µ 1 i1 µn1 . . xi x (mij,ơij) µ1j µij . . . µ1J xn µiJ ∏ αj(x) N dj(x) sj ∑ µnj ∏ N sJ µnJ Hirota lab 42/45 y Comparison of methods Modeling function: Yen’s System Juang’s System Hirota lab This paper’s System 43/45 Conclusion Using differential equations is useless for many control problems Advantages of this approach • flexibility • Simplicity Disadvantages • No explanation on methods used • Data-set needed Hirota lab 44/45 Thank you for listening Any questions? Hirota lab 45/45 Thank you for listening Any questions? Hirota lab 46/45 µ11 x1 µi1 . . µn1 xi (mij,ơij) x ∏ µ1j µij . . . µ1J xn µiJ ∏ N αj(x ) N s1 dj(x ) sj ∑ µnj µnJ ∏ N Hirota lab sJ 47/45