* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Nonnegative Matrix Factorization with Sparseness Constraints
Eigenvalues and eigenvectors wikipedia , lookup
Genetic algorithm wikipedia , lookup
Exact cover wikipedia , lookup
Algorithm characterizations wikipedia , lookup
Corecursion wikipedia , lookup
Laplace–Runge–Lenz vector wikipedia , lookup
Inverse problem wikipedia , lookup
Factorization of polynomials over finite fields wikipedia , lookup
Operational transformation wikipedia , lookup
Smith–Waterman algorithm wikipedia , lookup
Pattern recognition wikipedia , lookup
Drift plus penalty wikipedia , lookup
Least squares wikipedia , lookup
K-nearest neighbors algorithm wikipedia , lookup
Mathematical optimization wikipedia , lookup
Multidimensional empirical mode decomposition wikipedia , lookup
Nonnegative Matrix Factorization with Sparseness Constraints S. Race MA591R Introduction to NMF Factor A = WH A – matrix of data m non-negative scalar variables n measurements form the columns of A W – m x r matrix of “basis vectors” H – r x n coefficient matrix Describes how strongly each building block is present in measurement vectors Introduction to NMF con’t Purpose: “parts-based” representation of the data Data compression Noise reduction Examples: Term – Document Matrix Image processing Any data composed of hidden parts Introduction to NMF con’t Optimize accuracy of solution: min || A-WH ||F where W,H ≥ 0 We can drop nonnegative constraints min || A-(W.W)(H.H) || Many options for objective function Many options for algorithm W,H will depend on initial choices Convergence is not always guaranteed Common Algorithms Alternating Least Squares Paatero 1994 Multiplicative Update Rules Lee-Seung 2000 Nature Used by Hoyer Gradient Descent Hoyer 2004 Berry-Plemmons 2004 Why is sparsity important? Nature of some data Text-mining Disease patterns Better Interpretation of Results Storage concerns Non-negative Sparse Coding I Proposed by Patrik Hoyer in 2002 Add a penalty function to the objective to encourage sparseness OBJ: Parameter λ controls trade-off between accuracy and sparseness f is strictly increasing: f=Σ Hij works Sparse Objective Function The objective can always be decreased by scaling W up, H down Set W= cW and H=(1/c)H Thus, alone the objective will simply yield the NMF solution Constraint on the scale of H or W is needed Fix norm of columns of W or rows of H Non-negative Sparse Coding I Pros Simple, efficient Guaranteed to reach global minimum using multiplicative update rule Cons Sparseness controlled implicitly: Optimal λ found by trial and error Sparseness only constrained for H NMF with sparseness constraints II First need some way to define the sparseness of a vector A vector with one nonzero entry is maximally sparse A multiple of the vector of all ones, e, is minimally sparse CBS Inequality How can we combine these ideas? Hoyer’s Sparseness Parameter sparseness(x)= where n is the dimensionality of x This measure indicates that we can control a vector’s sparseness by manipulating its L1 and L2 norms Picture of Sparsity function for vectors w/ n=2 Implementing Sparseness Constraints Now that we have an explicit measure of sparseness, how can we incorporate it into the algorithm? Hoyer: at each step, project each column of a matrix onto the nearest vector of desired sparseness. Hoyer’s Projection Algorithm Problem: Given any vector, x, find the closest (in the Euclidean sense) nonnegative vector s with a given L1 norm and a given L2 norm We can easily solve this problem in the 3 dimensional case and extend the result. Hoyer’s Projection Algorithm Set si=xi + (L1-Σxi)/n for all i Set Z={} Iterate Set mi=L1/(n-size(Z)) if i in Z, 0 otherwise Set s=m+β(s-m) where β≥0 solves quadratic If s, non-negative we’re finished Set Z=Z U {i : si <0} Set si=0 for all i in Z Calculate c=(Σsi – L1)/(n-size(Z)) Set si=si-c for all i not in Z The Algorithm in words Project x onto hyperplane Σsi=L1 Within this space, move radially outward from center of joint constraint hypersphere toward point If result non-negative, destination reached Else, set negative values to zero and project to new point in similar fashion NMF with sparseness constraints Step 1: Initialize W, H to random positive matrices Step 2: If constraints apply to W or H or both, project each column or row respectively to have unchanged L2 norm and desired L1 norm NMF w/ Sparseness Algorithm Step 3: Iterate If sparseness constraints on W apply, Set W=W-μw(WH-A)HT Project columns of W as in step 2 Else, take standard multiplicative step If sparseness constraints on H apply Set H=H- μHWT(WH-A) Project rows of H as in step 2 Else, take standard multiplicative step Advantages of New Method Sparseness controlled explicitly with a parameter that is easily interpretted Sparseness of W, H or both can be constrained Number of iterations required grows very slowly with the dimensionality of the problem Dotted Lines Represent Min and Max Iterations Solid Line shows average number required An Example from Hoyer’s Work Text Mining Results Text to Matrix Generator Dimitrios Zeimpekis and E. Gallopoulos University of Patras http://scgroup.hpclab.ceid.upatras.gr/sc group/Projects/TMG/ NMF with sparseness constraints from Hoyer’s web page