* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Dia 1 - van der Veld
Basis (linear algebra) wikipedia , lookup
Elementary algebra wikipedia , lookup
Cartesian tensor wikipedia , lookup
Capelli's identity wikipedia , lookup
Signal-flow graph wikipedia , lookup
Jordan normal form wikipedia , lookup
Eigenvalues and eigenvectors wikipedia , lookup
Four-vector wikipedia , lookup
System of polynomial equations wikipedia , lookup
Singular-value decomposition wikipedia , lookup
Matrix (mathematics) wikipedia , lookup
Perron–Frobenius theorem wikipedia , lookup
Non-negative matrix factorization wikipedia , lookup
History of algebra wikipedia , lookup
Linear algebra wikipedia , lookup
Orthogonal matrix wikipedia , lookup
Matrix calculus wikipedia , lookup
Matrix multiplication wikipedia , lookup
Cayley–Hamilton theorem wikipedia , lookup
Multivariate Statistics Matrix Algebra II W. M. van der Veld University of Amsterdam Overview • The determinant of a matrix • The matrix inverse • System of equations The determinant of a matrix • The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix. • It has very important mathematical properties, but it is very difficult to provide a substantive definition. • The determinant is necessary to compute the inverse of a matrix (A-1). – The inverse of a matrix is needed for solving systems of linear equations; multivariate statistics often comes down to this. – When the determinant is zero, there exists no solution to a system of linear equations. • Let’s see how the value of the determinant is found. The determinant of a matrix • How to do it? The most simple case, a 2 by 2 matrix . • Det(A)=|A|=? Cofactors 1 3 A 0 4 1 3 Det ( A ) A 1* 4 3 * 0 4 0 4 The determinant of a matrix • One step further, a 3 by 3 matrix. • Det(A)=|A|=? 1 2 2 A 5 1 3 8 0 4 1 3 5 3 A 1* 2 * 0 4 8 4 Cofactor 5 1 2* 8 0 1(1 4 3 0 ) 2(5 4 3 8 ) 2(5 0 1 8 ) (4 0) ( 40 48) (0 16) 12 The determinant of a matrix • You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors. The determinant of a matrix • Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix. • Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the ith row and the jth column, the resulting minor will be square and its determinant will be symbolized |Mij|. This determinant is called a cofactor (cij) if we give it a sign equal to (-1)i+j, so that: cij = (-1)i+j |Mij|. Using this notation we can write a formula for the expansion of a determinant of order n: n A ai1ci1 ai 2ci 2 aincin aigcig In this version the determinant is g 1 expanded according to it’s ith row. The determinant of a matrix • The following rules are important for determinants, and can help you sometimes to simplify calculations: – The determinant of A has the same value as the determinant of A’. – The value of the determinant changes sign if one row (column) is interchanged with another row (column). – If a determinant has two equal rows (columns), its value is zero. – If a determinant has two rows (columns) with proportional elements, its value is zero. – If all elements in a row (column) are multiplied by a constant, the value of the determinant is multiplied by that constant. – If a determinant has a row (column) in which all elements are zero, the value of the determinant is zero. – The value of the determinant remains unchanged if one row (column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged. The determinant of a matrix • What is the determinant of: 2 4 8 A 1 2 3 3 2 1 1 0 B 0 1 1 4 5 10 C D 3 12 12 24 The matrix inverse • Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A-1. A-1, if exists, can be found as follows. • Let C be the matrix of cofactors of A (i.e., cij is the cofactor obtained from the minor |Mij|); then A 1 C / A • Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero. The matrix inverse • Illustration that AA-1 = A-1A = I. 1 4 A 3 2 .2 A .3 1 .4 - .1 1 4 0.2 0.4 AA 3 2 0.3 0.1 1 1( 0.2) 4(0.3) 1(0.4) 4( 0.1) AA 3( 0.2) 2(0.3) 3(0.4) 2( 0.1) 1 ( 0.2 1.2) (0.4 0.4) 1 0 AA I ( 0.6 0.6) (1.2 0.2) 0 1 1 The matrix inverse • How did I get A-1? 1 4 A 3 2 A A Compute determinant 1 4 2 12 10 3 2 2 C 2 4 C 3 1 Now Compute C 2 3 C 2 3 C 4 Calculate A-1 2 4 3 1 0.2 0.4 A 1 C / A 0.3 0.1 10 2 3 C 4 1 C transpose => C’ The matrix inverse • Another way to calculate A-1. This way introduces you to solving systems of equations. 1 4 A 3 2 1 1 AA 3 1 4 x11 x12 1 0 AA I x x 3 2 0 1 21 22 4 x11 x12 1x11 4 x21 1x12 4 x22 1 0 2 x21 x22 3x11 2 x21 3x12 2 x22 0 1 1 1x11 4 x21 1 x11 1 0.2 5 3x11 2 x21 0 2; 5x11 0 x21 1 1x11 4 x21 1 3; 3x11 2 x21 0 0 x11 10 x21 3 x21 3 10 x12 2 1x12 4 x22 3x12 2 x22 5 0.4 5x12 0 x22 1x12 4 x22 3x12 2 x22 0.3 x22 1 .2 A1 .3 1x11 4 x21 3x11 2 x21 1x12 4 x22 3x12 2 x22 10 .4 - .1 0.1 0 x12 10 x22 1 0 0 1 0 1 2; 2 0 3; 1 1 The matrix inverse • Rules for algebra with inverse matrices: – AA-1 = A-1A = I – (AB)-1 = B-1A-1 – (ABC)-1 = C-1B-1A-1 • Proof that (AB)-1 = B-1A-1. System of equations • In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods. • Here we will discuss how to solve systems of such linear equations. System of equations • Illustration. Suppose we have the following set of equations: -3=1x1+4x2 1=3x1+2x2 • The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied. • This can be solved in a simple way (old style). 3 1x1 4 x2 1 3x1 2 x2 5 5 x1 * 2, add x1 1 etc. x2 ? • The solution is basically the intersection of the lines represented by the equation. • You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra. System of equations • Solution for m equations with n unknowns: m=n. a12 a1n x1 k1 a11 k2 a21 a22 a2 n x2 k Ax k a a a x m2 mn n n m1 • What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x). • Matrix division is defined as multiplication by the inverse, so: if k Ax A 1k A 1Ax A 1k Ix 1 A kx System of equations • Example. Suppose we have the following set of equations: -3=1x1+4x2 1=3x1+2x2 • We already solved this one, resulting in x1=1 and x2=-1. • The set of equations can be written as a matrix operation. 3 1 1 3 4 x1 2 x2 k Ax System of equations • Thus, we have to find the inverse of: A => A-1 = C’/|A| 1 A 3 4 2 3 22 3 C 4 1 • We have to take the transpose of C 4 2 C 3 1 System of equations • We have to divide by |A|. 1 A 3 4 (1 2) (4 3) 10 2 • Thus the inverse matrix is. A 1 1 2 10 3 4 1 System of equations • Thus a solution for: -3=1x1+4x2 1=3x1+2x2 is found via k Ax 3 1 1 3 A 1k x 1 2 10 3 4 3 x1 1 1 x2 1 (2 3) ( 4 1) x1 10 ( 3 3) (1 1) x2 1 10 1 x1 10 10 1 x2 4 x1 2 x2 System of equations Exercise, solve: x1 + 2x2 = 0 3x1 + 7x2 = 1 1 2 A 3 7 A-1Ax = Ix = x = A-1k 0 k 1 7 2 0 0 2 2 A k x 3 1 1 0 1 1 1 • So if Ax = k solve via x = A-1k. • .... But it is not always so simple … System of equations • Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution. • But consider the following cases. 8 2 16 4 1 x1 (Row 2 = 2 x Row 1) 2 x2 1 1 2 1 2 3 1 2 (Row 3 = Row 1 + Row 2) 3 4 1 4 2 4 6 6 5 (Column 3 = Column 1 + Column 2), etc. 10 System of equations • These situations are called linear dependence: – Given vectors: x1, x2,…, xn-1 – Another vector xn is linearly dependent if there exists constants α1, α2,…, αn-1 such that: xn= α1x1+α2x2+ …+αn-1xn-1 • Otherwise the vector xn is linearly independent. • In case of linear dependence; |A|= 0. • And then the inverse is not defined: A-1=C’/|A|. • And when the inverse is not defined we cannot find a solution via: A-1k=x. System of equations • Generally a unique solution exists only if m=n, and |A|≠0 • When are there ‘problems’? – If m<n there are many solutions, the problem is underdetermined. 8x1+10x2+14x3=9 4x1+12x2+16x3=10 – if m>n there are no solutions, the problem is overdetermined. 8x1+10x2=9 4x1+12x2=10 4x1+10x2=2 System of equations • Using the idea of linear dependency, the rank of a matrix can be introduced. • rank(A) = number of linearly independent rows or columns. • Given an mxn matrix, with m ≥ n, then if – |A| ≠ 0 rank(A) = n full rank, solvable – |A| = 0 rank(A) < n rank deficient • We will get back to the issue of rank. Overdetermined Systems • Find Ax “closest” to k • Least-squares distance measure Φ ( Ax k )( Ax k ) • Minimization problem: • Normal equations: (A’A)x = A’k • Solution: x = (A’A)-1A’k – A’A must be nonsingular; i.e. |A’A|≠0 – (A’A)-1A’ is called the left inverse matrix Underdetermined Systems • Find “smallest” x that satisfies equations • Minimum norm objective 1 Φ xx 2 • Constrained minimization problem: • Solution: x = A’(AA’)-1k – AA’ must be nonsingular – A’(AA’)-1 is called the right inverse min Φ x subject to : Ax k