Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Table of Contents from A. P. Sakis Meliopoulos Power System Modeling, Analysis and Control Appendix B ____________________________________________________________ 1 Linear Programming ____________________________________________________ 1 B.1 Introduction ___________________________________________________________ 1 B.2 Forms of Linear Programming Problems ___________________________________ 1 B.3 Conversion of Optimization Problems into the Standard Form _________________ 2 B.4 Definitions _____________________________________________________________ 6 B.5 The Fundamental Theorem of Linear Programming __________________________ 7 B.6 Geometric Concepts _____________________________________________________ 9 B.7 The Simplex method____________________________________________________ 13 B.7.1 Feasibility Condition ________________________________________________________ 13 B.7.2 Optimality Condition ________________________________________________________ 14 B.8 Convergence __________________________________________________________ 19 B.9 Computational Considerations____________________________________________ 22 B.9.1 Pivoting __________________________________________________________________ 22 B.9.2 LU Decomposition and Sparcity Techniques _____________________________________ 24 B.9.3 Upper Bounds _____________________________________________________________ 24 B.10 Existence of Solution __________________________________________________ 24 B.11 The Dual Simplex Method ______________________________________________ 27 B.12 Duality – The Dual Linear Program______________________________________ 40 B.12.1 Complementary Slackness __________________________________________________ 40 B.12.2 Sensitivity _______________________________________________________________ 41 B.13 Interior Point Methods_________________________________________________ 41 B.14 Summary and Discussion_______________________________________________ 44 B.15 Problems_____________________________________________________________ 45 Appendix B Linear Programming B.1 Introduction This appendix presents the theory and solution techniques to optimization problems of the linear programming variety. The linear programming problem is studied in its standard form. Two classes of solution methods are introduced: (a) those based on the simplex method, and (b) those based on the interior point method. Various extensions are discussed with emphasis to those applicable to large scale systems. B.2 Forms of Linear Programming Problems A linear programming problem is a mathematical optimization problem in which the objective function is linear with respect to the unknowns and the constraints consist of linear equalities and inequalities. The general form is: Minimize cT x (or maximize - cT x) = b1 Subject to: Ae x ≤ b2 Ai x |Aa x| < b3 where Ae, Ai, Aa are constant matrices of proper dimensions, b1, b2, b3, and c are constant vectors of proper dimensions. A linear program of the previous form can be cast into the standard or canonical form with a series of transformations. The standard and canonical forms of a linear program are defined in Table B.1. Subsequent paragraphs define the necessary transformations to cast a given optimization problem into the standard form. Because solution techniques will be developed in the framework of the standard form, transformations to the standard form will be emphasized. B-1 Table B.1. Forms of a Linear Program Canonical Form Standard Form n Min n z = ∑cjx j Min j =1 j =1 n Subject to : z = ∑cjx j ∑ a ij x j < bi , n xj ≥0 Subject to : j =1 i = 1, 2,..., m , ∑a ij x j = bi , xj ≥0 j =1 j = 1, 2,..., n i = 1, 2,..., m , j = 1, 2,..., n B.3 Conversion of Optimization Problems into the Standard Form Many optimization problems can be formulated as linear programs. For example, a nonlinear optimization problem can be transformed into a linear program by linearization. Many times, the formulation involves a number of transformations and conversions. This section presents the most commonly encountered conversion problems. Specifically, the transformation may involve one or more of the following most common cases: Canonical to Standard Form: canonical form: Min z = c T x Subject to : Ax ≤ b, Consider the following linear program in x≥0 Above problem is transformed into the standard form upon introduction of the slack variables s: Min z = c T x Subject to : Ax + s = b, x ≥ 0, s≥0 Free Variables: Assume that the optimization problem contains free variables, that is one or more variables, xi, may assume positive, zero or negative values. There are two ways to handle this case: Approach A. A free variable xi is substituted with two nonnegative variables: xi = ui - vi ui ≥ 0, vi ≥ 0 B-2 Upon substitution, the constraint Ax = b is transformed appropriately. Note that this approach results in an increased dimensionality of the problem. Approach B. In this approach, the free variables are eliminated. For example, consider any one equation from Ax = b which includes the free variable xi. Solve this equation for xi. Then substitute xi in all other equations and objective function. This approach results in a decreased dimensionality of the problem. Surplus variables: An inequality constraint of the form, aT x ≥ bi is converted into an equality constraint upon introduction of a non negative variable si: aT x - s i = b i , si ≥ 0 The variable si is called surplus variable. Limits on variables: Consider the optimization problem Minimize Subject to: cT x Ax = b _ h≤x≤h The transformation y = x - h, yields: Minimize Subject to: cT (y + h) = cT h + cT y A(y + h)_ = b 0≤y≤h-h Above problem is equivalent to the following linear programming problem in standard form: Minimize Subject to: cT y Ay = b' y+s=d y≥0 s≥0 where b' = _b - Ah d=h-h s are slack/surplus variables B-3 Absolute Values of Functions: Many times optimization problems contain absolute values of functions or variables. Consider for example a constraint with an absolute value as follows: | ai1 x1 + ai2 x2 + ...... + ain xn | ≤ bi This constraint is equivalent to the following two constraints: n ∑ a ij x j ≤ bi j= 1 n - ∑ a ij x j ≤ bi j= 1 Absolute Values of Variables Another usual case is the appearance of absolute values in specific free variables. In this case, it can be proven that the problem does not alter if the following substitution is made: | xi | = ui + vi where xi = ui - vi, ui, vi ≥ 0 Can you prove it ? Transformations of optimization problems of the linear programming variety into the standard form will be illustrated with several examples. Example B.1: Consider the problem Minimize Subject to: x0 = 2x1 + 3x2 + 5x3 x1 + x2 - x3 ≥ -5 -6x1 + 7x2 - 9x3 = 15 | 19x1 - 7x2 + 5x3 | ≤ 15 x1 ≥ 0, x2 ≥ 0, x3 free Transform above problem into a linear program in standard form. Solution: Since the variable x3 is free, it will be eliminated from the problem. For example, the second equation yields: x3 = − 6 7 15 x1 + x2 9 9 9 Upon substitution, the problem is transformed into: Minimize x0 = − 75 12 62 x1 + x2 9 9 9 B-4 Subject to: − 15 2 60 x1 x2 ≤ 9 9 9 141 x1 - 28 x2 ≤ 210 - 141 x1 + 28 x2 ≤ 60 x1, x2 ≥ 0 Upon introduction of slack variables: Minimize Subject to: x0 = − 75 12 62 x1 + x2 9 9 9 -15 x1 -2 x2 + s1 = 60 141 x1 - 28 x2 + s2 = 210 - 141 x1 + 28 x2 + s3 = 60 x1, x2, s1, s2, s3 ≥ 0 Note in above that the constant term in the objective function may be omitted without altering the problem. Example B.2: Convert the following problem into a linear program in standard form. Minimize Subject to: |x|+|y|+|z| x+y≤1 2x + z = 3 x, y, z are free Solution: Since the x, y, z variables are free, define: x = x1 - x2 y = y1 - y2 z = z1 - z 2 x1, x2 ≥ 0 y1, y2 ≥ 0 z1, z2 ≥ 0 Then, the linear problem becomes Minimize Subject to: x1 + x2 + y1 + y2 + z1 + z2 x1 - x2 + y1 - y2 + s = 1 2x1 - 2x2 + z1 - z2 = 3 x1, x2, y1, y2, z1, z2, s ≥ 0 In summary, any optimization problem with linear objective and constraints can be transformed into the standard form. Thus, it is sufficient to study the standard form only. B-5 B.4 Definitions The discussion of Section B.3 suggests that it suffices to study the solution of the linear program in standard form. For this purpose, a number of definitions will be introduced. For conformity, all definitions will refer to the following linear program in standard form: Minimize Subject to: cT x Ax = b x≥0 (B.1) (B.2) (B.3) where: b is an m x 1 vector x is an n x 1 vector of unknown variables A is an m x n matrix c is an n x 1 vector Definition:: Basic Solution. A basic solution is a vector x which has at least n-m zero entries and which satisfies the constraint Ax = b. A basic solution can be constructed as follows: Select (n - m) unknowns from the vector x and set them equal to zero. These variables are called nonbasic variables and are denoted with the vector xN. Then Ax = B xB = b Where B is a square matrix. The matrix B is a submatrix of A and shall be called the basis matrix. If B has rank m, the solution for xB is given from xB = B-1 b. The solution xB = B −1b xN = 0 (B.4a) (B.4b) is called a basic solution. The variables included in x B are called basic variables. Definition: Degenerate Basic Solution. If one or more components of a basic solution, x B , are equal to zero, then this solution is called degenerate basic solution. Definition:: Basic Feasible Solution. Consider a basic solution. If x B ≥ 0 , then the solution is called basic feasible solution. B-6 Definition: Degenerate Basic Feasible Solution. If one or more components of a basic feasible solution, x B , are equal to zero, then this solution is called degenerate basic feasible solution. Example B.3: Consider the linear program: Minimize Subject to: x0 = -3x1 - 5x2 x1 + 2x2 + x3 = 8 3x1 + 2x2 + x4 = 18 x1, x2, x3, x4 ≥ 0 Determine all basic solution and all basic feasible solutions. Solution: The problem has four variables and two equations. There are six basic solutions (setting equal to zero two variables at a time out of fourand solving for the other two variables). The six basic solutions are shown in the table below. Based on the numerical values, the basic solutions are properly characterized in the table. Solution Number 1 2 3 4 5 6 x1 x2 x3 x4 Feasible Degenerate 0 0 0 8 6 5 0 4 9 0 0 1.5 8 0 -10 0 2 0 18 10 0 -6 0 0 Yes Yes No No Yes Yes No No No No No No B.5 The Fundamental Theorem of Linear Programming The determination of the optimal solution of a linear program is facilitated with the following theorem. Theorem: Fundamental Theorem of Linear Programming. Given a linear program in standard form: Minimize Subject to: x0 = cT x Ax = b x≥0 where: x is an n x 1 vector of unknown variables B-7 b is an m x 1 vector of constants n≥m and rank of A = m. The fundamental theorem states that: (a) If there is a feasible solution, there is a basic feasible solution. (b) If there is an optimal feasible solution, there is an optimal basic feasible solution. Proof of (a): Let x = (x1, ..., xn ) be a feasible solution. The constraint Ax = b is written as follows: x1 a1 + x2 a2 + ... = b (B.5) Note that ai denotes the ith column of matrix A. If more than m variables, let it be p, are greater than zero, then we can find numbers yi, i = 1, 2,..., p, such that [since rank A ≤ m]: y1 a1 + ... + yp ap = 0 (B.6) Now multiply equation (B.6) by ε and subtract from (B.5): (x1 - ε y1)a1 + (x2-εy2)a2 + ... = 0 (B.7) Select ε such that at least one of xi - εyi becomes zero while all others remain nonnegative. In this case, the number p will be decreased by at least 1. This process may be repeated as long as equation (B.6) can be written with nonzero yi values, that is, as long as p > m. That is, as long as p > m, the number of nonzero variables can be decreased by one. Thus, eventually p = m. Proof of (b): Let x = [x1 x2 ... xn ] be an optimal feasible solution. Then: Case 1: If p (> m) variables x1, x2 ..., xn are nonzero, the optimal feasible solution is also an optimal basic feasible solution. The theorem holds in this case. Case 2: If p (> m) variables x1, x2 ..., xn are nonzero, then without loss of generality, we can assume that and xT = [x1 x2 ... xp 0 0 ... 0] p i ∑ xi a = b i =1 B-8 Also nonzero numbers y1, ..., yp can be found such that p (B.8) i ∑yi a = b i =1 Multiply (B.9) by ε and subtract from (B.8) p (B.9) i ∑ ( x i − ε y i )a = b i =1 The last equation suggests that the vector x - εy is a solution. The value of objective function is cT x - ε cT y (ε positive or negative) Since cTx is the minimum value of the objective function and ε can be positive or negative, cTy must equal zero. Then select ε such that one entry is zeroed. This process can be repeated until only m entries are nonzero. In the next paragraph we will introduce a number of geometric concepts that will enable a geometric interpretation of the fundamental theorem. The importance of the fundamental theorem of linear programming is that it provides the necessary information to construct an algorithm for the computation of the optimal solution. In subsequent paragraphs we will introduce the simplex method which is based on this theorem. B.6 Geometric Concepts The discussion of the linear program can be facilitated with the introduction of a number of geometric concepts. This section points out the correspondence of linear program concepts and geometric concepts. Definition: Convex Set. A set K is called convex if and only if: Given any two vectors x1, x2 belonging to the set K and a number λ (0 ≤ λ ≤ 1), the vector x = λx1 + (1 - λ)x2 also belongs to the set K. Theorem: The constraints of a linear program B-9 Ax = b x≥0 ∆ define a convex set K which is denoted with K = { x ≥ 0, Ax = b}. Proof: Consider two vectors x1, x2 belonging to the set K. Thus, Ax1 = b , x1 ≥ 0, and Ax2 = b , x2 ≥ 0 Consider the vector x = λx1 + (1 - λ)x2 , 0 ≤ λ ≤ 1 Obviously x ≥ 0, and Ax = λAx1 + (1 - λ) Ax2 = λb + (1 - λ) b = b Thus the vector x belongs to the set K. Theorem B.2: The existence of a solution to a linear program is determined as follows. Consider the set K = {x ≥ 0, Ax = b}. a. If K = φ [empty set], then the linear program cT x Ax = b x≥0 Minimize Subject to: Has no solution. b. If K ≠ φ (non empty setr), then the linear program has a solution (bounded or unbounded). If the set K is bounded, the linear program solution is bounded. If the set K is unbounded, the linear program solution may be bounded or unbounded. Definition: Equivalence Between Extreme Points and Basic Solutions. An extreme point in a convex set is one which can not be expressed as a linear combination of any two other points in the set. Consider the convex set K ∆ K = { x ≥ 0, Ax = b} B-10 A point x (vector) is an extreme point if and only if x is a basic feasible solution to x ≥ 0, Ax = b. The correspondence between extreme points and basic feasible solutions can be utilized to conclude the following: (a) If K ≠ φ, then there is at least one extreme point. (b) If there is an optimal solution, it occurs on an extreme point of the set K. (c) The set K contains a finite number of extreme points (equal to the number of basic feasible solutions). (d) If K is bounded, a linear program with any objective function achieves its optimum at one of the extreme points. Definition: Degeneracy of Optimal Solution: If an optimal solution exists at more than one extreme point, then any convex combination of the extreme points is optimal. Let x1 , x2 be two optimal solutions, i.e., cTx1 = z0, cTx2 = z0. Then x = λx1 + (1 λ)x2 is optimal for 0 ≤ λ ≤ 1, because cTx = λcTx1 + (1 - λ)cTx2 = λz0 + (1 - λ)z0 = z0 Using the above geometric concepts, we can illustrate the fundametal theorem of linear programming as in Figure B.1. The shaded area denotes all the possible solutions of the constraining equations. The theorem states that if there is at least one point in the solution set, then there is a basic feasible solution. This property is illustrated in Figure B.1a. If there is an optimal feasible solution, then either the solution will be a basic feasible solution or such a point can be found. This property is illustrated in Figure B.1b. B-11 A: feasible solution B: basic feasible solution B A (a) A': optimal solution B': basic optimal solution B' A' (b) Figure B.1 Geometric Interpretation of the Fundamental Theorem. (a) Transition from a Feasible Solution to a Basic Feasible Solution (b) Transition from an Optimal Solution to a Basic Optimal Solution B-12 B.7 The Simplex method The simplex method is one of the solution methods (algorithms) for linear programs introduced in 1947. The method is based on the following simple idea. Assume that a basic feasible solution is known. Then find another basic feasible solution which improves the objective. If such solution cannot be found, the present basic feasible solution must be optimal. This basic idea can be implemented with the aid of two conditions: 1. Feasibility condition 2. Optimality condition. These two conditions provide the steering mechanism for going from one basic feasible solution to another with improved objective function. The conditions will be introduced in the sequel. B.7.1 Feasibility Condition Assume a basic feasible solution x1 is known. Thus: Ax1 = b Let A = [a1, a2, a3, ..., an]. Without loss of generality, it can be assumed that the variables 1, 2, ..., m are basic while variable m + 1, ..., n are nonbasic. Then: x1a1 + x2a2 + ... + xmam = b (B.10) Assume that the nonbasic variable xk (k > m) is to be brought into the basis. The vector ak of matrix A can be expressed as a linear combination of {a1 , a2 , ..., am } This is always possible since it has been assumed that equation (B.10) has a solution. Thus, the vectors a1, a2, ..., am are linearly independent. Therefore, (B. 11 ) a k = y 1k a 1 + y k2 a 2 + ... + y km a m Multiply equation (B.11) by ε and subtract from (B.10) to obtain ( x 1 - εy 1k )a 1 + ( x 2 - εy k2 )a 2 + ... + ( x m - εy km )a m + εa k = b B-13 Above equation states that the vector x(ε) = [x 1 - εy 1k , x 2 - εy k2 , ..., x m - εy km , 0, ..., ε , 0, ..., 0]T (B.12) is a solution of the constraints Ax = b. This solution is nonbasic since more than m variables are nonzero. It is given parametrically in terms of the parameter ε. By varying the parameter ε a new basic feasible solution can be found. For this purpose, one must find the smallest ε which makes one entry x i - εy ki zero. The smallest value ε0 with this property is x x ε 0 = min { ki , y ik > 0 } = λk over i yλ yi (B.13) What happens if all y ki < 0? In this case, the vector x(ε) is a feasible solution which may become as large as desired. Thus, in this case, the problem has an unbounded solution. In summary, a new basic feasible solution can be obtained from the vector, x(ε) by appropriate selection of the parameter ε. The vector x(ε) will be denoted with: ⎡x B ⎢ ⎢ ⎢ ⎢ ⎣ - εy k ⎤ ⎥ 0 ⎥ ⎥ ε ⎥ 0 ⎦ (B.14) where yk = B-1ak. (B.15) In above analysis, a variable k has been arbitrarily selected to become a basic variable. Depending on the selection of the variable k, the value of the objective function may deteriorate or improve. Obviously, the variable k must be selected in such a way as to improve the value of the objective function. This selection can be achieved with the optimality condition to be examined next. B.7.2 Optimality Condition Consider a linear program in standard form: Minimize Subject to: cT x Ax = b x≥0 B-14 Let x1 be a basic feasible solution of the problem. Without loss of generality, it can be assumed that the variables 1, 2, ..., m are basic and m + 1, ..., n are nonbasic. The vector c can be partitioned into: c T = [c TB : c rT ] The value of the objective function evaluated at x1 is cT x1 = c BT xB = z0 Similarly, the matrix A is partitioned into A = [B : Ar]. Now consider a general solution x of the problem: ⎡xb ⎤ x= ⎢ ⎥ ⎣ xr ⎦ (B.16) For arbitrary values of xr ≥ 0, the values of xb are computed from or Ax = Bxb + Ar xr = b xb = B-1 b - B-1 Ar xr xb = xB - B-1 Ar xr The value of the objective function evaluated at the general solution x is: ⎡x b ⎤ c T x = [c TB : c Tr ] ⎢ ⎥ = c TB x b + c Tr x r ⎣xr ⎦ T T = c B xB - c B B-1Arxr + c rT xr = z0 + ( c rT - c BT B-1Ar) xr In above expression, observe that in order to improve the objective function the quantity ( c rT - c BT B-1Ar) xr must be negative. Since xr can be arbitrarily selected, a procedure can be devised where only one variable is selected to enter the solution. To have best results, select the most negative component of the vector ( c rT - c BT B-1Ar). This will determine the entering variable k. The components of the vector ( c rT - c BT B-1Ar) are called the reduced cost coefficients. In summary, the optimality condition involves the following steps: Compute ( c rT - c BT B-1Ar). Select the most negative value. The index determines the entering variable k. If all components are nonnegative, optimum has been obtained. B-15 Example B.3: Solve the following linear program using the simplex method. Minimize Subject to: x0 = -3x1 - 5x2 x1 + 2x2 ≤ 8 3x1 + 2x2 ≤ 18 x1, x2 ≥ 0 Solution: Conversion of above problem in standard form yields: Minimize Subject to: x0 = -3x1 - 5x2 ⎡1 2 1 0⎤ ⎢3 2 0 1⎥ ⎣ ⎦ ⎡ x1 ⎤ ⎢x ⎥ 8 ⎢ 2⎥ = ⎡ ⎤ ⎢ ⎥ ⎢x3 ⎥ ⎣18⎦ ⎢ ⎥ ⎣x 4 ⎦ x1, x2, x3, x4 ≥ 0 First a basic feasible solution must be determined. By inspection, the solution x1 = x2 = 0, x3 = 8, x4 = 18 is a basic feasible solution. To cast the solution in the form (B.16) the following is defined: ⎡x3 ⎤ ⎡ y1 ⎤ ⎢x ⎥ ⎢y ⎥ ⎢ 2⎥ = ⎢ 4⎥ ⎢ x1 ⎥ ⎢ y3 ⎥ ⎢ ⎥ ⎢ ⎥ ⎣x 2 ⎦ ⎣y4 ⎦ Then the problem becomes: Minimize x0 = [0 0 -3 -5] y Subject to: ⎡1 0 1 2⎤ ⎡8⎤ ⎢0 1 3 2⎥ y = ⎢18⎥ ⎣ ⎦ ⎣ ⎦ y≥0 At this point, the first iteration of the simplex method can be performed. 1st iteration: The Optimality Condition is applied to determine the entering variable: B-16 Compute c rT - c BT B-1Ar = [-3 -5] - [0 0] B-1 Ar = [-3 -5] By inspection, the most negative value corresponds to the variable y4. Thus, y4 should be selected to be the entering variable (k = 4). The Feasibility Condition is applied to determine the new solution: ⎡2 ⎤ ak = a4 = ⎢ ⎥ ⎣2 ⎦ ⎡2⎤ yk = B-1 a4 = ⎢ ⎥ ⎣2⎦ The new solution in terms of the parameter ε is: ⎡ 8 - ε2 ⎤ ⎢18 - ε 2⎥ ⎥ y(ε) = ⎢ ⎢ 0 ⎥ ⎥ ⎢ ⎣ ε ⎦ where the ε value is to be selected from: ε = min (1, 2) y i0 , y ik > 0 y ik = min [4, 9] = 4 Set ε = 4 to obtain the new basic feasible solution. ⎡0⎤ ⎢10⎥ ynew = ⎢ ⎥ ⎢0⎥ ⎢ ⎥ ⎣4⎦ Thus, the leaving variable is y1. Note that the new solution is also a basic feasible solution. The objective function computed at the new solution is: x0 = [0 0 -3 -5] ynew = -20 B-17 2nd iteration: Again, the solution is cast in the form (B.16) with the following definition: ⎡x 4 ⎤ ⎡y2 ⎤ ⎡ z1 ⎤ ⎢x ⎥ ⎢y ⎥ ⎢z ⎥ ⎢ 2⎥ = ⎢ 4⎥ = ⎢ 2⎥ ⎢x3 ⎥ ⎢ y1 ⎥ ⎢z3 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ x1 ⎦ ⎣ y3 ⎦ ⎣z 4 ⎦ The problem is transformed into the following: Minimize x0 = [0 -5 0 -3] z Subject to: ⎡0 2 1 1⎤ ⎡8⎤ ⎢1 2 0 3⎥ z = ⎢18⎥ ⎣ ⎦ ⎣ ⎦ z≥0 The Optimality Condition is applied to determine the entering variable. −1 c - c B-1Ar T r T B ⎡0 2⎤ = [0 -3] - [0 -5] ⎢ ⎥ ⎣1 2 ⎦ ⎡ −1 1⎤ = [0 -3] - [0 -5] ⎢ ⎥ ⎣0.5 0⎦ = [0 -3] - [-2.5 -2.5] = [2.5 -0.5] ⎡1 1⎤ ⎢0 3⎥ ⎣ ⎦ ⎡1 1⎤ ⎢0 3⎥ ⎦ ⎣ Thus, the entering variable is z4 (k = 4). Application of the Feasibility Condition yields: ⎡1⎤ ak = a4 = ⎢ ⎥ ⎣3⎦ ⎡ 0.5⎤ y4 = B-1 a4 = ⎢ ⎥ ⎣2.0⎦ The new solution in terms of the parameter ε is: B-18 ⎡ 10 - ε (2) ⎤ ⎢4 - ε (0.5)⎥ ⎥ z(ε) = ⎢ ⎥ ⎢ 0 ⎥ ⎢ ε ⎦ ⎣ where ε = min y i0 , y ik > 0 = min(5, 8) = 5 y ik Set ε = 5 to obtain the new basic feasible solution: ⎡0⎤ ⎢15 . ⎥ ⎥ ⎢ znew = ⎢0⎥ ⎢ ⎥ ⎣5⎦ Thus, the leaving variable is z1 (= y2 = x4) The value of the objective function evaluated at the solution is: x0 = [0 -5 0 -3] znew = -22.5 Further application of the optimality condition reveals that above solution is the optimal. Thus, no more iterations are needed. Now, the solution in terms of the original variables x is: x1 = y3 = z4 = 5 x2 = y4 = z2 = 1.5 x3 = y1 = z3 = 0.0 x4 = y2 = z1 = 0.0 B.8 Convergence Consider a linear program problem in standard form: Minimize Subject to: cT x Ax = b x≥0 Assume that the problem has a solution. The simplex method will converge to the solution. The solution will occur at a basic feasible solution. The maximum B-19 n! number of basic feasible solutions of a linear program is m! (n-m)!. Thus, the simplex method will converge in a finite number of steps. The maximum n! number of steps will be m! (n-m)!. In general, the actual number of steps is much n! less than the maximum possible m! (n-m)!. In large system applications, the n! maximum possible number of iterations m! (n-m)! may be prohibitively large. For this reason, computational details are of tremendous importance in the simplex method. Some of the computational considerations will be discussed next. B-20 m equations in n unknowns Infinite Number of Solutions infinite number of solutions (feasible or infeasible) Set n-m variables to zero . number of solutions (=n ) m (basic solutions) . some feasible some infeasibl Finite Number of Solutions Basic feasible <( n ) solutions = N m (some may be degenerated) . Start from one basic feasible solution. Generate another one which improves value of objective function. Proceed in this fashion . Resulting final solution is calledoptimal basic feasible solution . Number of iterations <N Figure B.2. Evolution of the Simplex Method. B-21 B.9 Computational Considerations This section discusses a number of computational issues in the solution process of a linear program. These are: • • • • • Pivoting (Tableau Form) Two Phase Method Revised Simplex Method LU Decomposition/Sparcity Techniques Upper bounds These issues are discussed in the sequel. B.9.1 Pivoting Consider the constraints Ax = b and let [xB : 0] be a basic feasible solution that is xi ≥ 0, i = 1, 2, ..., m, xi = 0, i = m + 1, ..., n. Application of Gaussian elimination yields: x1 ... x2 ... .. ... xm + y1,m+1 xm+1 + y1,m+2 xm+2 + ... + y1,n xn = y10 + y2,m+1 xm+1 + y2,m+2 xm+2 + ... + y2,n xn = y20 ... ... ... ... ... ... ... ... ... ... ... ... + ym,m+1 xm+1 + ym,m+2 xm+2 + ... + ym,n xn = ym0 Above form provides the basic feasible solution by inspection: xi = yi0 xi = 0 i = 1, 2, ..., m i = m + 1, ..., n Above form can be cast into the following tableau form: x1 x2 M xi ... xm 0 1 0 ... 0 ... 0 0 0 1 ... 0 ... 0 0 0 0 ... 0 ... 0 ... ... ... ..1 ... ..0 ... ... ... ..0 ... ..1 rm+1 y1,m+1 y2,m+1 ... yi,m+1 ... ym,m+1 ... ... ... ... ... ... rk y1,k y2,k ... yi,k ... ym,k ... ... ... ... ... ... rn y1,n y2,n ... yi,n ... ym,n -z0 y10 y20 ... yi0 ... ym0 Assume that variable i has been selected to leave the basis and variable k has been selected to enter the basis. The new solution may be found with the following steps: B-22 1. Replace xi with xk 2. Divide ith row with yi,k 3. Multiply ith row by yj,k, j = 1, 2, ..., m, j ≠ i and subtract from jth row j = 1, 2, ..., m, j ≠ i. The new yj0 will provide the new basic feasible solution. Above procedure is called pivoting on yi,k. The tableau is usually augmented by the coefficients of the objective function as shown in the tableau. In this case, the row of these coefficients is treated the same way: the coefficients of the basic variables are always zero. The rest of them define the vector [ c rT - c BT B-1Ar]. In this case, the selection of the entering variable is done by inspection. The selection of the leaving variable, if the entering variable is k, is done as follows: The minimum number y i0 , yi,k > 0 determines the leaving variable λ. y ik Example B.4: Solve the linear program of example B.3 using the tableau form. Solution: The solution obtained in tableau form with the following sequence: The starting tableau is: x3 x4 -3 1 3 x1 -5 2 2 x2 0 1 0 x3 0 0 1 x4 0 8 18 In the first iteration, by inspection of the tablaue, the entering variable is x2. The leaving variable is computed to be x3. Pivoting on the element (1,2) of the matrix yields the following tableau: x2 x4 -0.5 0.5 2 x1 0 1 0 x2 2.5 0.5 -1 x3 0 0 1 x4 20 4 10 By inspection of the tableau, the entering variable must be x1. The leaving variable is computed to be x4. Pivoting on the element (2,1) of the matrix yields the following tableau: B-23 x2 x1 0 0 1 x1 0 1 0 x2 2.25 0.75 -0.5 x3 0.25 -.25 0.5 x4 22.5 1.5 5 The final solution is obtained by inspection of the tableau: x1 = 5, x2 = 1.5, x0 = -22.5 B.9.2 LU Decomposition and Sparcity Techniques In cases where the matrix A is highly sparse, it is expedient to exploit the sparcity of the matrix A and therefore matrix B in order to avoid storing zero elements and performing numerical operations with zero elements. These techniques, known as sparsity techniques, have been discussed in Appendix A. Note that the matrix B-1 may not be sparse even if B is sparse. In this case, Gaussian Elimination is employed to factor the matrix into the product of two matrices L and U: B = LU (B.22) where L is a lower triangular matrix and U is an upper triangular matrix. The solution AB-1 b is obtained by: (a) forward substitution to compute the vector y: y=L-1 b, and (b) back substitution to compute the solution vector x, x = U-1 y. The reduced cost coefficients are also computed using sparsity techniques. The pertinent equations are: crT − cBTU −1L−1 Ar Note that the reduced cost coefficients are computed with a series of forward and back substitutions. B.9.3 Upper Bounds To be added later. B.10 Existence of Solution B-24 The simplex method, as has been presented so far, is an algorithm which starts from a basic feasible solution and progressively moves to the optimal solution. Many times it is considerably difficult to identify a basic feasible solution. If the problem is infeasible, then obviously a basic feasible solution does not exist. Here we shall examine the question of existence of a basic feasible solution. If such a solution can be found then the problem has an optimal solution. The existence of a basic feasible solution is addressed by means of an auxiliary linear problem. This problem is defined with the aid of artificial variables as follows: Minimize ∑yi (B.17a) Ax + y = b x ≥ 0, y ≥ 0 (B.17b) (B.17c) i Subject to: It is easy to show that if the original problem has a feasible solution, then the optimal solution to the above problem will be a basic feasible solution of the original problem. The reader is encouraged to provide a formal proof of this statement. Conversely, if at the optimal solution of above problem, y ≠ 0, then the original problem has no solution. The definition and solution of above auxiliary problem is called Phase I of the simplex method. Once Phase I has been completed, the artificial variables y may be dropped. The simplex method can be initiated in the original problem with the found basic feasible solution (Phase II). Example B.5: Consider the linear program: Minimize Subject to: 3x1 + 4x2 + 5x3 x1 + x2 + x3 = 5 5x1 + x2 + 2x3 = 8 x1,2,3 ≥ 0 Compute a basic feasible solution using Phase I and then solve the problem. Solution: Phase I of this problem is defined as follows: B-25 Minimize y1 + y2 subject to: ⎡ x1 ⎤ ⎢x ⎥ 2 ⎡1 1 1 1 0⎤ ⎢ ⎥ ⎡5⎤ ⎢5 1 2 0 1⎥ ⎢ x 3 ⎥ = ⎢8⎥ ⎦⎢y ⎥ ⎣ ⎦ ⎣ ⎢ 1⎥ ⎢⎣ y 2 ⎥⎦ x1, x2, x3, y1, y2 ≥ 0 The solution in tableau form is: Tableau 1 y1 y2 0 1 5 0 1 1 0 1 2 1 1 0 1 0 1 0 5 8 y1 y2 -6 1 5 -2 1 1 -3 1 2 0 1 0 0 0 1 -13 5 8 y1 x1 0 0 1 -0.8 8 0.2 -0.6 0.6 0.4 0 1 0 1.2 -0.2 0.2 -3.4 3.4 1.6 x2 x1 0 0 1 0 1 0 0 0.75 0.25 1 1.25 -0.25 1 -0.25 0.25 0 4.25 0.75 Tableau 2 Tableau 3 Tableau 4 Comments: The solution from tableau to tableau proceeds as follows. In the first iteration the entering variable is x1, the leaving variable is y2. Therefore pivoting is performed on y2,1. In the second iteration the entering variable is x2, the leaving variable is y1. Therefore pivoting is performed on y1,2. Completion of Phase I yields the following basic feasible solution: x1 = .75 x2 = 4.25 x3 = 0 B-26 The Big M-Method: The two phases, I and II, can be integrated in one linear program problem as follows: Step 1. Artificial variables are introduced as in phase I. Step 2. The objective function is augmented with the artificial variables ∑ Mr i i where M is a large positive (negative in maximization problems) number. That is, the artificial variables are penalized heavily. This step guarantees that the artificial variables will not be in the optimal solution, if such a solution exists. Step 3. Solve the linear program problem. If the final solution is void of the variables y, then the original problem has an optimal solution. While the big M method combines the two phases into one linear program, it has the disadvantage that computations with small and large numbers must be performed. This fact may generate numerical stability problems. Example B.6: Consider the linear problem of example B.5. Formulate the problem as a linear program with a starting basic feasible solution in terms of artificial variables (big M method). Solution: becomes: Upon introduction of artificial variables r1 and r2, the problem Minimize Subject to: 3x1 + 3x2 + 5x3 + 106 r1 + 106 r2 x1 + x2 + x3 + r1 = 5 5x1 + x2 + 2x3 + r2 = 8 x1, x2, x3, r1, r2 ≥ 0 A starting basic feasible solution is: x1 = 0 x2 = 0 x3 = 0 r1 = 5 r2 = 8 B.11 The Dual Simplex Method B-27 The basic idea in the dual simplex method is to start from a basic solution which satisfies the optimality condition (all reduced cost coefficients nonnegative), but possibly violating feasibility (one or more variables negative) and to construct a new basic solution which again satisfies the optimality condition, but has moved closer to the feasible region. Obviously, when feasibility is satisfied, the optimal solution has been reached since the optimality condition is always satisfied. This method is suitable to a class of problems for which an optimal but infeasible starting solution can be determined easily. The method will be developed for a linear program in standard form: Min z = c T x Subject to : Ax = b, x≥0 Assume that a basic dual feasible solution is known. Then x B = AB−1 b, x r = 0, c rT − c BT AB−1 Ar ≥ 0 where the usual notation is applied. Note that if in addition to the above conditions: xB ≥ 0 then this solution is the optimal solution. If, however, one or more of the entries of x B is negative, it is necessary to design an algorithm which will generate a new basic dual feasible solution. The new solution should be selected in such a way that one of the negative variables returns to zero. The algorithm that performs this task is illustrated in Figure B.3. B-28 Assume a dual basic feasible solution x B , x r Is x B Yes >0? STOP No Select the most negative entry of vector x B : x Bj Variable j is the leaving variable A B ε 0 Compute c TB y i - c i = min i y ij { ; -1 i i y i=B a , y j <0 } i scans all nonbasic variables Assume minimum occurs for i = k Variable k is the entering variable C Leaving variable is j Entering variable is k Compute new solution Figure B.3 The Dual Simplex Method The algorithm of Figure B.3 can be developed in a number of different ways. The reader is referred to the extensive literature on the subject. Here the essentials of the algorithm will be proved. From block A of the algorithm, it is obvious that the algorithm moves closer to the feasible region since negative variables are selected to leave the basis. Now if it is proved that the selection of the entering variable in block B will result in a new basic solution which satisfies the optimality condition, the proof will be completed. This can B-29 be proven with simple matrix manipulations as follows: First, observe that for every basic variable, j, the reduced cost coefficient is zero: c j − c BT AB−1 a j = c j − c BT e j = 0 Then, let j be the leaving variable and k the entering variable. The new matrix of the basis will be AB' = AB − a j (e j ) + a k (e k ) T T (B.23) The following matrix identities hold: y k = AB−1 a k (A ) ' −1 B = EAB−1 ⎡ ⎢ E = ⎢e 1 ⎢ ⎢ ⎣ e2 − y 1k y kj M L 1 y kj M ⎤ ⎥ L em ⎥ ⎥ ⎥ ⎦ The above matrix identity can be proved by direct substitution. The new solution will be optimal if the reduced cost coefficients are nonnegative. c i − c BT AB−1 a i ≥ 0 By direct computation ⎡0⎤ ⎢0⎥ ⎢ ⎥ ⎢M ⎥ c'BT = cBT - cj eT + ck eT, e = ⎢ ⎥ ⎢1⎥ ← place of variable j ⎢M ⎥ ⎢ ⎥ ⎣0⎦ Thus: c'BT B'T ai = (cBT - cj eT + ck eT) E B-1 ai = cBT E yi - cj eT E yi + ck eT E yi B-30 Note that E yi = yi - y ij y kj β where ⎡ y1k ⎢ k ⎢ y2 ⎢ M β= ⎢ k ⎢yj ⎢ M ⎢ k ⎢⎣ y m ⎤ ⎥ ⎥ ⎥ ⎥ 1⎥ ⎥ ⎥ ⎥⎦ and eT E yi = yji / yjk Thus c'BT B'T ai = c BT y i =c y i T B = c TB y i - y ij y kj y ij y kj y ij y kj c BT β - c j T B k y ij y kj y ij c y + cj y kj + ck - cj y ij y kj y ij y kj + ck [c BT y k - c k ] The new reduced coefficient for variable i is rcc'i = ci - c'BT = rcci + B'T y i j y k j ai = ci - c y + T B i y ij y k j [c BT y k - c k ] [c BT y k - c k ] Now to guarantee optimality rcc'i = rcci + y ij y k j [c BT y k - c k ] ≥ 0 for every i Note that the entering variable k has been selected in such a way that yjk < 0 and since the previous solution was a basic dual feasible solution B-31 y ij y kj rcci ≥ 0 for every i and cBT yk - ck ≤ 0. Thus, two cases must be examined. yji ≥ 0 Case 1: In this case always rcci' ≥ 0 and further discussion is unnecessary. Case 2: yji < 0 In this case, since k is the index which minimizes { c BT y i - c i ; yi = B-1 ai, yji < 0 } i yj it follows that c BT y i - c i c BT ≥ y ij y ij T i cB y − ci ≤ k yj y ij c i - c BT y i + k yj or y k - ck y kj ( c BT y k − c k ) ( c TB y k − c k ) ≥ 0 rcc'i ≥ 0 Hence, the new reduced cost coefficients will be nonnegative. This completes the proof. The algorithm of Figure B.3 will be demonstrated with an example. Example B.8: Solve the following problem via the dual simplex method: Minimize Subject to x1 + x2 + 3x3 -.75x1+ .75x2 - .625x3 + x4 = -.05 .25x1 - .25x2 - .625x3 + x5 = -.10 B-32 x1, x2, x3, x4, x5 ≥ 0 Solution: A starting basic dual feasible solution is: xB ⎡ x1 ⎤ x r = ⎢⎢x 2 ⎥⎥ = 0 ⎢⎣ x 3 ⎥⎦ ⎡x 4 ⎤ ⎡-.05⎤ = ⎢ ⎥ = ⎢ ⎥ ⎣x5 ⎦ ⎣-.10⎦ 1st iteration: By selection, the leaving variable is x5. The entering variable is determined by: ⎡-.75⎤ y1 = ⎢ ⎥, ⎣ .25 ⎦ ⎡ .75 ⎤ y2 = ⎢ ⎥, ⎣-.25⎦ ε0 = min { , .1 -3 , }=4 -.25 -.625 ⎡-.625⎤ y3 = ⎢ ⎥ ⎣-.625⎦ Above minimum corresponds to variable x2. Thus x2 is the entering variable. The new solution is: xB ⎡x 2 ⎤ ⎡ .4 ⎤ = ⎢ ⎥ = ⎢ ⎥ ⎣x 4 ⎦ ⎣-.35⎦ ⎡ .75 1⎤ AB = ⎢ ⎥ ⎣-.25 0⎦ A −B1 ⎡ x1 ⎤ ⎡0 ⎤ ⎢ ⎥ x r = ⎢ x 2 ⎥ = ⎢⎢0⎥⎥ ⎢⎣ x 3 ⎥⎦ ⎢⎣0⎥⎦ ⎡0 -4⎤ = ⎢ ⎥ ⎣1 3 ⎦ ⎡1⎤ CB = ⎢ ⎥ ⎣0 ⎦ 2nd iteration: By selection, the leaving variable is x4. The entering variable is determined by: ⎡-1⎤ y1 = ⎢ ⎥ ⎣0⎦ ε0 = min { ⎡ 2.5 ⎤ y3 = ⎢ ⎥ ⎣-2.5⎦ , -.5 , -.25 ⎡-4⎤ y5 = ⎢ ⎥ ⎣3⎦ } = .2 Above minimum corresponds to variable x3. Thus x3 is the entering variable. The new solution is: xB ⎡x 2 ⎤ ⎡.05⎤ = ⎢ ⎥ = ⎢ ⎥ ⎣ x4 ⎦ ⎣.14 ⎦ B-33 ⎡0 ⎤ ⎡ x1 ⎤ ⎥ ⎢ x r = ⎢x 4 ⎥ = ⎢⎢0⎥⎥ ⎢⎣0⎥⎦ ⎢⎣ x 5 ⎥⎦ This is the optimal solution. B.12 The Dual Simplex Method With Upper Bounds A large class of problems of the linear programming variety include upper bounds on the variables. This additional constraint can be implicitly incorporated in the solution algorithm. This paragraph examines problems of this type. In addition, it will be assumed that the dual simplex method of solution is desirable. The dual simplex method with upper bounds is developed as follows: Consider the following linear program: Minimize Subject to: cT x Ax = b x≤h x≥0 (B.24a) (B.24b) (B.24c) (B.24d) Introduction of slack variables, s, will transform the problem into standard form: Minimize Subject to: cT x ⎡ A 0⎤ ⎡ x ⎤ ⎡b⎤ ⎢ I I ⎥ ⎢ s ⎥ = ⎢h ⎥ ⎣ ⎦⎣ ⎦ ⎣ ⎦ x, s ≥ 0 Assume a basic dual feasible solution with the following properties: xB = B-1b xr = 0 s=h-x>0 The last inequality can be always satisfied with an appropriate transformation of variables. xi = -x'i + hi (B.25) Thus, in the considered solution, the variables s will be always basic and nonnegative. This solution shall be called an extended dual feasible solution. Now consider the application of the dual simplex method to the above problem. Observe that in block A of Figure B.3, the leaving variable will be an x variable. B-34 An s variable cannot be a leaving variable because an extended basic dual feasible solution has always s ≥ 0. Thus j ≤ m. Block B of Figure B.3 will yield: [c BT 0] z i - c i ; z ij < 0 ε0 = min i i zj In above expression, the index i extends over all nonbasic variables. Since an extended basic dual feasible solution is assumed, the vector z will be given with ⎡ B 0⎤ z = ⎢ ⎥ ⎣E I ⎦ i −1 ⎡a i ⎤ ⎡ ⎤ B-1 a i = ⎢ i⎥ ⎢ -1 i i⎥ ⎣e ⎦ ⎣-E B a + e ⎦ Thus the entry z ij is the jth entry of the vector B-1 ai. Define yi = B-1 ai. Then equation (B.1) becomes c TB y i - c i ; y i = B-1 a i , y ij < 0 ε0 = min i i yj (B.26) Let the minimum correspond to variable k. Then variable k will be the entering variable. If in the new solution, one or more variables exceed their upper bounds, the simple transformation xl = -x'l + hl will bring the solution into the assumed form. The above transformation applied to basic variables leaves invariant the dual feasibility of the solution. The algorithm of the dual simplex method with upper bounds is illustrated in Figure B.4. The algorithm will be demonstrated with an example. B-35 Assume a dual basic feasible solution x B , x r Is x B Yes >0? STOP No Select the most negative entry of : x Bj vector x B Variable j is the leaving variable Compute ε 0 = min i { c TB y i - c i y i j ; -1 i i y i =B a , y j <0 } i scans all nonbasic variables Assume minimum occurs for i = k Variable k is the entering variable Leaving variable is j Entering variable is k Compute new solution For all variables: x l = - x l +h l If x l > h l Figure B.4. The Dual Simplex Algorithm with Upper Bounds Example EB.9: Solve the following linear program using the dual simplex method with upper bounds. Minimize x1 + x2 + x3 + x4 + 3x5 + 3x6 Subject to: -.25x1 + .25x2 - .25x3 + .25x4 - .25x5 - .5x6 + x7 = -.05 -.125x1 + .125x2 + .375x3 - .375x4 - .125x5 - .25x6 + x8 = -.05 B-36 x1, x2, ..., x6 ≤ .18 x7, x8 ≤ ∞ x1, x2, ..., x8 ≥ 0 Solution: Observe that the solution x7 = -0.05 x8 = -0.05 x1 = x2 = ... = x6 = 0 is an extended dual basic feasible solution. Thus the algorithm of Figure B.4 can start with this solution. 1st iteration: The present solution is characterized with ⎡1 0⎤ AB = ⎢ ⎥ ⎣0 1 ⎦ ⎡x7 ⎤ ⎡-.05⎤ xB = ⎢ ⎥ = ⎢ ⎥ ⎣ x8 ⎦ ⎣-.05⎦ xr = x 1 x 2 x 3 x 4 x 5 x 6 T By selection of the most negative variable (in this case both variables have the same negative value) the leaving variable is x7. The entering variable is determined with the optimality condition. For this purpose, the following is computed. ⎡ - 0.25 ⎤ y2 = ⎢ ⎥ ⎣- 0.125⎦ ⎡ - 0.25 ⎤ y5 = ⎢ ⎥ ⎣- 0.125⎦ ⎡ - 0.25 ⎤ y1 = ⎢ ⎥ ⎣− 0.125⎦ ⎡ 0.25 ⎤ y4 = ⎢ ⎥ ⎣- 0.375⎦ Then: ε = min [ ⎡ - 0.25 ⎤ y3 = ⎢ ⎥ ⎣- 0.375⎦ ⎡ - 0.5 ⎤ y6 = ⎢ ⎥ ⎣- 0.25⎦ − 1 − 0.25, − 1 − 0.25, − 3 − 0.25, − 3 − 0.5, ] = 4 ↑ x1 ↑ x3 ↑ x5 ↑ x6 Above minimum corresponds to variables x1 or x3. Either can enter the solution. Select x1 as the entering variable. The new solution is: x1 = 0.20 B-37 x8= -0.025 Note that x1 violates the upper bound. Thus, replace with x1 = - x'1 + .18 Upon substitution, the problem becomes: Minimize Subject to: -x'1 + x2 + x3 + x4 + 3x5 + 3x6 .25x'1 + .25x2 - .25x3 + .25x4 - .25x5 - .5x6 + x7 = -.005 .125x'1 + .125x2 + .375x3 - .375x4 - .125x5 - .25x6 + x8 = -.0275 The new solution is: x'1 = -.02, x8 = -.025 all others zero ⎡ .25 0⎤ ⎡ 4 0⎤ AB = ⎢ → AB-1 = ⎢ ⎥ ⎥ ⎣.125 1⎦ ⎣−.5 1⎦ From theory, it is known that this solution is an extended dual basic feasible solution. As an exercise, the dual feasibility of the solution will be verified by computing the reduced cost coefficients: rcc2 = 1 - [-1 rcc3 = 1 - [-1 rcc4 = 1 - [-1 rcc5 = 3 - [-1 rcc6 = 3 - [-1 rcc7 = 0 - [-1 ⎡4 0] ⎢ ⎣−.5 ⎡4 0] ⎢ ⎣−.5 .25 ⎤ =2>0 1 .125⎥⎦ 0 −.25⎤ =0 1 .375 ⎥⎦ ⎡4 0] ⎢ ⎣−.5 ⎡4 0] ⎢ ⎣−.5 .25 ⎤ =2>0 1 −.375⎥⎦ 0 0 −.25 ⎤ =2>0 1 −.125⎥⎦ ⎡ 4 0 −.5 ⎤ 0] ⎢ ⎥ =1>0 ⎣−.5 1 −.25⎦ 0 ⎡ 4 0 1⎤ 0] ⎢ ⎥ =4>0 ⎣−.5 1 0⎦ 2nd iteration: By selection, the leaving variable is x8. The entering variable is determined by: ε = min [ -2/-0.5, ↑ -4/-0.5 ↑ B-38 ]=4 x4 x7 Above minimum corresponds to variable x4. Thus x4 is the entering variable. The new solution is: x'1 = -0.07 x4 = 0.05 all other variables equal zero. .25 ⎤ ⎡ .25 AB = ⎢ ⎥ , ⎣.125 −.375⎦ ⎡3.0 2.0 ⎤ AB-1= ⎢ . −2.0⎥⎦ ⎣10 This solution is dual feasible. 3rd iteration: By selection, the leaving variable is x'1. The entering variable is determined by: ε = min [ -2/-1, ↑ x5 -1/-2 ↑ x6 ] = 0.5 Above minimum corresponds to variable x6 which will be selected as the entering variable. The new solution is: x4 = 0.05 x6 = 0.035 All other variables equal zero. −.5 ⎤ ⎡ .25 AB = ⎢ ⎥ ⎣−.375 −.25⎦ → ⎡ 1. −2.⎤ AB-1 = ⎢ . −1.⎥⎦ ⎣−15 Since x4, x6 ≥ 0, this is the optimal solution. In terms of the original variables, this solution is: x1 = .18 x4 = .05 x6 = .035 B-39 B.12 Duality – The Dual Linear Program To each linear program, a dual problem exists. We refer to the original problem as the primal problem. The dual problem is intimately related to the primal problem. The relationships between the primal and dual problems can be exploited for a variety of applications. We will define the dual linear program and then we will investigate the properties of it. Consider a linear program in standard form: Min z = c T x Subject to : Ax = b, x≥0 The dual linear program is: Max z = λT b Subject to : λT A ≤ c T Above linear programs have the following property (duality theorem of linear programming): If either of the linear programs has a finite optimal solution, so does the other. The optimal value of the objective function is the same for both problems. There are two very important properties that connect the primal and dual linear programs. The first is the property of complementary slackness. B.12.1 Complementary Slackness Consider the primal linear program in standard form and the corresponding dual problem. Let the optimal solution for the primal problem be: ⎡ x ⎤ ⎡ A −1 b ⎤ x = ⎢ B ⎥ = ⎢ B ⎥, ⎣ xr ⎦ ⎣ 0 ⎦ x B = basic var iables, x r = nonbasic var iables The complementary slackness theorem states that the optimal solution l for the dual problem will meet the following necessary and sufficient condition. For each basic variable i: x i > 0 → λT a i = c i For each nonbasic variable i: λT a i < c i → x i = 0 The implication of this theorem is that once the optimal solution of the primal problem is known, then the optimal solution of the dual problem can be immediately computed from the equation: B-40 λT AB = c BT B.12.2 Sensitivity Consider the primal linear program in standard form and the corresponding dual problem. Let the optimal solutions for the primal and dual problem be: ⎡ x ⎤ ⎡ A −1 b ⎤ x = ⎢ B ⎥ = ⎢ B ⎥, ⎣ xr ⎦ ⎣ 0 ⎦ λT = c BT AB−1 x B = basic var iables, x r = nonbasic var iables Assuming that the solution is not degenerate (all basic variables are positive – nonzero), then the following holds: ∆ z = λ T ∆b This property enables the computation of the sensitivity of the objective function with respect to the constants b. B.13 Interior Point Methods The simplex method with its various improved computational procedures is simple and efficient for relatively small problems, i.e. problems with a small number of constraints and variables. However as the size of the problem increases, the computational requirements of the method increase disproportionally with the system size. This has been recognized for a long time as a drawback of the simplex type methods. Minty and Glee have shown that it ⎛ n⎞ is possible for an LP problem to require ⎜ ⎟ iterations to converge. Based on ⎝ m⎠ Minty and Glee's work, Barnes constructed the following example: Minimize: Subject to: x0 = - 2n-1x1 - 2n-2x2 - ... - 2xn-1 - xn x1 22x1 + x2 23x1 + 22x2 + x3 ...... 2nx1 + 2n-1x2 + ... + 22xn-1 + xn xi ≥ 0, i = 1, 2, ..., n ≤5 ≤ 52 ≤ 53 ≤ 5n It can be proven that the number of iterations required for the solution of above problem using the described simplex algorithm (entering variable is always the B-41 one with the most negative reduced cost coefficient) is 2n. The reader is encouraged to work out this proof by mathematical induction. (Hint: work problem for n = 2, then for n = 3. Then examine the common parts of the two solutions.) Note that even for moderate values of n, i.e. n = 100, the number of required iterations is N = 1.26 x 1030 !!! For above problem, the computational requirements of the simplex method are extremely high. Recently, a new method has been developed for linear problems, which overcomes the drawbacks of the simplex method for large scale problems. This method starts from a feasible solution, which is an interior point in the space of the constraints Ax = b, and projects to a point which is closer to the optimal solution. For this reason these methods are coined interior point method to be distinguished from the simplex type methods which move from vertex to vertex. Interior point methods have been undergone extensive developments with noticeable milestones established by Kachian[xxx], Karmakar[xxx] and Fiacco & McCormick[xxx]. Presently, the most efficient method is a variation of a barrier method based on the primal dual interior point method. This method is presented here. We study the linear problem defined with Minimize Subject to: x0 = cT x = be Ae x ≤ bi Ai x x ≤h ≥0 x By introducing slack / surplus variables s and w above problem is transformed to: Minimize Subject to cT x A e x = be A i x + s = bi x + w = h x > 0, s > 0, w > 0 (26) Let λ, µ, and ν denote the Lagrange multipliers for the constraints in Eq.(26). The inequalities are dealt with logarithmic barrier functions [5], yielding the following Lagrangian: L(x, s, w, λ, µ, ν, σ) = cTx - σ (Σlnx i+ Σlnsi + Σlnwj) B-42 + λT(be - Aex) + µT [bi - Aix - s] + νT(x+w-h) In above equation σ is a barrier penalty. The barrier σ is formulated as a function of duality gap in view of the fact that feasibility is retained. Therefore, as the solution is approaching convergence, the barrier reduces from relatively very large value to zero. It is important that σ be initialized at a relatively large value and then be decreased at each iteration of the algorithm. McShane, et al [10] suggests that the parameter σ be selected with: σk+1 = duality gap(k) / Ψ(n) where: k iteration number duality gap(k) is the duality gap at iteration k ⎧⎪ n2 if n < 5000 Ψ(n) = ⎨ if n > 5000 ⎩⎪ n n n number of variables For a fixed σ, the Kuhn-Tucker first order necessary conditions are expressed by equating the gradient ∇L to zero yielding the following: Ae x Ai x + s x+w AeTλ + AiTµ - ν + z XZe SMe WNe = be = bi = h = c = σe = -σ e = σe (27) where: X = diag(x1,...,xn), S = diag(s1,...,sI), N = diag(ν1,...,νn), M = diag(µ1,...,µI), e = (1,1,...,1)T, n: the dimension of control variables x, I: the number of inequality constraints W = diag(w1,...,wn), Z = diag(z1,...,zn), Equations (27) are solved iteratively using Newton's method which consists of successively solving the linearized Equataions (27). The Newton's algorithm is as follows: ⎛ ∆λ ⎞ ⎧⎛ A e ⎜ ⎟ = ⎨⎜ ⎝ ∆µ ⎠ ⎩⎝ Ai 0⎞ ⎛θ x ⎟⎜ I⎠ ⎝ 0 0 ⎞ ⎛ AeT ⎟⎜ θ s⎠⎝ 0 AiT ⎞ ⎪⎫ ⎟⎬ I ⎠ ⎪⎭ B-43 −1 ⎪⎧⎡⎛ be ⎞ ⎛ Ae ⎨⎢⎜ ⎟ − ⎜ ⎪⎩⎣⎝ bi ⎠ ⎝ Ai 0⎞ ⎛ x ⎞ ⎤ ⎟⎜ ⎟⎥ I ⎠ ⎝ s⎠ ⎦ ⎛ Ae +⎜ ⎝ Ai 0⎞ ⎛ θ x ⎟⎜ I⎠ ⎝ 0 ⎛ ∆x⎞ ⎛θ x 0 ⎞ ⎧⎪⎛ AeT ⎟ ⎨⎜ ⎜ ⎟ =⎜ ⎝ ∆s ⎠ ⎝ 0 θ s ⎠ ⎪⎩⎝ 0 0 ⎞ ⎡⎛ ε x (σ )⎞ ⎛ ⎛ c⎞ ⎛ AeT ⎟ ⎢⎜ ⎟ + ⎜⎜ ⎟ − ⎜ θ s ⎠ ⎣⎝ ε s (σ )⎠ ⎝ ⎝ 0⎠ ⎝ 0 AiT ⎞ ⎛ ∆λ ⎞ ⎛ ε x (σ )⎞ ⎡⎛ c⎞ ⎛ AeT ⎟⎜ ⎟ − ⎜ ⎟ − ⎢⎜ ⎟ − ⎜ I ⎠ ⎝ ∆µ ⎠ ⎝ ε s (σ )⎠ ⎣⎝ 0⎠ ⎝ 0 AiT ⎞ ⎛ λ ⎞ ⎛ ν ⎞ ⎛ z ⎞ ⎞ ⎤ ⎫⎪ ⎟ ⎜ ⎟ + ⎜ ⎟ − ⎜ ⎟ ⎟ ⎥⎬ I ⎠ ⎝ µ ⎠ ⎝ 0⎠ ⎝ − µ ⎠ ⎠ ⎥⎦ ⎪⎭ AiT ⎞ ⎛ λ ⎞ ⎛ ν ⎞ ⎛ z ⎞ ⎤ ⎪⎫ ⎟ ⎜ ⎟ + ⎜ ⎟ − ⎜ ⎟ ⎥⎬ I ⎠ ⎝ µ ⎠ ⎝ 0⎠ ⎝ − µ ⎠ ⎦ ⎪⎭ ∆z = -X-1 Z ∆x - Z e + σ X-1 e ∆ν = W-1 N ∆x - N e + σ W-1 e ∆w = -∆x where: θx = ( X-1Z + W-1 N )-1 , εx(σ) = -( N - Z ) e - σ ( X-1 - W-1) e θs = -M -1 S , εs(σ) = -( M + σ S-1 ) e In order to maintain feasibility, the full Newton step has to be scaled first before updating variables. Specifically, first the step lengths αP and αD in the primal and dual space are computed on the basis of feasibility. Next the variables are updated with: xk+1 sk+1 wk+1 λk+1 µk+1 νk+1 zk+1 = = = = = = = xk + αP ∆x sk + αP ∆s wk + αP ∆w λk + αD ∆λ µk + αD ∆µ νk + αD ∆ν zk + αD ∆z (28) The algorithm terminates when the normalized duality gap (objective function value of primal problem minus objective function of dual problem) becomes smaller than a predetermined tolerance value ε: cT x − ( b eTλ + b Ti µ − h T ν) <ε 1 + b eTλ + b Ti µ − h T ν (29) More details of this algorithm can be found in [8] and [9]. B.14 Summary and Discussion This chapter presented a review of linear programming and extensions. The basic concepts of LP have been presented. Computational issues were discussed. Then LP extensions suitable for power system applications were discussed. In particular, the dual simplex method has been presented. An extension of this B-44 method, the dual simplex method with upper bounds is extremely efficient in a certain class of problems. B.15 Problems Problem PB.1: A certain engineering problem results in the following optimization problem. Minimize Subject to J = |∆Pg2|+|∆Pg3| |1.9 + 0.294 ∆Pg2 - 0.156∆Pg3 | ≤ 1.7 -0.5 ≤ ∆Pg2 +∆Pg3 ≤ 2.0 -2.1 ≤ ∆Pg2≤ 0.15 -1.1 ≤ ∆Pg3≤ 0.35 Transform above problem into a linear programming problem in standard form. Then determine whether this problem has a solution (use Phase I). Problem PB.2: Solve the following linear program using the simplex method. Minimize: x0 = x1 + x2 + x3 + x4 + 3x5 + 3x6 Subject to: -0.25x1 + 0.25x2 - 0.25x3 + 0.25x4 - 0.25x5 - 0.5x6 + x7 = -0.05 -0.125x1 + 0.125x2 + 0.375x3 - 0.375x4 - 0.125x5 - 0.25x6 + x8 = -0.05 x1, x2, ..., x6 ≤ 0.18 x1, x2, x3, ..., x8 ≥ 0 Problem PB.3: Find a basic feasible solution for the following LP problem using Phase I of the simplex method. Then apply phase II of the simplex method to find the optimal solution. Minimize Subject to 4x1 + 3x2 + 5x3 + 6x4 2x1 + 2x2 + 3x3 + x4 - x5 = 2 -4x1 + x2 - x3 + 3x4 + x6 = -3 x1, x2, x3, x4, x5 x6 ≥ 0 Problem PB.4: Consider the following linear program in standard form: Minimize Subject to x1 + x2 + 2x3 -0.9x1 + 0.9x2 - 0.5x3 + x4 = 0.05 -1.1x1 + 1.1x2 + 0.5x3 - x5 = 0.05 x1, x2, x3, x4, x5 ≥ 0 B-45 and the basic solution x1 = x2 = x3 = 0 x4 = 0.05 x5 = -0.05 (a) Show that the above basic solution is dual feasible. (b) Starting with the above basic dual feasible solution, solve the problem with the dual simplex method. Problem PB.5: Solve the following LP problem using the dual simplex method: Minimize Subject to 6x1 + 4x2 2x1 + 2x2 - x3 = 3 3x1 + x2 + - x4 = 2 x1, x2, x3, x4, ≥ 0 Problem PB.6: Solve the following linear program problem using the dual simplex method with upper bounds: Minimize Subject to x4 -0.3x1 - 0.022x2 + 0.022x3 + x4 = -0.013 x1 ≤ 0.15 x2 ≤ 0.0536 x3 ≤ 0.5964 x4 ≤ ∞ x1, x2, x3, x4, ≥ 0 Problem PB.7: The following linear program results from a certain security constraint economic dispatch: Minimize Subject to x1 + x2 + x3 + x4 + 5x5 -0.21x1 + 0.21x2 - 0.20x3 + 0.20x4 - 0.25x5 + x6 = -0.04 -0.12x1 + 0.12x2 + 0.30x3 - 0.30x4 - 0.125x5 + x7 = -0.05 x1 ≤ 0.20 x2 ≤ 0.20 x3 ≤ 0.15 x4 ≤ 0.05 x5 ≤ 0.35 x1, x2, x3, x4, x5, x6, x7 ≥ 0 (a) Identify (by inspection) a dual feasible solution. B-46 (b) Perform one iteration of the dual simplex method with upper bounds. (c) Compute the new solution and cast problem in a form ready for the next iteration. Problem PB.8: Solve the following linear program using the dual simplex method with upper bounds. Use the following starting solution: xi = 0, i = 1, 2, ..., 6 , x7 = 0.0875 , x8 = - 0.013 Minimize x1 + x2 Subject to 0.269x1 - 0.269x2 + 0.2x3 - 0.2x4 + 0.053x5 - 0.053x6 + x7 = - 0.0875 - 0.3x3 + 0.3x4 + 0.022x5 - 0.022x6 + x8 = - 0.013 x1 ≤ 1.0 x2 ≤ 0.8 x3 ≤ 0.05 x4 ≤ 0.1 x5 ≤ 0.0536 x6 ≤ 0.5964 x1, x2, x3, x4, x5, x6 ≥ 0 Problem PB.9: Consider the following linear programming problem. Minimize x1 + x2 Subject to 0.25x1 - 0.25x2 + 0.1x3 - 0.1x4 - 0.05x5 + x6 = - 0.05 x1 ≤ 0.1 x2 ≤ 0.5 x3 ≤ 0.18 x4 ≤ 0.12 x5 ≤ 0.5 x6 unbounded x1, x2, x3, x4, x5, x6 ≥ 0 Consider the following basic solution x1 = x2 = x3 = x4 = x5 = 0 x6 = - 0.05 (a) Is this basic solution dual feasible? Is this basic solution an extended dual feasible solution? B-47 (b) Solve above problem with the dual simplex method with upper bounds. Problem PB.10: Solve the following linear program using the barrier method based on the primal - dual interior point method. Minimize: x0 = x1 + x2 + x3 + x4 + 3x5 + 3x6 Subject to: -0.25x1 + 0.25x2 - 0.25x3 + 0.25x4 - 0.25x5 - 0.5x6 + x7 = -0.05 -0.125x1 + 0.125x2 + 0.375x3 - 0.375x4 - 0.125x5 - 0.25x6 + x8 = -0.05 x1, x2, ..., x6 ≤ 0.18 x1, x2, x3, ..., x8 ≥ 0 B-48