Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and b are constant real or complex numbers The constant a i is called the coefficient of x i ; and b is called the constant term of the equation A system of linear equations (or linear system) is a finite collection of linear equations in same variables For instance, a linear system of m equations in n variables x, x,, x n can be written as a x + a x + + a n x n = b a x + a x + + a n x n = b () a m x + a m x + + a mn x n = b m A solution of a linear system () is a tuple (s, s,, s n ) of numbers that makes each equation a true statement when the values s, s,, s n are substituted for x, x,, x n, respectively The set of all solutions of a linear system is called the solution set of the system Theorem Any system of linear equations has one of the following exclusive conclusions (a) No solution (b) Unique solution (c) Infinitely many solutions A linear system is said to be consistent if it has at least one solution; and is said to be inconsistent if it has no solution Geometric interpretation The following three linear systems x +x = 3 (a) x x = x x = 4 (b) x +x = 3 x x = 5 x x = 4 (c) x +x = 3 4x +x = 6 6x +3x = 9 have no solution, a unique solution, and infinitely many solutions, respectively See Figure Note: A linear equation of two variables represents a straight line in R A linear equation of three variables represents a plane in R 3 In general, a linear equation of n variables represents a hyperplane in the n-dimensional Euclidean space R n Matrices of a linear system

x x x o x o x o x Figure : No solution, unique solution, and infinitely many solutions Definition The augmented matrix of the general linear system () is the table a a a n b a a a n b a m a m a mn b m and the coefficient matrix of () is a a a n a a a n a m a m a mn () (3)

Systems of linear equations can be represented by matrices Operations on equations (for eliminating variables) can be represented by appropriate row operations on the corresponding matrices For example, x +x x 3 = x 3x +x 3 = 8 3 8 3x +x +4x 3 = 7 3 4 7 [Eq ] [Eq ] [Eq 3] 3[Eq ] x +x x 3 = 5x +5x 3 = x +x 3 = 4 ( /5)[Eq ] ( /)[Eq 3] x +x x 3 = x x 3 = x 5x 3 = R R R 3 3R 5 5 4 ( /5)R ( /)R 3 5 [Eq 3] [Eq ] R 3 R x +x x 3 = x x 3 = 4x 3 = 4 4 4 ( /4)[Eq 3] ( /4)R 3 x +x x 3 = x x 3 = x 3 = [Eq ] + [Eq 3] [Eq ] + [Eq 3] x +x = 3 x = 3 x 3 = R + R 3 R + R 3 3 3 [Eq ] [Eq ] R R x = x = 3 3 x 3 = Elementary row operations Definition 3 There are three kinds of elementary row operations on matrices: (a) Adding a multiple of one row to another row; (b) Multiplying all entries of one row by a nonzero constant; (c) Interchanging two rows Definition 4 Two linear systems in same variables are said to be equivalent if their solution sets are the same A matrix A is said to be row equivalent to a matrix B, written A B, if there is a sequence of elementary row operations that changes A to B 3

Theorem 5 If the augmented matrices of two linear systems are row equivalent, then the two systems have the same solution set In other words, elementary row operations do not change solution set Proof It is trivial for the row operations (b) and (c) in Definition 3 Consider the row operation (a) in Definition 3 Without loss of generality, we may assume that a multiple of the first row is added to the second row Let us only exhibit the first two rows as follows a a a n b a a a n b (4) a m a m a mn b m Do the row operation (Row ) + c(row ) We obtain a a a n b a + ca a + ca a n + ca n b + cb a 3 a 3 a 3n b 3 a m a m a mn b m (5) Let (s, s,, s n ) be a solution of (4), that is, In particular, Multiplying c to both sides of (7), we have Adding both sides of (8) and (9), we obtain a i s + a i s + + a in s n = b i, i m (6) a s + a x + + a n s n = b, (7) a s + a x + + a n s n = b (8) ca s + ca + + ca n s n = cb (9) (a + ca )s + (a + ca )s + + (a n + ca n )s n = b + cb () This means that (s, s,, s n ) is a solution of (5) Conversely, let (s, s,, s n ) be a solution of (5), ie, () is satisfied and the equations of (6) are satisfied except for i = Since a s + a x + + a n s n = b, multiplying c to both sides we have Note that () can be written as c(a s + a + + a n s n ) = cb () (a s + a s + + a n s n ) + c(a s + a s + + a n s n ) = b + cb () Subtracting () from (), we have This means that (s, s,, s n ) is a solution of (4) a s + a s + + a n s n = b 4

Row echelon forms Definition A matrix is said to be in row echelon form if it satisfies the following two conditions: (a) All zero rows are gathered near the bottom (b) The first nonzero entry of a row, called the leading entry of that row, is ahead of the first nonzero entry of the next row A matrix in row echelon form is said to be in reduced row echelon form if it satisfies two more conditions: (c) The leading entry of every nonzero row is (d) Each leading entry is the only nonzero entry in its column A matrix in (reduced) row echelon form is called a (reduced) row echelon matrix Note Sometimes we call row echelon forms just as echelon forms and row echelon matrices as echelon matrices without mentioning the word row Row echelon form pattern The following are two typical row echelon matrices, where the circled stars represent arbitrary nonzero numbers, and the stars represent arbitrary numbers, including zero The following are two typical reduced row echelon matrices, Definition If a matrix A is row equivalent to a row echelon matrix B, we say that A has the row echelon form B; if B is further a reduced row echelon matrix, then we say that A has the reduced row echelon form B Row reduction algorithm Definition 3 A pivot position of a matrix A is a location of entries of A that corresponds to a leading entry in a row echelon form of A A pivot column (pivot row) is a column (row) of A that contains a pivot position Algorithm (Row Reduction Algorithm) () Begin with the leftmost nonzero column, which is a pivot column; the top entry is a pivot position () If the entry of the pivot position is zero, select a nonzero entry in the pivot column, interchange the pivot row and the row containing this nonzero entry (3) If the pivot position is nonzero, use elementary row operations to reduce all entries below the pivot position to zero, (and the pivot position to and entries above the pivot position to zero for reduced row echelon form) (4) Cover the pivot row and the rows above it; repeat ()-(3) to the remaining submatrix 5

Theorem 4 Every matrix is row equivalent to one and only one reduced row echelon matrix In other words, every matrix has a unique reduced row echelon form Proof The Row Reduction Algorithm show the existence of reduced row echelon matrix for any matrix M We only need to show the uniqueness Suppose A and B are two reduced row echelon forms for a matrix M Then the systems Ax = and Bx = have the same solution set Write A = [a ij ] and B = [b ij ] We first show that A and B have the same pivot columns Let i,, i k be the pivot columns of A, and let j,, j l be the pivot columns of B Suppose i = j,, i r = j r, but i r j r Assume i r < j r Then the i r th row of A is [,,,, ar,ir+,, a r,jr, a r,jr+,, a r,n ] While the j r th row of B is [,,,, br,jr+,, b r,n ] Since i r = j r and i r < j r, we have j r < i r < j r So x ir is a free variable for Bx = Let u i = b,ir,, u ir = b r,ir, u ir =, and u i = for i > i r Then u is a solution of Bx =, but is not a solution of Ax = This is a contradiction Of course, k = l Next we show that for r k = l, we have a rj = b rj, j r + j j r+ Otherwise, we have a rj b rj such that r is smallest and then j is smallest Set u j =, u i = a,j,, u r = a rj, and u j = otherwise Then u is a solution of Ax =, but is not a solution of Bx = This is a contradiction Solving linear system Example Find all solutions for the linear system x +x x 3 = x +x +4x 3 = 3x +3x +4x 3 = Solution Perform the row operations: 4 3 3 4 7 4 R R R 3 3R R + R 3 R + R 3 3 6 3 7 4 ( /3)R R 3 R R R The system is equivalent to which means the system has a unique solution x = 7 x = 4 x 3 = 6

Example Solve the linear system x x +x 3 x 4 = x x +x 3 +x 4 = 4x 4x +4x 3 = 4 x +x x 3 +x 4 = 3 Solution Do the row operations: 4 4 4 4 3 R R R 3 4R R 4 + R R + R 4 4 () [ ] [] () (/)R R 3 R R 4 + (/)R The linear system is equivalent to { x = + x x 3 x 4 = We see that the variables x, x 3 can take arbitrary numbers; they are called free variables Let x = c, x 3 = c, where c, c R Then x = + c c, x 4 = All solutions of the system are given by x = + c c x = c x 3 = c x 4 = The general solutions may be written as x x = x x 3 = + c x 4 + c, where c, c R Set c = c =, ie, set x = x 3 =, we have a particular solution x = For the corresponding homogeneous linear system Ax =, ie, x x +x 3 x 4 = x x +x 3 +x 4 = 4x 4x +4x 3 = x +x x 3 +x 4 = we have 4 4 4 () [ ] [] () 7

Set c =, c =, ie, x =, x 3 =, we obtain one basic solution x = for the homogeneous system Ax = Set c =, c =, ie, x =, x 3 =, we obtain another basic solution x = for the homogeneous system Ax = Example 3 The linear system with the augmented matrix 5 3 3 4 has no solution because its augmented matrix has the row echelon form () ( 3) [7] The last row represents a contradictory equation = Theorem 5 A linear system is consistent if and only if the row echelon form of its augmented matrix contains no row of the form [,, b ], where b Example 4 Solve the linear system whose augmented matrix is A = 3 6 3 3 7 4 4 6 5 4 Solution Interchanging Row and Row, we have 3 6 3 3 7 4 4 6 5 4 4 4 5 6 4 4 5 6 R 3R R 4 R R R 3 R 4 + R 8

Then the system is equivalent to 4 3 6 R 4 + 3 R 3 R 3 R R 3 () [] () [ ] [] () x = x x 4 + x 5 x 3 = + x 4 x 5 x 6 = The unknowns x, x 4 and x 5 are free variables Set x = c, x 4 = c, x 5 = c 3, where c, c, c 3 are arbitrary The general solutions of the system are given by x = c c + c 3 x = c x 3 = + c c 3 x 4 = c x 5 = c 3 x 6 = The general solution may be written as x x x 3 x 4 = + c x 5 x 6 + c + c 3, c, c, c 3 R Definition 6 A variable in a consistent linear system is called free if its corresponding column in the coefficient matrix is not a pivot column Theorem 7 For any homogeneous system Ax =, 3 Vector Space R n #{variables} = #{pivot positions of A} + #{free variables} Vectors in R and R 3 Let R be the set of all ordered pairs (u, u ) of real numbers, called the -dimensional Euclidean space Each ordered pair (u, u ) is called a point in R For each point (u, u ) we associate a column [ u u ], called the vector associated to the point (u, u ) The set of all such vectors is still denoted by R So by a point in the Euclidean space R we mean an ordered pair (x, y) 9

and, by a vector in the vector space R we mean a column [ ] x y Similarly, we denote by R 3 the set of all tuples (u, u, u 3 ) of real numbers, called points in the Euclidean space R 3 We still denote by R 3 the set of all columns u u, u 3 called vectors in R 3 For example, (, 3, ), ( 3,, ), (,, ) are points in R 3, while 3 3,, are vectors in the vector space R 3 Definition 3 The addition, subtraction, and scalar multiplication for vectors in R are defined by [ ] [ ] [ ] u v u + v + =, u v u + v [ u ] [ ] v = u v [ ] u c = u [ ] u v, u v [ cu cu ] Similarly, the addition, subtraction, and scalar multiplication for vectors in R 3 are defined by u u u 3 + =, Vectors in R n u u u 3 c v v v 3 u u u 3 v v v 3 = = u + v u + v u 3 + v 3 cu cu cu 3 u v u v u 3 v 3 Definition 3 Let R n be the set of all tuples (u, u,, u n ) of real numbers, called points in the n- dimensional Euclidean space R n We still use R n to denote the set of all columns u u u n,

of real numbers, called vectors in the n-dimensional vector space R n The vector = is called the zero vector in R n The addition and the scalar multiplication in R n are defined by u v u v u n + c u u u n v n = = cu cu cu n u + v u + v u n + v n Proposition 33 For vectors u, v, w in R n and scalars c, d in R, () u + v = v + u, () (u + v) + w = u + (v + w), (3) u + = u, (4) u + ( u) =, (5) c(u + v) = cu + cv, (6) (c + d)u = cu + du (7) c(du) = (cd)u (8) u = u Subtraction can be defined in R n by Linear combinations u v = u + ( v) Definition 34 A vector v in R n is called a linear combination of vectors v, v,, v k in R n if there exist scalars c, c,, c k such that v = c v + c v + + c k v k The set of all linear combinations of v, v,, v k is called the span of v, v,, v k, denoted Span {v, v,, v k } Example 3 The span of a single nonzero vector in R 3 is a straight line through the origin For instance, Span 3 = t 3 : t R,

is a straight line through the origin, having the parametric form x = 3t x = t, t R x 3 = t Eliminating the parameter t, the parametric equations reduce to two equations about x, x, x 3, { x +3x = x +x 3 = Example 3 The span of two linearly independent vectors in R 3 is a plane through the origin instance, Span 3, = s + t 3 : s, t R is a plane through the origin, having the following parametric form x = s +3t x = s t x 3 = s +t Eliminating the parameters s and t, the plane can be described by a single equation Example 33 Given vectors in R 3, v =, v = 3x 5x 7x 3 =, v 3 =, v 4 = 4 3 For (a) Every vector b = [b, b, b 3 ] T in R 3 is a linear combination of v, v, v 3 (b) The vector v 4 can be expressed as a linear combination of v, v, v 3, and it can be expressed in one and only one way as a linear combination v = v v + 3v 3 (c) The span of {v, v, v 3 } is the whole vector space R 3 Solution (a) Let b = x v + x v + x 3 v 3 Then the vector equation has the following matrix form x b x = b x 3 b 3 Perform row operations: Thus b b b 3 b b b b b 3 + b b b b b b 3 b b 3 b + b + b 3 b b b 3 x = b b 3, x = b + b + b 3, x 3 = b b b 3 So b is a linear combination of v, v, v 3 (b) In particular, v 4 = v v + 3v 3 (c) Since b is arbitrary, we have Span {v, v, v 3 } = R 3

Example 34 Consider vectors in R 3, v =, v =, v 3 = 5 ; u = 7, v = (a) The vector u can be expressed as linear combinations of v, v, v 3 in more than one ways For instance, u = v + v + v 3 = 4v + 3v = v v + v 3 (b) The vector v can not be written as a linear combination of v, v, v 3 Geometric interpretation of vectors Multiplication of matrices Definition 35 Let A be an m n matrix and B an n p matrix, a a a n a a a n A = a m a m a mn, B = The product (or multiplication) of A and B is an m p matrix c c c p c c c p C = c m c m c mp whose (i, k)-entry c ik, where i m and k p, is given by c ik = b b b p b b b p b n a n b np n a ij b jk = a i b k + a i b k + + a in b nk j= Proposition 36 Let A be an m n matrix, whose column vectors are denoted by a, a,, a n Then for any vector v in R n, v v Av = [a, a,, a n ] = v a + v a + + v n a n v n Proof Write Then a = A = a a a m a a a n a a a n a m a m a mn, a = a a a m, v = v v v n,, a n = a n a n a mn 3

Thus Av = = = = v a a a n a a a n a m a m a mn v v v n a v + a v + + a n v n a v + a v + + a n v n a m v + a m v + + a mn v n a v a v a m v a a a m + + v a v a v a m v a a a m + + + + v n a n v n a n v n a mn v n a n a n a mn = v a + v a + + v n a n Theorem 37 Let A be an m n matrix Then for any vectors u, v in R n and scalar c, (a) A(u + v) = Au + Av, (b) A(cu) = cau 4 The four expressions of a linear system A general system of linear equations can be written as a x + a x + + a n x n = b a x + a x + + a n x n = b a m x + a m x + + a mn x n = b m (4) We introduce the column vectors: a = a a m,, a n = a n a mn ; x = x x n ; b = b b m ; and the coefficient matrix: A = a a a n a a a n = [ a, a,, a n ] a m a m a mn Then the linear system (4) can be expressed by 4

(a) The vector equation form: (b) The matrix equation form: x a + x a + + x n a n = b, Ax = b, (c) The augmented matrix form: [ a, a,, a n b ] Theorem 4 The system Ax = b has a solution if and only if b is a linear combination of the column vectors of A Theorem 4 Let A be an m n matrix The following statements are equivalent (a) For each b in R m, the system Ax = b has a solution (b) The column vectors of A span R m (c) The matrix A has a pivot position in every row Proof (a) (b) and (c) (a) are obvious (a) (c): Suppose A has no pivot position for at least one row; that is, A ρ ρ ρ k ρ k A Ak Ak, where ρ, ρ,, ρ k are elementary row operations, and A k is a row echelon matrix Let e m = [,,, ] T Clearly, the system [A e n ] is inconsistent Let ρ i denote the inverse row operation of ρ i, i k Then [A k e m ] ρ k [A k b ] ρ k ρ [A b k ] ρ [A b k ] Thus, for b = b k, the system [A b] has no solution, a contradiction Example 4 The following linear system has no solution for some vectors b in R 3 x +x 3 +3x 4 = b x +4x +6x 3 +7x 4 = b x +x +x 3 +x 4 = b 3 The row echelon matrix of the coefficient matrix for the system is given by 3 4 6 7 R R 3 4 6 7 R R 3 3 3 R 3 R 3 Then the following systems have no solution 3 R 3 + R 4 6 7 R 3 R 3 3 3 3 4 6 7 R + R Thus the original system has no solution for b =, b = b 3 = 5

5 Solution structure of a linear System Homogeneous system A linear system is called homogeneous if it is in the form Ax =, where A is an m n matrix and is the zero vector in R m Note that x = is always a solution for a homogeneous system, called the zero solution (or trivial solution); solutions other than the zero solution are called nontrivial solutions Theorem 5 A homogeneous system Ax = has a nontrivial solution if and only if the system has at least one free variable Moreover, # { pivot positions } + # { free variables } = # { variables } Example 5 Find the solution set for the nonhomogeneous linear system x x +x 4 +x 5 = x +x x 3 4x 4 3x 5 = x x +x 3 +3x 4 +x 5 = x +x +x 3 +x 4 3x 5 = Solution Do row operations to reduce the coefficient matrix to the reduced row echelon form: R + R 4 3 R 3 R 3 3 R 4 + R () [ ] () [] [ ] Then the homogeneous system is equivalent to { x = x x 4 x 5 ( )R R 3 + R R 4 + R x 3 = x 4 +x 5 The variables x, x 4, x 5 are free variables Set x = c, x 4 = c, x 5 = c 3 We have x c c c 3 x x 3 x 4 = c c + c 3 c = c + c + c 3 x 5 c 3 Set x =, x 4 =, x 5 =, we obtain the basic solution v = 6

Set x =, x 4 =, x 5 =, we obtain the basic solution v = Set x =, x 4 =, x 5 =, we obtain the basic solution v 3 = The general solution of the system is given by In other words, the solution set is Span {v, v, v 3 } x = c v + c v + c 3 v 3, c, c, c 3, R Theorem 5 Let Ax = be a homogeneous system If u and v are solutions, then the addition and the scalar multiplication u + v, cu are also solutions solution Moreover, any linear combination of solutions for a homogeneous system is again a Theorem 53 Let Ax = be a homogeneous linear system, where A is an m n matrix with p pivot positions Then system has n p free variables and n p basic solutions The basic solutions can be obtained as follows: Setting one free variable equal to and all other free variables equal to Nonhomogeneous systems A linear system Ax = b is called nonhomogeneous if b The homogeneous linear system Ax = is called its corresponding homogeneous linear system Proposition 54 Let u and v be solutions of a nonhomogeneous system Ax = b Then the difference u v is a solution of the corresponding homogeneous system Ax = Theorem 55 (Structure of Solution Set) Let x nonh be a solution of a nonhomogeneous system Ax = b Let x hom be the general solutions of the corresponding homogeneous system Ax = Then are the general solutions of Ax = b x = x nonh + x hom Example 5 Find the solution set for the nonhomogeneous linear system x x +x 4 +x 5 = x +x x 3 4x 4 3x 5 = 3 x x +x 3 +3x 4 +x 5 = x +x +x 3 +x 4 3x 5 = 3 7

Solution Do row operations to reduce the augmented matrix to the reduced row echelon form: R + R ( )R 4 3 3 R 3 R R 3 + R 3 3 3 R 4 + R R 4 + R () [ ] () [] [ ] Then the nonhomogeneous system is equivalent to { x = +x x 4 x 5 x 3 = x 4 +x 5 The variables x, x 4, x 5 are free variables Set x = c, x 4 = c, x 5 = c 3 We obtain the general solutions x + c c c 3 x x 3 x 4 = c c + c 3 c = + c + c + c 3, x 5 c 3 or x = v + c v + c v + c 3 v 3, c, c, c 3 R, called the parametric form of the solution set In particular, setting x = x 4 = x 5 =, we obtain the particular solution v = The solution set of Ax = b is given by S = v + Span {v, v, v 3 } Example 53 The augmented matrix of the nonhomogeneous system x x +x 3 +3x 4 +3x 5 +4x 6 = x +x x 3 3x 4 4x 5 5x 6 = x +x x 5 4x 6 = 5 x +x +x 3 +3x 4 +x 5 x 6 = has the reduced row echelon form () [ ] 3 () [3] [] 3 () Then a particular solution x p for the system and the basic solutions of the corresponding homogeneous system are given by 3 x p = 3, v =, v = 3, v 3 = 8

The general solution of the system is given by x = x p + c v + c v + c 3 v 3 Example 54 For what values of a and b the linear system x +x +ax 3 = x +bx = 3x +x +x 3 = has nontrivial solutions Answer: 8a + (3a )(b 4) = 6 Linear dependence and independence Lecture 6 Definition 6 Vectors v, v,, v k in R n are said to be linearly independent provided that, whenever c v + c v + + c k v k = for some scalars c, c,, c k, then c = c = = c k = The vectors v, v,, v k are called linearly dependent if there exist constants c, c,, c k, not all zero, such that c v + c v + + c k v k = Example 6 The vectors v =, v = Solution Consider the linear system x + x 3, v 3 = + x 3 3 3 = in R 3 are linearly independent The system has the only zero solution x = x = x 3 = Thus v, v, v 3 are linearly independent Example 6 The vector v =, v =, v 3 = 3 in R 3 are linearly dependent 5 Solution Consider the linear system x 3 5 + x + x 3 3 5 3 6 = The system has one free variable There is nonzero solution Thus v, v, v 3 are linearly dependent Example 63 The vectors,,, 3 in R 3 are linearly dependent 3 9

Theorem 6 The basic solutions of any homogeneous linear system are linearly independent Proof Let v, v,, v k be basic solutions of a linear system Ax =, corresponding to free variables x j, x j,, x jk Consider the linear combination c v + c v + + c k v k = Note that the j i coordinate of c v + c v + + c k v k is c ji, i k It follows that c ji i k This means that v, v,, v k are linearly independent = for all Theorem 63 Any set of vectors containing the zero vector is linearly dependent Theorem 64 Let v, v,, v p be vectors in R n If p > n, then v, v,, v p are linearly dependent Proof Let A = [v, v,, v p ] Then A is an n p matrix, and the equation Ax = has n equations in p unknowns Recall that for the matrix A the number of pivot positions plus the number of free variables is equal to p, and the number of pivot positions is at most n Thus, if p > n, there must be some free variables Hence Ax = has nontrivial solutions This means that the column vectors of A are linearly dependent Theorem 65 Let S = {v, v,, v p } be a set of vectors in R n, (p ) Then S is linearly dependent if and only if one of vectors in S is a linear combination of the other vectors Moreover, if S is linearly dependent and v, then there is a vector v j with j such that v j is a linear combination of the preceding vectors v, v,, v j Note The condition v in the above theorem can not be deleted For example, the set { [ ] [ ]} v =, v = is linearly dependent But v is not a linear combination of v Theorem 66 The column vectors of a matrix A are linearly independent if and only if the linear system has the only zero solution Ax = Proof Let A = [a, a,, a n ] Then the linear system Ax = is the vector equation x a + x a + + x n a n = Then a, a,, a n are linear independent is equivalent to that the system has only the zero solution