4 Approximations. 4.1 Background. D. Levy

Transcription

1 D. Levy 4 Approximtions 4.1 Bckground In this chpter we re interested in pproximtion problems. Generlly speking, strting from function f(x) we would like to find different function g(x) tht belongs to given clss of functions nd is close to f(x) in some sense. As fr s the clss of functions tht g(x) belongs to, we will typiclly ssume tht g(x) is polynomil of given degree (though it cn be trigonometric function, or ny other function). A typicl pproximtion problem, will therefore be: find the closest polynomil of degree n to f(x). Wht do we men by close? There re different wys of mesuring the distnce between two functions. We will focus on two such mesurements (mong mny): the L - norm nd the L -norm. We chose to focus on these two exmples becuse of the different mthemticl techniques tht re required to solve the corresponding pproximtion problems. We strt with severl definitions. We recll tht norm on vector spce V over R is function : V R with the following properties: 1. λ f = λ f, λ R nd f V.. f 0, f V. Also f = 0 iff f is the zero element of V. 3. The tringle inequlity: f + g f + g, f, g V. We ssume tht the function f(x) C 0 [, b] (continuous on [, b]). A continuous function on closed intervl obtins mximum in the intervl. We cn therefore define the L norm (lso known s the mximum norm) of such function by f = mx f(x). (4.1) x b The L -distnce between two functions f(x), g(x) C 0 [, b] is thus given by f g = mx f(x) g(x). (4.) x b We note tht the definition of the L -norm cn be extended to functions tht re less regulr thn continuous functions. This generliztion requires some subtleties tht we would like to void in the following discussion, hence, we will limit ourselves to continuous functions. We proceed by defining the L -norm of continuous function f(x) s f = f(x) dx. (4.3) 1

2 4.1 Bckground D. Levy The L function spce is the collection of functions f(x) for which f <. Of course, we do not hve to ssume tht f(x) is continuous for the definition (4.3) to mke sense. However, if we llow f(x) to be discontinuous, we then hve to be more rigorous in terms of the definition of the intervl so tht we end up with norm (the problem is, e.g., in defining wht is the zero element in the spce). We therefore limit ourselves lso in this cse to continuous functions only. The L -distnce between two functions f(x) nd g(x) is f g = f(x) g(x) dx. (4.4) At this point, nturl question is how importnt is the choice of norm in terms of the solution of the pproximtion problem. It is esy to see tht the vlue of the norm of function my vry substntilly bsed on the function s well s the choice of the norm. For exmple, ssume tht f <. Then, clerly f = f dx (b ) f. On the other hnd, it is esy to construct function with n rbitrry smll f nd n rbitrrily lrge f. Hence, the choice of norm my hve significnt impct on the solution of the pproximtion problem. As you hve probbly lredy nticipted, there is strong connection between some pproximtion problems nd interpoltion problems. For exmple, one possible method of constructing n pproximtion to given function is by smpling it t certin points nd then interpolting the smpled dt. Is tht the best we cn do? Sometimes the nswer is positive, but the problem still remins difficult becuse we hve to determine the best smpling points. We will ddress these issues in the following sections. The following theorem, the Weierstrss pproximtion theorem, plys centrl role in ny discussion of pproximtions of functions. Loosely speking, this theorem sttes tht ny continuous function cn be pproched s close s we wnt to with polynomils, ssuming tht the polynomils cn be of ny degree. We formulte this theorem in the L norm nd note tht similr theorem holds lso in the L sense. We let Π n denote the spce of polynomils of degree n. Theorem 4.1 (Weierstrss Approximtion Theorem) Let f(x) be continuous function on [, b]. Then there exists sequence of polynomils P n (x) tht converges uniformly to f(x) on [, b], i.e., ε > 0, there exists n N N nd polynomils P n (x) Π n, such tht x [, b] f(x) P n (x) < ε, n N.

3 D. Levy 4.1 Bckground We will provide constructive proof of the Weierstrss pproximtion theorem: first, we will define fmily of polynomils, known s the Bernstein polynomils, nd then we will show tht they uniformly converge to f(x). We strt with the definition. Given continuous function f(x) in [0, 1], we define the Bernstein polynomils s (B n f)(x) = f ( j n ) ( ) n x j (1 x) n j, 0 x 1. j We emphsize tht the Bernstein polynomils depend on the function f(x). Exmple 4. Three Bernstein polynomils B 6 (x), B 10 (x), nd B 0 (x) for the function f(x) = (x 0.5) on the intervl [0, 1] re shown in Figure 4.1. Note the grdul convergence of B n (x) to f(x) f(x) B 6 (x) B 10 (x) B 0 (x) x Figure 4.1: The Bernstein polynomils B 6 (x), B 10 (x), nd B 0 (x) for the function f(x) = (x 0.5) on the intervl [0, 1] We now stte nd prove severl properties of B n (x) tht will be used when we prove Theorem

4 4.1 Bckground D. Levy Lemm 4.3 The following reltions hold: 1. (B n 1)(x) = 1. (B n x)(x) = x 3. (B n x )(x) = n 1 n x + x n. Proof. (B n 1)(x) = (B n x)(x) = Finlly, ( ) ( ) j n n j Hence (B n x )(x) = ( ) n x j (1 x) n j = (x + (1 x)) n = 1. j ( ) j n x j (1 x) n j = x n j j=1 ( ) n 1 x j 1 (1 x) n j j 1 n 1 ( ) n 1 = x x j (1 x) n 1 j = x[x + (1 x)] n 1 = x. j = j (n 1)! n (n j)!(j 1)! = n 1 j 1 (n 1)! n 1 n (n j)!(j 1)! + 1 (n 1)! n (n j)!(j 1)! = n 1 ( ) n + 1 ( ) n 1. n j n j 1 ( j n = n 1 n x ) ( ) n x j (1 x) n j j j= ( ) n x j (1 x) n j + 1 j n x j=1 ( ) n 1 x j 1 (1 x) n j j 1 = n 1 n x (x + (1 x)) n + 1 n x(x + (1 x))n 1 = n 1 n x + x n. In the following lemm we stte severl dditionl properties of the Bernstein polynomils. The proof is left s n exercise. Lemm 4.4 For ll functions f(x), g(x) tht re continuous in [0, 1], nd α R 1. Linerity. (B n (αf + g))(x) = α(b n f)(x) + (B n g)(x). 4

5 D. Levy 4.1 Bckground. Monotonicity. If f(x) g(x) x [0, 1], then (B n f)(x) (B n g)(x). Also, if f(x) g(x) x [0, 1] then (B n f)(x) (B n g)(x). 3. Positivity. If f(x) 0 then (B n f)(x) 0. We re now redy to prove the Weierstrss pproximtion theorem, Theorem 4.1. Proof. We will prove the theorem in the intervl [0, 1]. The extension to [, b] is left s n exercise. Since f(x) is continuous on closed intervl, it is uniformly continuous. Hence x, y [0, 1], such tht x y δ, f(x) f(y) ε. (4.5) In ddition, since f(x) is continuous on closed intervl, it is lso bounded. Let M = mx f(x). x [0,1] Fix ny point [0, 1]. If x δ then (4.5) holds. If x > δ then ( ) x f(x) f() M M. δ (t first sight this seems to be strnge wy of upper bounding function. We will use it lter on to our dvntge). Combining the estimtes for both cses we hve f(x) f() ε + M δ (x ). We would now like to estimte the difference between B n f nd f. The linerity of B n nd the property (B n 1)(x) = 1 imply tht B n (f f())(x) = (B n f)(x) f(). Hence using the monotonicity of B n nd the mpping properties of x nd x, we hve, ( B n f(x) f() B n ε + M ) δ (x ) = ε + M ( n 1 δ n x + x ) x + n = ε + M δ (x ) + M x x. δ n 5

6 4. The Minimx Approximtion Problem D. Levy Evluting t x = we hve (observing tht mx [0,1] ( ) = 1 4 ) B n f() f() ε + M δ n ε + M δ n. (4.6) The point ws rbitrry so the result (4.6) holds for ny point [0, 1]. Choosing N M we hve n N, δ ε B n f f ε + M δ N ε. Is interpoltion good wy of pproximting functions in the -norm? necessrily. Discuss Runge s exmple... Not 4. The Minimx Approximtion Problem We ssume tht the function f(x) is continuous on [, b], nd ssume tht P n (x) is polynomil of degree n. We recll tht the L -distnce between f(x) nd P n (x) on the intervl [, b] is given by f P n = mx x b f(x) P n(x). (4.7) Clerly, we cn construct polynomils tht will hve n rbitrry lrge distnce from f(x). The question we would like to ddress is how close cn we get to f(x) (in the L sense) with polynomils of given degree. We define d n (f) s the infimum of (4.7) over ll polynomils of degree n, i.e., d n (f) = inf f P n (4.8) P n Π n The gol is to find polynomil P n(x) for which the infimum (4.8) is ctully obtined, i.e., d n (f) = f P n(x). (4.9) We will refer to polynomil Pn(x) tht stisfies (4.9) s polynomil of best pproximtion or the minimx polynomil. The miniml distnce in (4.9) will be referred to s the minimx error. The theory we will explore in the following sections will show tht the minimx polynomil lwys exists nd is unique. We will lso provide chrcteriztion of the minimx polynomil tht will llow us to identify it if we ctully see it. The generl construction of the minimx polynomil will not be ddressed in this text s it is reltively techniclly involved. We will limit ourselves to simple exmples. 6

7 D. Levy 4. The Minimx Approximtion Problem Exmple 4.5 We let f(x) be monotoniclly incresing nd continuous function on the intervl [, b] nd re interested in finding the minimx polynomil of degree zero to f(x) in tht intervl. We denote this minimx polynomil by P 0 (x) c. Clerly, the smllest distnce between f(x) nd P 0 in the L -norm will be obtined if c = f() + f(b). The mximl distnce between f(x) nd P0 equl to will be ttined t both edges nd will be ± f(b) f() Existence of the minimx polynomil The existence of the minimx polynomil is provided by the following theorem. Theorem 4.6 (Existence) Let f C 0 [, b]. Then for ny n N there exists P n(x) Π n, tht minimizes f(x) P n (x) mong ll polynomils P (x) Π n. Proof. We follow the proof s given in [?]. Let η = (η 0,..., η n ) be n rbitrry point in R n+1 nd let P n (x) = We lso let η i x i Π n. i=0 φ(η) = φ(η 0,..., η n ) = f P n. Our gol is to show tht φ obtins minimum in R n+1, i.e., tht there exists point η = (η 0,..., η n) such tht φ(η ) = min φ(η). η Rn+1 Step 1. We first show tht φ(η) is continuous function on R n+1. For n rbitrry δ = (δ 0,..., δ n ) R n+1, define q n (x) = δ i x i. i=0 7

8 4. The Minimx Approximtion Problem D. Levy Then Hence φ(η + δ) = f (P n + q n ) f P n + q n = φ(η) + q n. φ(η + δ) φ(η) q n mx ( δ 0 + δ 1 x δ n x n ). x [,b] For ny ε > 0, let δ = ε/(1 + c c n ), where c = mx(, b ). Then for ny δ = (δ 0,..., δ n ) such tht mx δ i δ, 0 i n, Similrly φ(η + δ) φ(η) ε. (4.10) φ(η) = f P n = f (P n +q n )+q n f (P n +q n ) + q n = φ(η+δ)+ q n, which implies tht under the sme conditions s in (4.10) we lso get Altogether, φ(η) φ(η + δ) ε, φ(η + δ) φ(η) ε, which mens tht φ is continuous t η. Since η ws n rbitrry point in R n+1, φ is continuous in the entire R n+1. Step. We now construct compct set in R n+1 on which φ obtins minimum. We let We hve S = { η R n+1 φ(η) f }. φ(0) = f, hence, 0 S, nd the set S is nonempty. We lso note tht the set S is bounded nd closed (check!). Since φ is continuous on the entire R n+1, it is lso continuous on S, nd hence it must obtin minimum on S, sy t η R n+1, i.e., min φ(η) = η S φ(η ). 8

9 D. Levy 4. The Minimx Approximtion Problem Step 3. Since 0 S, we know tht min φ(η) φ(0) = f. η S Hence, if η R n+1 but η S then φ(η) > f min η S φ(η). This mens tht the minimum of φ over S is the sme s the minimum over the entire R n+1. Therefore Pn(x) = ηi x i, (4.11) i=0 is the best pproximtion of f(x) in the L norm on [, b], i.e., it is the minimx polynomil, nd hence the minimx polynomil exists. We note tht the proof of Theorem 4.6 is not constructive proof. The proof does not tell us wht the point η is, nd hence, we do not know the coefficients of the minimx polynomil s written in (4.11). We will discuss the chrcteriztion of the minimx polynomil nd some simple cses of its construction in the following sections. 4.. Bounds on the minimx error It is trivil to obtin n upper bound on the minimx error, since by the definition of d n (f) in (4.8) we hve d n (f) f P n, P n (x) Π n. A lower bound is provided by the following theorem. Theorem 4.7 (de l Vllée-Poussin) Let x 0 < x 1 < < x n+1 b. Let P n (x) be polynomil of degree n. Suppose tht f(x j ) P n (x j ) = ( 1) j e j, j = 0,..., n + 1, where ll e j 0 nd re of n identicl sign. Then min e j d n (f). j Proof. By contrdiction. Assume for some Q n (x) tht f Q n < min j e j. Then the polynomil (Q n P n )(x) = (f P n )(x) (f Q n )(x), is polynomil of degree n tht hs the sme sign t x j s does f(x) P n (x). This implies tht (Q n P n )(x) chnges sign t lest n + times, nd hence it hs t lest n + 1 zeros. Being polynomil of degree n this is possible only if it is identiclly zero, i.e., if P n (x) Q n (x), which contrdicts the ssumptions on Q n (x) nd P n (x). 9

10 4. The Minimx Approximtion Problem D. Levy 4..3 Chrcteriztion of the minimx polynomil The following theorem provides chrcteriztion of the minimx polynomil in terms of its oscilltions property. Theorem 4.8 (The oscillting theorem) Suppose tht f(x) is continuous in [, b]. The polynomil P n(x) Π n is the minimx polynomil of degree n to f(x) in [, b] if nd only if f(x) P n(x) ssumes the vlues ± f P n with n lternting chnge of sign t lest n + times in [, b]. Proof. We prove here only the sufficiency prt of the theorem. For the necessry prt of the theorem we refer to [?]. Without loss of generlity, suppose tht Let nd let (f P n)(x i ) = ( 1) i f P n, 0 i n + 1. D = f P n, d n (f) = min P n Π n f P n. We replce the infimum in the originl definition of d n (f) by minimum becuse we lredy know tht minimum exists. de l Vllée-Poussin s theorem (Theorem 4.7) implies tht D d n. On the other hnd, the definition of d n implies tht d n D. Hence D = d n nd P n(x) is the minimx polynomil. Remrk. In view of these theorems it is obvious why the Tylor expnsion is poor uniform pproximtion. The sum is non oscilltory Uniqueness of the minimx polynomil Theorem 4.9 (Uniqueness) Let f(x) be continuous on [, b]. Then its minimx polynomil P n(x) Π n is unique. Proof. Let d n (f) = min P n Π n f P n. Assume tht Q n (x) is lso minimx polynomil. Then f P n = f Q n = d n (f). 10

11 D. Levy 4. The Minimx Approximtion Problem The tringle inequlity implies tht f 1 (P n + Q n ) 1 f P n + 1 f Q n = d n (f). Hence, 1 (P n + Q n ) Π n is lso minimx polynomil. The oscillting theorem (Theorem 4.8) implies tht there exist x 0,..., x n+1 [, b] such tht f(x i ) 1 (P n(x i ) + Q n (x i )) = d n (f), 0 i n + 1. (4.1) Eqution (4.1) cn be rewritten s f(x i ) P n(x i ) + f(x i ) Q n (x i ) = d n (f), 0 i n + 1. (4.13) Since P n(x) nd Q n (x) re both minimx polynomils, we hve nd f(x i ) P n(x i ) f P n = d n (f), 0 i n + 1. (4.14) f(x i ) Q n (x i ) f Q n = d n (f), 0 i n + 1. (4.15) For ny i, equtions (4.13) (4.15) men tht the bsolute vlue of two numbers tht re d n (f) dd up to d n (f). This is possible only if they re equl to ech other, i.e., i.e., f(x i ) P n(x i ) = f(x i ) Q n (x i ), 0 i n + 1, (P n Q n )(x i ) = 0, 0 i n + 1. Hence, the polynomil (P n Q n )(x) Π n hs n + distinct roots which is possible for polynomil of degree n only if it is identiclly zero. Hence Q n (x) P n(x), nd the uniqueness of the minimx polynomil is estblished The ner-minimx polynomil We now connect between the minimx pproximtion problem nd polynomil interpoltion. In order for f(x) P n (x) to chnge its sign n + times, there should be n + 1 points on which f(x) nd P n (x) gree with ech other. In other words, we cn think of P n (x) s function tht interpoltes f(x) t (lest in) n + 1 points, sy x 0,..., x n. Wht cn we sy bout these points? 11

12 4. The Minimx Approximtion Problem D. Levy We recll tht the interpoltion error is given by (??), f(x) P n (x) = 1 (n + 1)! f (n+1) (ξ) n (x x i ). If P n (x) is indeed the minimx polynomil, we know tht the mximum of f (n+1) (ξ) i=0 n (x x i ), (4.16) i=0 will oscillte with equl vlues. Due to the dependency of f (n+1) (ξ) on the intermedite point ξ, we know tht minimizing the error term (4.16) is difficult tsk. We recll tht interpoltion t the Chebyshev points minimizes the multiplictive prt of the error term, i.e., n (x x i ). i=0 Hence, choosing x 0,..., x n to be the Chebyshev points will not result with the minimx polynomil, but nevertheless, this reltion motivtes us to refer to the interpolnt t the Chebyshev points s the ner-minimx polynomil. We note tht the term nerminimx does not men tht the ner-minimx polynomil is ctully close to the minimx polynomil Construction of the minimx polynomil The chrcteriztion of the minimx polynomil in terms of the number of points in which the mximum distnce should be obtined with oscillting signs llows us to construct the minimx polynomil in simple cses by direct computtion. We re not going to del with the construction of the minimx polynomil in the generl cse. The lgorithm for doing so is known s the Remez lgorithm, nd we refer the interested reder to [?] nd the references therein. A simple cse where we cn demonstrte direct construction of the polynomil is when the function is convex, s done in the following exmple. Exmple 4.10 Problem: Let f(x) = e x, x [1, 3]. Find the minimx polynomil of degree 1, P 1 (x). Solution: Bsed on the chrcteriztion of the minimx polynomil, we will be looking for liner function P 1 (x) such tht its mximl distnce between P 1 (x) nd f(x) is obtined 3 times with lterntive signs. Clerly, in the cse of the present problem, since the function is convex, the mximl distnce will be obtined t both edges nd t one interior point. We will use this observtion in the construction tht follows. The construction itself is grphiclly shown in Figure 4.. 1

13 D. Levy 4. The Minimx Approximtion Problem e 3 l 1 (x) e x l 1 () ȳ f() P 1 (x) e 1 l (x) 1 3 x Figure 4.: A construction of the liner minimx polynomil for the convex function e x on [1, 3] We let l 1 (x) denote the line tht connects the endpoints (1, e) nd (3, e 3 ), i.e., l 1 (x) = e + m(x 1). Here, the slope m is given by m = e3 e. (4.17) Let l (x) denote the tngent to f(x) t point tht is identified such tht the slope is m. Since f (x) = e x, we hve e = m, i.e., Now nd = log m. f() = e log m = m, l 1 () = e + m(log m 1). Hence, the verge between f() nd l 1 () which we denote by ȳ is given by ȳ = f() + l 1() = m + e + m log m m 13 = e + m log m.

14 4.3 Lest-squres Approximtions D. Levy The minimx polynomil P 1 (x) is the line of slope m tht psses through (, ȳ), i.e., P 1 (x) e + m log m = m(x log m), P1 (x) = mx + e m log m, where the slope m is given by (4.17). We note tht the mximl difference between P 1 (x) nd f(x) is obtined t x = 1,, Lest-squres Approximtions The lest-squres pproximtion problem We recll tht the L -norm of function f(x) is defined s f = f(x) dx. As before, we let Π n denote the spce of ll polynomils of degree n. The lestsqures pproximtion problem is to find the polynomil tht is the closest to f(x) in the L -norm mong ll polynomils of degree n, i.e., to find Q n Π n such tht f Q n = min Q n Π n f Q n Solving the lest-squres problem: direct method Our gol is to find polynomil in Π n tht minimizes the distnce f(x) Q n (x) mong ll polynomils Q n Π n. We thus consider Q n (x) = i x i. i=0 For convenience, insted of minimizing the L norm of the difference, we will minimize its squre. We thus let φ denote the squre of the L -distnce between f(x) nd Q n (x), i.e., φ( 0,..., n ) = = (f(x) Q n (x)) dx f (x)dx i=0 i x i f(x)dx + i j x i+j dx. i=0 14

15 D. Levy 4.3 Lest-squres Approximtions φ is function of the n + 1 coefficients in the polynomil Q n (x). This mens tht we wnt to find point = ( 0,..., n) R n+1 for which φ obtins minimum. At this point φ k = 0. (4.18) = The condition (4.18) implies tht 0 = = [ i=0 x k f(x)dx + i x i+k dx i i=0 x i+k dx + x k f(x)dx This is liner system for the unknowns ( 0,..., n): i=0 i x i+k dx = ]. j x j+k dx (4.19) x k f(x), k = 0,..., n. (4.0) We thus know tht the solution of the lest-squres problem is the polynomil Q n(x) = i x i, i=0 where the coefficients i, i = 0,..., n, re the solution of (4.0), ssuming tht this system cn be solved. Indeed, the system (4.0) lwys hs unique solution, which proves tht not only the lest-squres problem hs solution, but tht it is lso unique. We let H n+1 (, b) denote the (n+1) (n+1) coefficients mtrix of the system (4.0) on the intervl [, b], i.e., (H n+1 (, b)) i,k = x i+k dx, 0 i, k n. For exmple, in the cse where [, b] = [0, 1], 1/1 1/... 1/n 1/ 1/3... 1/(n + 1) H n (0, 1) =.. 1/n 1/(n + 1)... 1/(n 1) (4.1) The mtrix (4.1) is known s the Hilbert mtrix. Lemm 4.11 The Hilbert mtrix is invertible. 15

16 4.3 Lest-squres Approximtions D. Levy Proof. We leve it is n exercise to show tht the determinnt of H n is given by det(h n ) = (1!! (n 1)!)4 1!! (n 1)!. Hence, det(h n ) 0 nd H n is invertible. Is inverting the Hilbert mtrix good wy of solving the lest-squres problem? No. There re numericl instbilities tht re ssocited with inverting H. We demonstrte this with the following exmple. Exmple 4.1 The Hilbert mtrix H 5 is 1/1 1/ 1/3 1/4 1/5 1/ 1/3 1/4 1/5 1/6 H 5 = 1/3 1/4 1/5 1/6 1/7 1/4 1/5 1/6 1/7 1/8 1/5 1/6 1/7 1/8 1/9 The inverse of H 5 is H 5 = The condition number of H 5 is , which indictes tht it is ill-conditioned. In fct, the condition number of H n increses with the dimension n so inverting it becomes more difficult with n incresing dimension Solving the lest-squres problem: with orthogonl polynomils Let {P k } n k=0 be polynomils such tht deg(p k (x)) = k. Let Q n (x) be liner combintion of the polynomils {P k } n k=0, i.e., Q n (x) = c j P j (x). (4.) Clerly, Q n (x) is polynomil of degree n. Define φ(c 0,..., c n ) = [f(x) Q n (x)] dx. 16

17 D. Levy 4.3 Lest-squres Approximtions We note tht the function φ is qudrtic function of the coefficients of the liner combintion (4.), {c k }. We would like to minimize φ. Similrly to the clcultions done in the previous section, t the minimum, c = (c 0,..., c n), we hve 0 = φ c k = P k (x)f(x)dx + c j P j (x)p k (x)dx, c=c i.e., c j P j (x)p k (x)dx = P k (x)f(x)dx, k = 0,..., n. (4.3) Note the similrity between eqution (4.3) nd (4.0). There, we used the bsis functions {x k } n k=0 ( bsis of Π n), while here we work with the polynomils {P k (x)} n k=0 insted. The ide now is to choose the polynomils {P k (x)} n k=0 such tht the system (4.3) cn be esily solved. This cn be done if we choose them in such wy tht { 1, i = j, P i (x)p j (x)dx = δ ij = (4.4) 0, j j. Polynomils tht stisfy (4.4) re clled orthonorml polynomils. If, indeed, the polynomils {P k (x)} n k=0 re orthonorml, then (4.3) implies tht c j = P j (x)f(x)dx, j = 0,..., n. (4.5) The solution of the lest-squres problem is polynomil Q n(x) = c jp j (x), (4.6) with coefficients c j, j = 0,..., n, tht re given by (4.5). Remrk. Polynomils tht stisfy b (P i(x)), i = j, P i (x)p j (x)dx = 0, i j, with (P i(x)) dx tht is not necessrily 1 re clled orthogonl polynomils. In this cse, the solution of the lest-squres problem is given by the polynomil Q n(x) in (4.6) with the coefficients c j = P j(x)f(x)dx (P, j = 0,..., n. (4.7) j(x)) dx 17

18 4.3 Lest-squres Approximtions D. Levy The weighted lest squres problem A more generl lest-squres problem is the weighted lest squres pproximtion problem. We consider weight function, w(x), to be continuous on (, b), nonnegtive function with positive mss, i.e., w(x)dx > 0. Note tht w(x) my be singulr t the edges of the intervl since we do not require it to be continuous on the closed intervl [, b]. For ny weight w(x), we define the corresponding weighted L -norm of function f(x) s f,w = (f(x)) w(x)dx. The weighted lest-squres problem is to find the closest polynomil Q n Π n to f(x), this time in the weighted L -norm sense, i.e., we look for polynomil Q n(x) of degree n such tht f Q n,w = min Q n Π n f Q n,w. (4.8) In order to solve the weighted lest-squres problem (4.8) we follow the methodology described in Section 4.3.3, nd consider polynomils {P k } n k=0 such tht deg(p k(x)) = k. We then consider polynomil Q n (x) tht is written s their liner combintion: Q n (x) = c j P j (x). By repeting the clcultions of Section 4.3.3, we obtin for the coefficients of the minimizer Q n, c j w(x)p j (x)p k (x)dx = w(x)p k (x)f(x)dx, k = 0,..., n, (4.9) (compre with (4.3)). The system (4.9) cn be esily solved if we choose {P k (x)} to be orthonorml with respect to the weight w(x), i.e., P i (x)p j (x)w(x)dx = δ ij. Hence, the solution of the weighted lest-squres problem is given by Q n(x) = c jp j (x), (4.30) 18

19 D. Levy 4.3 Lest-squres Approximtions where the coefficients re given by c j = P j (x)f(x)w(x)dx, j = 0,..., n. (4.31) Remrk. In the cse where {P k (x)} re orthogonl but not necessrily normlized, the solution of the weighted lest-squres problem is given by Q n(x) = c jp j (x) with c j = P j(x)f(x)w(x)dx (P, j = 0,..., n. j(x)) w(x)dx Orthogonl polynomils At this point we lredy know tht orthogonl polynomils ply centrl role in the solution of lest-squres problems. In this section we will focus on the construction of orthogonl polynomils. The properties of orthogonl polynomils will be studies in Section We strt by defining the weighted inner product between two functions f(x) nd g(x) (with respect to the weight w(x)): f, g w = f(x)g(x)w(x)dx. To simplify the nottions, even in the weighted cse, we will typiclly write f, g insted of f, g w. Some properties of the weighted inner product include 1. αf, g = f, αg = α f, g, α R.. f 1 + f, g = f 1, g + f, g. 3. f, g = g, f 4. f, f 0 nd f, f = 0 iff f 0. Here we must ssume tht f(x) is continuous in the intervl [, b]. If it is not continuous, we cn hve f, f = 0 nd f(x) cn still be non-zero (e.g., in one point). The weighted L -norm cn be obtined from the weighted inner product by f,w = f, f w. 19

20 4.3 Lest-squres Approximtions D. Levy Given weight w(x), we re interested in constructing orthogonl (or orthonorml) polynomils. This cn be done using the Grm-Schmidt orthogonliztion process, which we now describe in detil. In the generl context of liner lgebr, the Grm-Schmidt process is being used to convert one set of linerly independent vectors to n orthogonl set of vectors tht spns the sme subspce s the originl set. In our context, we should think bout the process s converting one set of polynomils tht spn the spce of polynomils of degree n to n orthogonl set of polynomils tht spns the sme spce Π n. Accordingly, we set the initil set of polynomils s {1, x, x,..., x n }, which we would like to convert to orthogonl polynomils (of n incresing degree) with respect to the weight w(x). We will first demonstrte the process with the weight w(x) 1. We will generte set of orthogonl polynomils {P 0 (x),..., P n (x)} from {1, x,..., x n }. The degree of the polynomils P i is i. We strt by setting P 0 (x) = 1. We then let P 1 (x) = x c 0 1P 0 (x) = x c 0 1. The orthogonlity condition P 1P 0 dx = 0, implies tht 1 (x c 0 1)dx = 0, from which c= (+b)/, nd thus P 1 (x) = x + b. The computtion continues in similr fshion. We set P (x) = x c 0 P 0 (x) c 1 (x). The two unknown coefficients, c 0 nd c 1, re computed from the orthogonlity conditions. This time, P (x) should be orthogonl to P 0 (x) nd to P 1 (x), i.e., P (x)p 0 (x)dx = 0, nd P (x)p 1 (x)dx = 0, nd so on. If, in ddition to the orthogonlity condition, we would like the polynomils to be orthonorml, ll tht remins is to normlize: ˆP n (x) = P n(x) P n (x) = P n (x), n. b (P n(x)) dx The orthogonliztion process is identicl to the process tht we described even when the weight w(x) is not uniformly one. In this cse, every integrl will contin the weight. 0

21 D. Levy 4.4 Exmples of orthogonl polynomils 4.4 Exmples of orthogonl polynomils This section includes severl exmples of orthogonl polynomils nd very brief summry of some of their properties. 1. Legendre polynomils. We strt with the Legendre polynomils. This is fmily of polynomils tht re orthogonl with respect to the weight w(x) 1, on the intervl [ 1, 1]. In ddition to deriving the Legendre polynomils through the Grm-Schmidt orthogonliztion process, it cn be shown tht the Legendre polynomils cn be obtined from the recurrence reltion (n + 1)P n+1 (x) (n + 1)xP n (x) + np n 1 (x) = 0, n 1, (4.3) strting from the first two polynomils: P 0 (x) = 1, P 1 (x) = x. Insted of clculting these polynomils one by one from the recurrence reltion, they cn be obtined directly from Rodrigues formul d n P n (x) = 1 [ (x 1) n], n 0. (4.33) n n! dx n The Legendre polynomils stisfy the orthogonlity condition P n, P m = n + 1 δ nm. (4.34). Chebyshev polynomils. Our second exmple is of the Chebyshev polynomils. These polynomils re orthogonl with respect to the weight w(x) = 1 1 x, on the intervl [ 1, 1]. They stisfy the recurrence reltion T n+1 (x) = xt n (x) T n 1 (x), n 1, (4.35) together with T 0 (x) = 1 nd T 1 (x) = x (see (??)). They nd re explicitly given by T n (x) = cos(n cos 1 x), n 0. (4.36) 1

22 4.4 Exmples of orthogonl polynomils D. Levy (see (??)). The orthogonlity reltion tht they stisfy is 0, n m, T n, T m = π, n = m = 0, π, n = m 0. (4.37) 3. Lguerre polynomils. We proceed with the Lguerre polynomils. Here the intervl is given by [0, ) with the weight function w(x) = e x. The Lguerre polynomils re given by L n (x) = ex d n n! dx n (xn e x ), n 0. (4.38) The normliztion condition is L n = 1. (4.39) A more generl form of the Lguerre polynomils is obtined when the weight is tken s e x x α, for n rbitrry rel α > 1, on the intervl [0, ). 4. Hermite polynomils. The Hermite polynomils re orthogonl with respect to the weight w(x) = e x, on the intervl (, ). The cn be explicitly written s H n (x) = ( 1) n e x dn e x Another wy of expressing them is by H n (x) = [n/] k=0, n 0. (4.40) dxn ( 1) k n! k!(n k)! (x)n k, (4.41) where [x] denotes the lrgest integer tht is x. The Hermite polynomils stisfy the recurrence reltion H n+1 (x) xh n (x) + nh n 1 (x) = 0, n 1, (4.4)

23 D. Levy 4.4 Exmples of orthogonl polynomils together with H 0 (x) = 1, H 1 (x) = x. They stisfy the orthogonlity reltion e x H n (x)h m (x)dx = n n! πδ nm. (4.43) Another pproch to the lest-squres problem In this section we present yet nother wy of deriving the solution of the lest-squres problem. Along the wy, we will be ble to derive some new results. We recll tht our gol is to minimize f(x) Q n (x),w, Q n Π n, i.e., to minimize the integrl w(x)(f(x) Q n (x)) dx (4.44) mong ll the polynomils Q n (x) of degree n. The minimizer of (4.44) is denoted by Q n(x). Assume tht {P k (x)} k 0 is n orthogonl fmily of polynomils with respect to w(x), nd let Q n (x) = c j P j (x). Then f Q n,w = w(x) ( f(x) c j P j (x)) dx. Hence 0 f c j P j, f = f,w = f,w c j P j c j f, P j w + f, P j w P j,w + The lst expression is miniml iff w = f, f w c j P j,w c j f, P j w + ( ) f, Pj w c j P j,w. P j,w c i c j P i, P j w i=0 f, P j w P j,w c j P j,w = 0, 0 j n, 3

24 4.4 Exmples of orthogonl polynomils D. Levy i.e., if c j = f, P j w. P j,w Hence, there exists unique lest-squres pproximtion which is given by Q n(x) = f, P j w P P j j (x). (4.45),w If the polynomils {P j (x)} re lso normlized so tht P j,w = 1, then the minimizer Q n(x) in (4.45) becomes Q n(x) = f, P j w P j (x). Remrks. 1. We cn write f Q n,w = w(x) ( = f,w f(x) Since P j,w = 1, c j = f, P j w, so tht f Q n,w = f,w Hence c j P j (x)) dx = f, P j w c j + f, P j w. f, P j w = f f Q n f, P j,wc j. i.e., f, P j w f,w. (4.46) The inequlity (4.46) is clled Bessel s inequlity. 4

25 D. Levy 4.4 Exmples of orthogonl polynomils. Assuming tht [, b] is finite, we hve lim f n Q n,w = 0. Hence f,w = f, P j w, (4.47) which is known s Prsevl s equlity. Exmple 4.13 Problem: Let f(x) = cos x on [ 1, 1]. Find the polynomil in Π, tht minimizes 1 1 [f(x) Q (x)] dx. Solution: The weight w(x) 1 on [ 1, 1] implies tht the orthogonl polynomils we need to use re the Legendre polynomils. We re seeking for polynomils of degree so we write the first three Legendre polynomils P 0 (x) 1, P 1 (x) = x, P (x) = 1 (3x 1). The normliztion fctor stisfies, in generl, 1 1 Hence 1 1 P n(x) = n + 1. P 0 (x)dx =, 1 1 P 1 (x)dx = 1 3, P (x)dx = 5. We cn then replce the Legendre polynomils by their normlized counterprts: P 0 (x) 1 3 5, P 1 (x) = x, P (x) = (3x 1). We now hve Hence f, P 0 = 1 1 Q 0(x) sin 1. cos x 1 dx = 1 sin x = sin 1. 5

26 4.4 Exmples of orthogonl polynomils D. Levy We lso hve f, P 1 = cos x xdx = 0. which mens tht Q 1(x) = Q 0(x). Finlly, f, P = cos x 3x 1 = 1 5 (1 cos 1 8 sin 1), nd hence the desired polynomil, Q (x), is given by ( ) 15 Q (x) = sin 1 + cos 1 5 sin 1 (3x 1). In Figure 4.3 we plot the originl function f(x) = cos x (solid line) nd its pproximtion Q (x) (dshed line). We zoom on the intervl x [0, 1] x Figure 4.3: A second-order L -pproximtion of f(x) = cos x. Solid line: f(x); Dshed line: its pproximtion Q (x) If the weight is w(x) 1 but the intervl is [, b], we cn still use the Legendre polynomils if we mke the following chnge of vribles. Define x = b + + (b )t. 6

27 D. Levy 4.4 Exmples of orthogonl polynomils Then the intervl 1 t 1 is mpped to x b. Now, define ( ) b + + (b )t F (t) = f = f(x). Hence [f(x) Q n (x)] dx = b 1 1 [F (t) q n (t)] dt. Exmple 4.14 Problem: Let f(x) = cos x on [0, π]. Find the polynomil in Π 1 tht minimizes π 0 Solution: Letting π 0 [f(x) Q 1 (x)] dx. (f(x) Q 1(x)) dx = π x = π + πt = π (1 + t), we hve ( π ) F (t) = cos (1 + t) 1 1 = sin πt. [F (t) q n (t)] dt. We lredy know tht the first two normlized Legendre polynomils re P 0 (t) = 1 3, P 1 (t) = t. Hence F, P 0 = sin πt dt = 0, which mens tht Q 0(t) = 0. Also 1 F, P 1 = sin πt tdt = Hence q 1(t) = 3 [ sin πt ) ( π ] 1 πt t cos 3 π = 1 8 π t = 1 π t = Q 1(x) = 1 ( ) π π x 1. In Figure 4.4 we plot the originl function f(x) = cos x (solid line) nd its pproximtion Q 1(x) (dshed line). 8 π. 7

28 4.4 Exmples of orthogonl polynomils D. Levy x Figure 4.4: A first-order L -pproximtion of f(x) = cos x on the intervl [0, π]. Solid line: f(x), Dshed line: its pproximtion Q 1(x) Exmple 4.15 Problem: Let f(x) = cos x in [0, ). Find the polynomil in Π 1 tht minimizes 0 e x [f(x) Q 1 (x)] dx. Solution: The fmily of orthogonl polynomils tht correspond to this weight on [0, ) re Lguerre polynomils. Since we re looking for the minimizer of the weighted L norm mong polynomils of degree 1, we will need to use the first two Lguerre polynomils: L 0 (x) = 1, L 1 (x) = 1 x. We thus hve Also f, L 0 w = f, L 1 w = 0 This mens tht 0 e x cos xdx = e x ( cos x + sin x) 0 = 1. e x cos x(1 x)dx = 1 [ xe x ( cos x + sin x) Q 1(x) = f, L 0 w L 0 (x) + f, L 1 w L 1 (x) = 1. ] e x ( sin x) =

29 D. Levy 4.4 Exmples of orthogonl polynomils 4.4. Properties of orthogonl polynomils We strt with theorem tht dels with some of the properties of the roots of orthogonl polynomils. This theorem will become hndy when we discuss Gussin qudrtures in Section??. We let {P n (x)} n 0 be orthogonl polynomils in [, b] with respect to the weight w(x). Theorem 4.16 The roots x j, j = 1,..., n of P n (x) re ll rel, simple, nd re in (, b). Proof. Let x 1,..., x r be the roots of P n (x) in (, b). Let Q r (x) = (x x 1 )... (x x r ). Then Q r (x) nd P n (x) chnge their signs together in (, b). Also deg(q r (x)) = r n. Hence (P n Q r )(x) is polynomil with one sign in (, b). This implies tht P n (x)q r (x)w(x)dx 0, nd hence r = n since P n (x) is orthogonl to polynomils of degree less thn n. Without loss of generlity we now ssume tht x 1 is multiple root, i.e., P n (x) = (x x 1 ) P n (x). Hence ( ) Pn (x) P n (x)p n (x) = 0, x x 1 which implies tht P n (x)p n (x)dx > 0. This is not possible since P n is orthogonl to P n. Hence roots cn not repet. Another importnt property of orthogonl polynomils is tht they cn ll be written in terms of recursion reltions. We hve lredy seen specific exmples of such reltions for the Legendre, Chebyshev, nd Hermite polynomils (see (4.3), (4.35), nd (4.4)). The following theorem sttes such reltions lwys hold. 9

30 4.4 Exmples of orthogonl polynomils D. Levy Theorem 4.17 (Triple Recursion Reltion) Any three consecutive orthonorml polynomils re relted by recursion formul of the form P n+1 (x) = (A n x + B n )P n (x) C n P n 1 (x). If k nd b k re the coefficients of the terms of degree k nd degree k 1 in P k (x), then A n = n+1, B n = ( n+1 bn+1 b ) n, C n = n+1 n 1. n n n+1 n n Proof. For let A n = n+1 n, Q n (x) = P n+1 (x) A n xp n (x) = ( n+1 x n+1 + b n+1 x n +...) n+1 x( n x n + b n x n ) ( n = b n+1 ) n+1b n x n +... n Hence deg(q(x)) n, which mens tht there exists α 0,..., α n such tht Q(x) = α i P i (x). i=0 For 0 i n, Hence α i = Q, P i P i, P i = Q, P i = P n+1 A n xp n, P i = A n xp n, P i = 0. Q n (x) = α n P n (x) + α n 1 P n 1 (x). Set α n = B n nd α n 1 = C n. Then, since xp n 1 = n 1 n P n + q n 1, we hve C n = A n xp n, P n 1 = A n P n, xp n 1 = A n P n, n 1 P n + q n 1 30 n = A n n 1 n.

31 D. Levy 4.4 Exmples of orthogonl polynomils Finlly P n+1 = (A n x + B n )P n C n P n 1, cn be explicitly written s n+1 x n+1 +b n+1 x n +... = (A n x+b n )( n x n +b n x n ) C n ( n 1 x n 1 +b n 1 x n +...). The coefficient of x n is b n+1 = A n b n + B n n, which mens tht B n = (b n+1 A n b n ) 1 n. 31