1 HighDimensional Space


 Scot Scott Horton
 1 years ago
 Views:
Transcription
1 Contents HighDimensional Space. Properties of HighDimensional Space The HighDimensional Sphere The Sphere an the Cube in Higher Dimensions Volume an Surface Area of the Unit Sphere The Volume is Near the Equator The Volume is in a Narrow Annulus The Surface Area is Near the Equator The HighDimensional Cube an Chernoff Bouns Volumes of Other Solis Generating Points Uniformly at Ranom on the surface of a Sphere Gaussians in High Dimension Ranom Projection an the JohnsonLinenstrauss Theorem Bibliographic Notes Exercises
2 HighDimensional Space Consier representing a ocument by a vector each component of which correspons to the number of occurrences of a particular wor in the ocument. The English language has on the orer of 5,000 wors. Thus, such a ocument is represente by a 5,000 imensional vector. The representation of a ocument is calle the wor vector moel [?]. A collection of n ocuments may be represente by a collection of 5,000imensional vectors, one vector per ocument. The vectors may be arrange as columns of a 5, 000 n matrix. Another example of highimensional ata arises in customerprouct ata. If there are,000 proucts for sale an a large number of customers, recoring the number of times each customer buys each prouct results in a collection of,000imensional vectors. There are many other examples where each recor of a ata set is represente by a highimensional vector. Consier a collection of n web pages that are linke. A link is a pointer from one web page to another. Each web page can be represente by a 0 vector with n components where the j th component of the vector representing the i th web page has value, if an only if there is a link from the i th web page to the j th web page. In the vector space representation of ata, properties of vectors such as ot proucts, istance between vectors, an orthogonality often have natural interpretations. For example, the square istance between two 0 vectors representing links on web pages is the number of web pages to which only one of them is linke. In Figure., pages 4 an 5 both have links to pages, 3, an 6 but only page 5 has a link to page. Thus, the square istance between the two vectors is one. When a new web page is create a natural question is which are the closest pages to it, that is the pages that contain a similar set of links. This question translates to the geometric question of fining nearest neighbors. The nearest neighbor query nees to be answere quickly. Later in this chapter we will see a geometric theorem, calle the Ranom Projection Theorem, that helps with this. If each web page is a imensional vector, then instea of spening time to rea the vector in its entirety, once the ranom projection to a kimensional space is one, one nees only rea k entries per vector. Dot proucts also play a useful role. In our first example, two ocuments containing many of the same wors are consiere similar. One way to measure cooccurrence of wors in two ocuments is to take the ot prouct of the vectors representing the two ocuments. If the most frequent wors in the two ocuments cooccur with similar frequencies, the ot prouct of the vectors will be close to the maximum, namely the prouct of the lengths of the vectors. If there is no cooccurrence, then the ot prouct will be close to zero. Here the objective of the vector representation is information retrieval.
3 Figure.: A ocument an its termocument vector along with a collection of ocuments represente by their termocument vectors. After preprocessing the ocument vectors, we are presente with queries an we want to fin for each query the most relevant ocuments. A query is also represente by a vector which has one component per wor; the component measures how important the wor is to the query. As a simple example, to fin ocuments about cars that are not race cars, a query vector will have a large positive component for the wor car an also for the wors engine an perhaps oor an a negative component for the wors race, betting, etc. Here ot proucts represent relevance. An important task for search algorithms is to rank a collection of web pages in orer of relevance to the collection. An intrinsic notion of relevance is that a ocument in a collection is relevant if it is similar to the other ocuments in the collection. To formalize this, one can efine an ieal irection for a collection of vectors as the line of bestfit, or the line of leastsquares fit, i.e., the line for which the sum of square perpenicular istances of the vectors to it is minimize. Then, one can rank the vectors accoring to their ot prouct similarity with this unit vector. We will see in Chapter?? that this is a wellstuie notion in linear algebra an that there are efficient algorithms to fin the line of best fit. Thus, this ranking can be efficiently one. While the efinition of rank seems ahoc, it yiels excellent results in practice an has become a workhorse for moern search, information retrieval, an other applications. Notice that in these examples, there was no intrinsic geometry or vectors, just a collection of ocuments, web pages or customers. Geometry was ae an is extremely useful. Our aim in this book is to present the reaer with the mathematical founations to eal with highimensional ata. There are two important parts of this founation. The first is highimensional geometry along with vectors, matrices, an linear algebra. The secon more moern aspect is the combination with probability. When there is a stochastic moel of the highimensional ata, we turn to the stuy of ranom points. Again, there are omainspecific etaile stochastic moels, but keeping with our objective of introucing the founations, the book presents the reaer with the mathematical results neee to tackle the simplest stochastic moels, often assuming inepenence an uniform or Gaussian istributions. 3
4 (,0,,0,0,) web page 4 (,,,0,0,) web page 5 Figure.: Two web pages as vectors. The square istance between the two vectors is the number of web pages linke to by just one of the two web pages.. Properties of HighDimensional Space Our intuition about space was forme in two an three imensions an is often misleaing in high imensions. Consier placing 00 points uniformly at ranom in a unit square. Each coorinate is generate inepenently an uniformly at ranom from the interval [0, ]. Select a point an measure the istance to all other points an observe the istribution of istances. Then increase the imension an generate the points uniformly at ranom in a 00imensional unit cube. The istribution of istances becomes concentrate about an average istance. The reason is easy to see. Let x an y be two such points in imensions. The istance between x an y is x y = (x i y i ). i= Since i= (x i y i ) is the summation of a number of inepenent ranom variables of boune variance, by the law of large numbers the istribution of x y is concentrate about its expecte value. Contrast this with the situation where the imension is two or three an the istribution of istances is sprea out. For another example, consier the ifference between picking a point uniformly at ranom from the unitraius circle an from a unit raius sphere in imensions. In imensions the istance from the point to the center of the sphere is very likely to be between c an, where c is a constant inepenent of. Furthermore, the first coorinate, x, of such a point is likely to be between c an + c, which we express by saying that most of the mass is near the equator. The equator perpenicular to the x axis is the set {x x = 0}. We will prove these facts in this chapter. 4
5 best fit line Figure.3: The best fit line is that line that minimizes the sum of perpenicular istances square.. The HighDimensional Sphere One of the interesting facts about a unitraius sphere in high imensions is that as the imension increases, the volume of the sphere goes to zero. This has important implications. Also, the volume of a highimensional sphere is essentially all containe in a thin slice at the equator an is simultaneously containe in a narrow annulus at the surface. There is essentially no interior volume. Similarly, the surface area is essentially all at the equator. These facts are contrary to our two or threeimensional intuition; they will be prove by integration... The Sphere an the Cube in Higher Dimensions Consier the ifference between the volume of a cube with unitlength sies an the volume of a unitraius sphere as the imension of the space increases. As the imension of the cube increases, its volume is always one an the maximum possible istance between two points grows as. In contrast, as the imension of a unitraius sphere increases, its volume goes to zero an the maximum possible istance between two points stays at two. Note that for =, the unit square centere at the origin lies completely insie the unitraius circle. The istance from the origin to a vertex of the square is ( ) +( ) = = an thus the square lies insie the circle. At =4, the istance from the origin to a vertex of a unit cube centere at the origin is ( ) +( ) +( ) +( ) = an thus the vertex lies on the surface of the unit 4sphere centere at the origin. As the imension increases, the istance from the origin to a vertex of the cube increases as, an for large, the vertices of the cube lie far outsie the unit sphere. Figure.5 illustrates conceptually a cube an a sphere. The vertices of the cube are at istance 5
6 Figure.4: Illustration of the relationship between the sphere an the cube in, 4, an imensions. Nearly all of the volume Unit sphere Vertex of hypercube Figure.5: Conceptual rawing of a sphere an a cube. from the origin an for large lie outsie the unit sphere. On the other han, the mi point of each face of the cube is only istance / from the origin an thus is insie the sphere. For large, almost all the volume of the cube is locate outsie the sphere... Volume an Surface Area of the Unit Sphere For fixe imension, the volume of a sphere is a function of its raius an grows as r. For fixe raius, the volume of a sphere is a function of the imension of the space. What is interesting is that the volume of a unit sphere goes to zero as the imension of the sphere increases. To calculate the volume of a sphere, one can integrate in either Cartesian or polar 6
7 Ω r Ω r r r Figure.6: Infinitesimal volume in imensional sphere of unit raius. coorinates. In Cartesian coorinates the volume of a unit sphere is given by x = x = x x = x x V () = x x x. x = x = x x = x x Since the limits of the integrals are complex, it is easier to integrate using polar coorinates. In polar coorinates, V () is given by V () = r Ωr. S r=0 Here, Ω is the surface area of the infinitesimal piece of the soli angle S of the unit sphere. See Figure.6. The convex hull of the Ω piece an the origin form a cone. At raius r, the surface area of the top of the cone is r Ω since the surface area is imensional an each imension scales by r. The volume of the infinitesimal piece is base times height, an since the surface of the sphere is perpenicular to the raial irection at each point, the height is r giving the above integral. Since the variables Ω an r o not interact, V () = S Ω r=0 r r = Ω = S A(). The question remains, how to etermine the surface area A () = S Ω? Consier a ifferent integral I () = e (x +x + x ) x x x. 7
8 Incluing the exponential allows one to integrate to infinity rather than stopping at the surface of the sphere. Thus, I() can be compute by integrating in Cartesian coorinates. Integrating in polar coorinates relates I() to the surface area A(). Equating the two results for I() gives A(). First, calculate I() by integration in Cartesian coorinates. I () = e x x = ( π ) = π Next, calculate I() by integrating in polar coorinates. The volume of the ifferential element is r Ωr. Thus I () = Ω e r r r. S 0 The integral Ω is the integral over the entire soli angle an gives the surface area, S A(), of a unit sphere. Thus, I () = A () e r r r. Evaluating the remaining integral gives 0 ) e r r r = e t t t = Γ ( 0 0 an hence, I() = A() Γ ( ) where the gamma function Γ (x) is a generalization of the factorial function for noninteger values of x. Γ (x) = (x ) Γ (x ), Γ () = Γ () =, an Γ ( ) = π. For integer x, Γ (x) = (x )!. Combining I () = π with I () = A () Γ ( ) yiels This establishes the following lemma. A () = π Γ ( ). Lemma. The surface area A() an the volume V () of a unitraius sphere in  imensions are given by A () = π Γ ( ) an V () = π Γ ( ). 8
9 To check the formula for the volume of a unit sphere, note that V () = π an V (3) = π 3 3 Γ( 3 ) = 4 π, which are the correct volumes for the unit spheres in two an 3 three imensions. To check the formula for the surface area of a unit sphere, note that A() = π an A(3) = π 3 π = 4π, which are the correct surface areas for the unit sphere in two an three imensions. Note that π is an exponential in an Γ ( ) grows as the factorial of. This implies that lim V () = 0, as claime...3 The Volume is Near the Equator Consier a highimensional unit sphere an fix the North Pole on the x axis at x =. Divie the sphere in half by intersecting it with the plane x = 0. The intersection of the plane with the sphere forms a region of one lower imension, namely {x x, x = 0} which we call the equator. The intersection is a sphere of imension  an has volume V ( ). In three imensions this region is a circle, in four imensions the region is a threeimensional sphere, etc. In general, the intersection is a sphere of imension. It turns out that essentially all of the mass of the upper hemisphere lies between the plane x = 0 an a parallel plane, x = ε, that is slightly higher. For what value of ε oes essentially all the mass lie between x = 0 an x = ε? The answer epens on the imension. For imension it is O( ). To see this, calculate the volume of the portion of the sphere above the slice lying between x = 0 an x = ε. Let T = {x x, x ε} be the portion of the sphere above the slice. To calculate the volume of T, integrate over x from ε to. The incremental volume is a isk of with x whose face is a sphere of imension  of raius x (see Figure.7) an, therefore, the surface area of the isk is Thus, Volume (T ) = ε ( x ) V ( ). ( ) x V ( ) x = V ( ) ε ( x ) x. Note that V () enotes the volume of the imensional unit sphere. For the volume of other sets such as the set T, we use the notation Volume(T ) for the volume. The above integral is ifficult to evaluate so we use some approximations. First, we use the inequality + x e x for all real x an change the upper boun on the integral to be infinity. Since x is always greater than ε over the region of integration, we can insert x /ε in the integral. This gives Volume (T ) V ( ) ε e x x V ( ) 9 ε x ε e x x.
10 x x x raius ( )imensional sphere Figure.7: The volume of a crosssectional slab of a imensional sphere. Now, x e x x = e x an, hence, Volume (T ) e ε V ( ). (.) ε( ) Next, we lower boun the volume of the entire upper hemisphere. Clearly the volume of the upper hemisphere is at least the volume between the slabs x = 0 an x =, which is at least the volume of the cyliner of raius an height. The volume of the cyliner is / times the imensional volume of the isk R = {x x ; x = }. Now R is a imensional sphere of raius an so its volume is Using ( x) a ax ( Volume(R) = V ( ) ) ( )/. Volume(R) V ( ) ( ) = V ( ). Thus, the volume of the upper hemisphere is at least V ( ). The fraction of the volume above the plane x = ε is upper boune by the ratio of the upper boun on the volume of the hemisphere above the plane x = ε to the lower boun on the total volume. This ratio is e ε which leas to the following lemma. ε ( ) Lemma. For any c > 0, the fraction of the volume of the hemisphere above the plane x = c is less than / c e c. Proof: Substitute c for ε in the above. For a large constant c, c e c / is small. The important item to remember is that most of the volume of the imensional sphere of raius r lies within istance O(r/ ) of the 0
11 r 0( r ) Figure.8: Most of the volume of the imensional sphere of raius r is within istance O( r ) of the equator. equator as shown in Figure.8. For c, the fraction of the volume of the hemisphere above x = c is less than e 0.4 an for c 4 the fraction is less than e Essentially all the mass of the sphere lies in a narrow slice at the equator. Note that we selecte a unit vector in the x irection an efine the equator to be the intersection of the sphere with a ( )imensional plane perpenicular to the unit vector. However, we coul have selecte an arbitrary point on the surface of the sphere an consiere the vector from the center of the sphere to that point an efine the equator using the plane through the center perpenicular to this arbitrary vector. Essentially all the mass of the sphere lies in a narrow slice about this equator also...4 The Volume is in a Narrow Annulus The ratio of the volume of a sphere of raius ε to the volume of a unit sphere in imensions is = ( ε) ( ε) V () V () an thus goes to zero as goes to infinity, when ε is a fixe constant. In high imensions, all of the volume of the sphere is concentrate in a narrow annulus at the surface. Inee, ( ε) e ε, so if ε = c, for a large constant c, all but e c of the volume of the sphere is containe in a thin annulus of with c/. The important item to remember is that most of the volume of the imensional sphere of raius r < is containe in an annulus of with O( r/) near the bounary...5 The Surface Area is Near the Equator Just as a imensional circle has an area an a circumference an a 3imensional sphere has a volume an a surface area, a imensional sphere has a volume an a surface area. The surface of the sphere is the set {x x = }. The surface of the equator is the
12 Annulus of with Figure.9: Most of the volume of the imensional sphere of raius r is containe in an annulus of with O(r/) near the bounary. set S = {x x =, x = 0} an it is the surface of a sphere of one lower imension, i.e., for a 3imensional sphere, the circumference of a circle. Just as with volume, essentially all the surface area of a highimensional sphere is near the equator. To see this, calculate the surface area of the slice of the sphere between x = 0 an x = ε. Let S = {x x =, x ε}. To calculate the surface area of S, integrate over x from ε to. The incremental surface unit will be a ban of with x whose ege is the surface area of a imensional sphere of raius epening on x. The raius of the ban is x an therefore the surface area of the ( )imensional sphere is A ( ) ( x ) where A( ) is the surface area of a unit sphere of imension. Thus, Area (S) = A ( ) ε ( x ) x. Again the above integral is ifficult to integrate an the same approximations as in the earlier section on volume leas to the boun Area (S) e ε A ( ). (.) ε( ) Next we lower boun the surface area of the entire upper hemisphere. Clearly the surface area of the upper hemisphere is greater than the surface area of the sie of a imensional cyliner of height an raius. The surface area of the cyliner is times the circumference area of the imensional cyliner of raius which is A( )( ). Using ( x) a ax, the surface area of the hemisphere is at
13 most ( ) A( ) ( )A( ) A( ) (.3) Comparing the upper boun on the surface area of S, in (.), with the lower boun on the surface area of the hemisphere in (.3), we see that the surface area above the ban {x x =, 0 x ε} is less than ε e ε of the total surface area. Lemma.3 For any c > 0, the fraction of the surface area above the plane x = less than or equal to c e c. Proof: Substitute c for ε in the above. c is So far we have consiere unitraius spheres of imension. Now fix the imension an vary the raius r. Let V (, r) enote the volume an let A(, r) enote the surface area of a imensional sphere. Then, V (, r) = r x=0 A(, x)x. Thus, it follows that the surface area is the erivative of the volume with respect to the raius. In two imensions the volume of a circle is πr an the circumference is πr. In three imensions the volume of a sphere is 4 3 πr3 an the surface area is 4πr..3 The HighDimensional Cube an Chernoff Bouns We can ask the same questions about the imensional unit cube C = {x 0 x i, i =,,..., } as we i for spheres. First, is the volume concentrate in an annulus? The answer to this question is simple. If we shrink the cube from its center (,,..., ) by a factor of (c/) for some constant c, the volume clearly shrinks by ( (c/)) e c, so much of the volume of the cube is containe in an annulus of with O(/). See Figure.0. We can also ask if the volume is concentrate about the equator as in the sphere. A natural efinition of the equator is the set { } H = x x i =. i= We will show that most of the volume of C is within istance O() of H. See Figure.. The cube oes not have the symmetries of the sphere, so the proof is ifferent. The starting point is the observation that picking a point uniformly at ranom from C is 3
14 (,,..., ) Annulus of with O(/) (0, 0,..., 0) Figure.0: Most of the volume of the cube is in an O(/) annulus. (,,..., ) O() (0, 0,..., 0) Figure.: Most of the volume of the cube is within O() of equator. equivalent to inepenently picking x, x,..., x, each uniformly at ranom from [0, ]. The perpenicular istance of a point x = (x, x,..., x ) to H is ( ) x i. i= Note that i= x i = c efines the set of points on a hyperplane parallel to H. The perpenicular istance of a point x on the hyperplane i= x ( i = c to H is c ) or ( ) i= x i. The expecte square istance of a point from H is E (( ) ) ( ) x i = Var x i. i= i= By inepenence, the variance of i= x i is the sum of the variances of the x i, so Var( i= x i) = i= Var(x i) = /4. Thus, the expecte square istance of a point 4
15 from H is /4. By Markov s inequality ( ) ( istance from H is greater istance square from H is Prob = Prob than or equal to t greater than or equal to t ) 4t. Lemma.4 A point picke { at ranom in a unit cube will be within istance t of the equator efine by H = x } i= x i = with probability at least. 4t The proof of Lemma.4 basically relie on the fact that the sum of the coorinates of a ranom point in the unit cube will, with high probability, be close to its expecte value. We will frequently see such phenomena an the fact that the sum of a large number of inepenent ranom variables will be close to its expecte value is calle the law of large numbers an epens only on the fact that the ranom numbers have a finite variance. How close is given by a Chernoff boun an epens on the actual probability istributions involve. The proof of Lemma.4 also covers the case when the x i are Bernoulli ranom variables with probability / of being 0 or since in this case Var(x i ) also equals /4. In this case, the argument claims that at most a /(4t ) fraction of the corners of the cube are at istance more than t away from H. Thus, the probability that a ranomly chosen corner is at a istance t from H goes to zero as t increases, but not nearly as fast as the exponential rop for the sphere. We will prove that the expectation of the r th power of the istance to H is at most some value a. This implies that the probability that the istance is greater than t is at most a/t r, ensuring a faster rop than /t. We will prove this for a more general case than that of uniform ensity for each x i. The more general case inclues inepenent ientically istribute Bernoulli ranom variables. We begin by consiering the sum of ranom variables, x, x,..., x, an bouning the expecte value of the r th power of the sum of the x i. Each variable x i is boune by 0 x i with an expecte value p i. To simplify the argument, we create a new set of variables, y i = x i p i, that have zero mean. Bouning the r th power of the sum of the y i is equivalent to bouning the r th power of the sum of the x i p i. The reaer my woner why the µ = i= p i appears in the statement of Lemma.5 since the y i have zero mean. The answer is because the y i are not boune by the range [0, ], but rather each y i is boune by the range [ p i, p i ]. Lemma.5 Let x, x,..., x be inepenent ranom variables with 0 x i an E(x i ) = p i. Let y i = x i p i an µ = i= p i. For any positive integer r, [( ) r ] E y i Max [ (rµ) r/, r r]. (.4) i= Proof: There are r terms in the multinomial expansion of E ((y + y + y ) r ); each term a prouct of r not necessarily istinct y i. Focus on one term. Let r i be the number 5
16 of times y i occurs in that term an let I be the set of i for which r i is nonzero. The r i associate with the term sum to r. By inepenence ( ) E y r i i = E(y r i i ). i I i I If any of the r i equals one, then the term is zero since E(y i ) = 0. Thus, assume each r i is at least two implying that I is at most r/. Now, E( y r i i ) E(y i ) = E(x i ) p i E(x i ) E(x i ) = p i. Thus, E(y r i i ) E( y r i i ) p i. Let p(i) enote p i. So, i I i I i I i I [( ) r ] E y i i= I I r/ p(i)n(i), ( r where n(i) is number of terms in the expansion of i= i) y with I as the set of i with nonzero r i. Each term correspons to selecting one of the variables among y i, i I from each of the r brackets in the expansion of (y + y + + y ) r. Thus n(i) I r. Also, ( p(i) I I =t i= p i ) t t! = µt t!. ( t. To see this, o the multinomial expansion of i= i) p For each Iwith I = t, we get i I p i exactly t! times. We also get other terms with repeate p i, hence the inequality. Thus, using the Stirling approximation t! = πt ( t t, e) [( ) r ] r/ µ t t r r/ µ t E y i t! πtt e t tr ( ) r/ Max r/ t=f(t) t r, π i= t= where f(t) = (eµ)t. Taking logarithms an ifferentiating, we get t t t= ln f(t) = t ln(eµ) t ln t ln f(t) = ln(eµ) + ln t t = ln(µ) ln(t) Setting ln(µ) ln(t) to zero, we see that the maximum of f(t) is attaine at t = µ. If µ < r/, then the maximum of f(t) occurs for t = u an Max r/ t=f(t) e µ e r/. If 6 t=
17 µ r/, then Max r/ t=f(t) (eµ)r/. The geometric sum r/ r r/ t= tr is boune by twice its last term or (r/) r. Thus, proving the lemma. [( ) r ] E y i i= π Max [ (eµ ) r r [ (erµ ) r Max, Max [ (rµ) r/, r r], e r ( e ) ] r 4 r Theorem.6 (Chernoff Bouns): Suppose x i, y i, an µ are as in the Lemma.5. Then ( ) Prob y i t 3e t /µ, for 0 < t 3µ i= ( ) Prob y i t 4 t/3, for t > 3µ. i= Proof: Let r be a positive even integer. Let y = y i. Since r is even, y r is nonnegative. By Markov inequality i= ] (r ) r Applying Lemma.5, Prob ( y t) = Prob (y r t r ) E(yr ) t r. [ ] (rµ) r/ Prob ( y t) Max, rr. (.5) t r t r Since this hols for every even positive integer r, choose r to minimize the right han sie. By calculus, the r that minimizes (rµ)r/ is r = t /(eµ). This is seen by taking t r logarithms an ifferentiating with respect to r. Since the r that minimizes the quantity may not be an even integer, choose r to be the largest even integer that is at most t /(eµ). Then, ( ) r/ rµ e r/ e (t /4eµ) 3e t /µ t for all t. When t 3µ, since r was choosen such that r t ( ) r r r ( ) r t 3µ t r eµ eµ 7 eu, ( ) r e ( e ) r e r/, 3
18 which completes the proof of the first inequality. For the secon[ inequality, ] choose r to be the largest even integer less than or equal to t/3. Then, Max (rµ) r/, rr r/ an the proof is complete similar to the first case. t r t r Concentration for heaviertaile istributions The only place 0 x i is use in the proof of (.4) is in asserting that E y k i p i for all k =, 3,..., r. Imitating the proof above, one can prove stronger theorems that only assume bouns on moments up to the r th moment an so inclue cases when x i may be unboune as in the Poisson or exponential ensity on the real line as well as power law istributions for which only moments up to some r th moment are boune. We state one such theorem. The proof is left to the reaer. Theorem.7 Suppose x, x,..., x are inepenent ranom variables with E(x i ) = p i, i= p i = µ an E (x i p i ) k p i for k =, 3,..., t /6µ. Then, Prob ( ) x i µ t i=.4 Volumes of Other Solis ) Max (3e t /µ, 4 t/e. There are very few highimensional solis for which there are closeform formulae for the volume. The volume of the rectangular soli R = {x l x u, l x u,..., l x u }, is the prouct of the lengths of its sies. Namely, it is (u i l i ). A parallelepipe is a soli escribe by P = {x l Ax u}, where A is an invertible matrix, an l an u are lower an upper boun vectors, respectively. The statements l Ax an Ax u are to be interprete row by row asserting inequalities. A parallelepipe is a generalization of a parallelogram. It is easy to see that P is the image uner an invertible linear transformation of a rectangular soli. Inee, let R = {y l y u}. Then the map x = A y maps R to P. This implies that i= Volume(P ) = Det(A ) Volume(R). 8
19 Simplices, which are generalizations of triangles, are another class of solis for which volumes can be easily calculate. Consier the triangle in the plane with vertices {(0, 0), (, 0), (, )} which can be escribe as {(x, y) 0 y x }. Its area is / because two such right triangles can be combine to form the unit square. The generalization is the simplex in space with + vertices, which is the set {(0, 0,..., 0), (, 0, 0,..., 0), (,, 0, 0,... 0),..., (,,..., )}, S = {x x x x 0}. How many copies of this simplex exactly fit into the unit square, {x 0 x i }? Every point in the square has some orering of its coorinates an since there are! orerings, exactly! simplices fit into the unit square. Thus, the volume of each simplex is /!. Now consier the right angle simplex R whose vertices are the unit vectors (, 0, 0,..., 0), (0,, 0,..., 0),..., (0, 0, 0,..., 0, ) an the origin. A vector y in R is mappe to an x in S by the mapping: x = y ; x = y + y ;... ; x = y + y + + y. This is an invertible transformation with eterminant one, so the volume of R is also /!. A general simplex is obtaine by a translation (aing the same vector to every point) followe by an invertible linear transformation on the right simplex. Convince yourself that in the plane every triangle is the image uner a translation plus an invertible linear transformation of the right triangle. As in the case of parallelepipes, applying a linear transformation A multiplies the volume by the eterminant of A. Translation oes not change the volume. Thus, if the vertices of a simplex T are v, v,..., v +, then translating the simplex by v + results in vertices v v +, v v +,..., v v +, 0. Let A be the matrix with columns v v +, v v +,..., v v +. Then, A T = R an AR = T. Thus, the volume of T is! Det(A)..5 Generating Points Uniformly at Ranom on the surface of a Sphere Consier generating points uniformly at ranom on the surface of a unitraius sphere. First, consier the imensional version of generating points on the circumference of a unitraius circle by the following metho. Inepenently generate each coorinate uniformly at ranom from the interval [, ]. This prouces points istribute over a square that is large enough to completely contain the unit circle. Project each point onto the unit circle. The istribution is not uniform since more points fall on a line from the origin to a vertex of the square, than fall on a line from the origin to the mipoint of an ege of the square ue to the ifference in length. To solve this problem, iscar all points outsie the unit circle an project the remaining points onto the circle. One might generalize this technique in the obvious way to higher imensions. However, the ratio of the volume of a imensional unit sphere to the volume of a imensional 9
20 unit cube ecreases rapily making the process impractical for high imensions since almost no points will lie insie the sphere. The solution is to generate a point each of whose coorinates is a Gaussian variable. The probability istribution for a point (x, x,..., x ) is given by p (x, x,..., x ) = (π) e x +x + +x an is spherically symmetric. Normalizing the vector x = (x, x,..., x ) to a unit vector gives a istribution that is uniform over the sphere. Note that once the vector is normalize, its coorinates are no longer statistically inepenent..6 Gaussians in High Dimension A imensional Gaussian has its mass close to the origin. However, as the imension is increase something ifferent happens. The imensional spherical Gaussian with zero mean an variance σ has ensity function p(x) = ( ) (π) / σ exp x. σ The value of the Gaussian is maximum at the origin, but there is very little volume there. When σ =, integrating the probability ensity over a unit sphere centere at the origin yiels nearly zero mass since the volume of such a sphere is negligible. In fact, one nees to increase the raius of the sphere to before there is a significant nonzero volume an hence a nonzero probability mass. If one increases the raius beyon, the integral ceases to increase even though the volume increases since the probability ensity is ropping off at a much higher rate. The natural scale for the Gaussian is in units of σ. Expecte square istance of a point from the center of a Gaussian Consier a imensional Gaussian centere at the origin with variance σ. For a point x = (x, x,..., x ) chosen at ranom from the Gaussian, the expecte square length of x is E ( x + x + + x ) = E ( x ) = σ. For large, the value of the square length of x is tightly concentrate about its mean. We call the square root of the expecte square istance (namely σ ) the raius of the Gaussian. In the rest of this section we consier spherical Gaussians with σ = ; all results can be scale up by σ. The probability mass of a unit variance Gaussian as a function of the istance from its center is given by r e r / times some constant normalization factor where r is the 0
Foundations of Data Science 1
Foundations of Data Science John Hopcroft Ravindran Kannan Version /4/204 These notes are a first draft of a book being written by Hopcroft and Kannan and in many places are incomplete. However, the notes
More informationWhich Networks Are Least Susceptible to Cascading Failures?
Which Networks Are Least Susceptible to Cascaing Failures? Larry Blume Davi Easley Jon Kleinberg Robert Kleinberg Éva Taros July 011 Abstract. The resilience of networks to various types of failures is
More informationDatabasefriendly random projections: JohnsonLindenstrauss with binary coins
Journal of Computer an System Sciences 66 (2003) 671 687 http://www.elsevier.com/locate/jcss Databasefrienly ranom projections: JohnsonLinenstrauss with binary coins Dimitris Achlioptas Microsoft Research,
More informationWHICH SCORING RULE MAXIMIZES CONDORCET EFFICIENCY? 1. Introduction
WHICH SCORING RULE MAXIMIZES CONDORCET EFFICIENCY? DAVIDE P. CERVONE, WILLIAM V. GEHRLEIN, AND WILLIAM S. ZWICKER Abstract. Consider an election in which each of the n voters casts a vote consisting of
More informationAn Elementary Introduction to Modern Convex Geometry
Flavors of Geometry MSRI Publications Volume 3, 997 An Elementary Introduction to Modern Convex Geometry KEITH BALL Contents Preface Lecture. Basic Notions 2 Lecture 2. Spherical Sections of the Cube 8
More informationCrossOver Analysis Using TTests
Chapter 35 CrossOver Analysis Using ests Introuction his proceure analyzes ata from a twotreatment, twoperio (x) crossover esign. he response is assume to be a continuous ranom variable that follows
More informationThere are two different ways you can interpret the information given in a demand curve.
Econ 500 Microeconomic Review Deman What these notes hope to o is to o a quick review of supply, eman, an equilibrium, with an emphasis on a more quantifiable approach. Deman Curve (Big icture) The whole
More informationIEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 4, APRIL 2006 1289. Compressed Sensing. David L. Donoho, Member, IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 4, APRIL 2006 1289 Compressed Sensing David L. Donoho, Member, IEEE Abstract Suppose is an unknown vector in (a digital image or signal); we plan to
More informationNew Trade Models, New Welfare Implications
New Trae Moels, New Welfare Implications Marc J. Melitz Harvar University, NBER an CEPR Stephen J. Reing Princeton University, NBER an CEPR August 13, 2014 Abstract We show that enogenous firm selection
More informationSketching as a Tool for Numerical Linear Algebra
Foundations and Trends R in Theoretical Computer Science Vol. 10, No. 12 (2014) 1 157 c 2014 D. P. Woodruff DOI: 10.1561/0400000060 Sketching as a Tool for Numerical Linear Algebra David P. Woodruff IBM
More informationIf A is divided by B the result is 2/3. If B is divided by C the result is 4/7. What is the result if A is divided by C?
Problem 3 If A is divided by B the result is 2/3. If B is divided by C the result is 4/7. What is the result if A is divided by C? Suggested Questions to ask students about Problem 3 The key to this question
More informationA Modern Course on Curves and Surfaces. Richard S. Palais
A Modern Course on Curves and Surfaces Richard S. Palais Contents Lecture 1. Introduction 1 Lecture 2. What is Geometry 4 Lecture 3. Geometry of InnerProduct Spaces 7 Lecture 4. Linear Maps and the Euclidean
More informationONEDIMENSIONAL RANDOM WALKS 1. SIMPLE RANDOM WALK
ONEDIMENSIONAL RANDOM WALKS 1. SIMPLE RANDOM WALK Definition 1. A random walk on the integers with step distribution F and initial state x is a sequence S n of random variables whose increments are independent,
More informationHow to Use Expert Advice
NICOLÒ CESABIANCHI Università di Milano, Milan, Italy YOAV FREUND AT&T Labs, Florham Park, New Jersey DAVID HAUSSLER AND DAVID P. HELMBOLD University of California, Santa Cruz, Santa Cruz, California
More informationMatthias Beck Gerald Marchesi Dennis Pixton Lucas Sabalka
Matthias Beck Gerald Marchesi Dennis Pixton Lucas Sabalka Version.5 Matthias Beck A First Course in Complex Analysis Version.5 Gerald Marchesi Department of Mathematics Department of Mathematical Sciences
More informationFrom Few to Many: Illumination Cone Models for Face Recognition Under Variable Lighting and Pose. Abstract
To Appear in the IEEE Trans. on Pattern Analysis and Machine Intelligence From Few to Many: Illumination Cone Models for Face Recognition Under Variable Lighting and Pose Athinodoros S. Georghiades Peter
More informationWhen Is Nearest Neighbor Meaningful?
When Is Nearest Neighbor Meaningful? Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft CS Dept., University of WisconsinMadison 1210 W. Dayton St., Madison, WI 53706 {beyer, jgoldst,
More informationON THE DISTRIBUTION OF SPACINGS BETWEEN ZEROS OF THE ZETA FUNCTION. A. M. Odlyzko AT&T Bell Laboratories Murray Hill, New Jersey ABSTRACT
ON THE DISTRIBUTION OF SPACINGS BETWEEN ZEROS OF THE ZETA FUNCTION A. M. Odlyzko AT&T Bell Laboratories Murray Hill, New Jersey ABSTRACT A numerical study of the distribution of spacings between zeros
More informationA Tutorial on Spectral Clustering
A Tutorial on Spectral Clustering Ulrike von Luxburg Max Planck Institute for Biological Cybernetics Spemannstr. 38, 7276 Tübingen, Germany ulrike.luxburg@tuebingen.mpg.de This article appears in Statistics
More informationDecoding by Linear Programming
Decoding by Linear Programming Emmanuel Candes and Terence Tao Applied and Computational Mathematics, Caltech, Pasadena, CA 91125 Department of Mathematics, University of California, Los Angeles, CA 90095
More informationAn Introduction to Tensors for Students of Physics and Engineering
NASA/TM 2002211716 An Introduction to Tensors for Students of Physics and Engineering Joseph C. Kolecki Glenn Research Center, Cleveland, Ohio September 2002 The NASA STI Program Office... in Profile
More informationOrthogonal Bases and the QR Algorithm
Orthogonal Bases and the QR Algorithm Orthogonal Bases by Peter J Olver University of Minnesota Throughout, we work in the Euclidean vector space V = R n, the space of column vectors with n real entries
More informationYou know from calculus that functions play a fundamental role in mathematics.
CHPTER 12 Functions You know from calculus that functions play a fundamental role in mathematics. You likely view a function as a kind of formula that describes a relationship between two (or more) quantities.
More informationHow many numbers there are?
How many numbers there are? RADEK HONZIK Radek Honzik: Charles University, Department of Logic, Celetná 20, Praha 1, 116 42, Czech Republic radek.honzik@ff.cuni.cz Contents 1 What are numbers 2 1.1 Natural
More informationRecommendation Systems
Chapter 9 Recommendation Systems There is an extensive class of Web applications that involve predicting user responses to options. Such a facility is called a recommendation system. We shall begin this
More informationIndexing by Latent Semantic Analysis. Scott Deerwester Graduate Library School University of Chicago Chicago, IL 60637
Indexing by Latent Semantic Analysis Scott Deerwester Graduate Library School University of Chicago Chicago, IL 60637 Susan T. Dumais George W. Furnas Thomas K. Landauer Bell Communications Research 435
More informationSemiSimple Lie Algebras and Their Representations
i SemiSimple Lie Algebras and Their Representations Robert N. Cahn Lawrence Berkeley Laboratory University of California Berkeley, California 1984 THE BENJAMIN/CUMMINGS PUBLISHING COMPANY Advanced Book
More informationHighRate Codes That Are Linear in Space and Time
1804 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 48, NO 7, JULY 2002 HighRate Codes That Are Linear in Space and Time Babak Hassibi and Bertrand M Hochwald Abstract Multipleantenna systems that operate
More informationFrom Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images
SIAM REVIEW Vol. 51,No. 1,pp. 34 81 c 2009 Society for Industrial and Applied Mathematics From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images Alfred M. Bruckstein David
More informationIntellectual Need and ProblemFree Activity in the Mathematics Classroom
Intellectual Need 1 Intellectual Need and ProblemFree Activity in the Mathematics Classroom Evan Fuller, Jeffrey M. Rabin, Guershon Harel University of California, San Diego Correspondence concerning
More information