ERDŐS SZEKERES THEOREM WITH FORBIDDEN ORDER TYPES. II

ERDŐS SEKERES THEOREM WITH FORBIDDEN ORDER TYPES. II GYULA KÁROLYI Institute of Mathematics, Eötvös University, Pázmány P. sétány /C, Budaest, H 7 Hungary GÉA TÓTH Alfréd Rényi Institute of Mathematics, Reáltanoda u. 5, Budaest, H 053 Hungary Abstract. According to the Erdős-Szekeres theorem, every set of n oints in the lane contains roughly log n in convex osition. We investigate how this bound changes if our oint set does not contain a subset that belongs to a fixed order tye.. Introduction Throughout this aer we will always assume that every oint set is in general osition in the lane, that is, no three oints of the configuration are collinear. Two such configurations are said to be of the same order tye, if there is a one-toone corresondence between them which reserves the orientation of each trile. Thus, order tyes are equivalence classes of configurations. For examle, given an integer n 3, the vertex set of any convex n-gon belongs to the same order tye we denote by C n. It is clear that the order tye of a configuration stays invariant Visiting the CWI in Amsterdam and the EPFL Lausanne. Suorted by Bolyai Research Fellowshi and OTKA Grant NK67867. Suorted by OTKA Grant K6047.

GY. KÁROLYI, G. TÓTH under orientation reserving non-singular affine transformations and also under a wide class of rojective transformations. The cardinality T of an order tye T is the common cardinality of all configurations contained in it. We will say that the order tye T contains the order tye S if some (hence any) configuration in T contains a subset which belongs to S. We denote this relation by S T and will use this notation on the level of configurations as well. All configurations belonging to the same order tye ossess the same searation roerties. Thus, several notions of discrete geometry carry over to order tyes in a natural way. For examle, if the convex hull of some element of T is a convex n-gon, then it is true for every element of T, in which case it makes sense to say that convt = C n. An order tye is convex, if it equals its convex hull. Ramsey theoretic asects of order tyes have been studied by Nešetřil and Valtr in [7]. Order tyes lay an imortant role in canonical versions of the Erdős Szekeres theorem [8]. A connection was first established via the so called same tye lemma by Bárány and Valtr [4], see also [3] for a survey. According to the Erdős Szekeres theorem, there is an integer N 0 such that every order tye T with T N 0 contains C n. Denoting the smallest such number by F(n), it is known [9, 8] that n + F(n) ( ) n 5 +, n the lower bound conjectured to be tight. This is a truly Ramsey-tye result whose relation to Ramsey s theorem is widely exlored e.g. in [6]. Motivated by a conjecture of Erdős and Hajnal [7] in grah Ramsey theory, Gil Kalai [3] suggested the following roblem. For a fixed non-convex order tye T, define F T (n) as the smallest integer N 0 such that any order tye of size at least N 0 that does not contain T necessarily contains C n. Note that F T is an increasing function. Is it always true that F T (n) is bounded above by a olynomial function of n? Somewhat surrisingly, the analogue with grah Ramsey theory breaks here. In [4] we have shown the existence of an order tye T with F T (n) > n, in contrast with the original Erdős Hajnal roblem where a sub-exonential uer bound is known [7]. For more on this roblem, see [, 5, 0, ]. Our roof however was based on a general result of Nešetřil and Valtr [7] from which it is not easy to extract a concrete order tye T with the above roerty. One novelty in the resent aer is the exhibition of exlicit order tyes T for

FORBIDDEN ORDER TYPES 3 which F T (n) is exonentially large (Theorem ). Such an order tye of size 6 can be obtained, for examle, by utting an extra oint at the centre of a regular entagon. Large order tyes containing neither this, nor C n can be constructed by a doubling rocess we call twin construction, similar to the one found in [5]. We discuss these constructions in Section. On the other hand, several families of order tyes satisfy the analogue of the Erdős Hajnal conjecture, see [4]. In Section 3 we exhibit new families of order tyes T for which the function F T has olynomial growth (Theorems 4 and 5). Our final result (Theorem 8) in Section 4 concerns a comlete characterization of order tyes whose convex hull is a triangle according to the behavior of the function F T (n). They each fall in one of the following three categories: (i) F T (n) is bounded by a linear function in n; (ii) F T (n) is at least quadratic in n but bounded by a olynomial in n; (iii) F T (n) is exonentially large in n. Part of this result originates in [4]. Besides that and the methods involved therein the most crucial element is a new construction that we obtain via modification of Horton s well-known examle []. Somewhat surrisingly these neo- Horton sets can be also obtained by the twin construction.. Exlicite Constructions There exist three different non-convex order tyes T of size less than 6. It was shown in [4], that for any of them F T (n) is bounded from above by a olynomial function in n. In contrast, we have the following result. Theorem. The 6-element order tyes T = A and T = P deicted below satisfy F T (n) > n/. Figure. Order tyes A and P.

4 GY. KÁROLYI, G. TÓTH A common feature of these order tyes is that they have the following searation roerty: any two of their oints can be searated by a line determined by two other oints of theirs. To rove the theorem we first introduce the twin construction. Define sets T k, k 0 recursively. T 0 consists of one oint. Suose we already have defined T k. Take a line l which is not arallel to any line determined by the oints of T k. Relace each oint T k by two oints,,, both very close to, such that the line is arallel to l. The oints and are called the twins of each other and is the arent of them. For a more formal and somewhat more restricted definition, choose k different unit vectors v,...,v k such that no two of them add u to zero, and a small ositive number ε. Given T i, define T i+ as T i (T i + ε i v i+ ). For all sufficiently small values of ε the k -element set T k thus constructed will belong to the same order tye, which by a slight abuse of notation we denote by T k = T (v,...,v k ). Note that different choice of unit vectors may yield the same order tye. On the other hand, reordering of a given sequence of unit vectors usually results in a different order tye. Observe that T k does not have the searation roerty, since two twins cannot be searated by any line determined by other oints. Lemma. For k, no order tye T k contains C k+. Proof. We need to rove that T k does not contain more than k oints in convex osition, which clearly holds for k =. Suose it holds for k and let,,..., m T k be a sequence of m oints in convex osition, in clockwise order. If i and j are twins of each other, then they are consecutive oints. Therefore there can be at most two airs of twins among,,..., m. Relacing each oint by its arent in T k we find at least m oints of T k in convex osition. By the induction hyothesis, m k. It follows that m k. Since both A and P have the searation roerty, Theorem is an immediate consequence of the following claim. Lemma 3. Suose that the order tye S has the searation roerty. Then F S (n + ) > n. Proof. Since T k = k, in view of the revious lemma it will be sufficient to show that T k does not contain a subset whose order tye is S. We rove it by induction on k. It is obviously true for k =. Suose that the statement holds for k.

FORBIDDEN ORDER TYPES 5 Assume that {,,... m } T k belongs to S. Consider any two oints i and j. Since they are searated by some line u v, they cannot be twins, so their arents i and j are different. The set of arents,,... m thus form an m-element subset in T k whose order tye is again S, which contradicts the induction hyothesis. This concludes the roof. In Section 4 we will make use of the following secial case of the twin construction. For any sequence of vectors v,...,v k whose tangents form a decreasing sequence of ositive numbers, the order tye T (v,...,v k ) will be the same. We denote this order tye by RN H k and its mirror image by LN H k. The order tye RN H k resemble very much the order tye H k of Horton s famous construction H k (see []). The difference lies therein, that RN H k is obtained as the order tye of T k = T(v,..., v k ) using very small values of ε, whereas H k can be obtained using vectors v i of sloe α i = π/4 i, and using ( i cosα i ) (/) i instead of ε i for very small ε. This order tye is indeed different from RN H k, which can be easily seen from the following exlicit construction. For any nonnegative integer m let m = i=0 a i i (a i {0, }) be its unique binary reresentation. Given that, define m = i=0 a i i. Identifying the lane with R, ut m = (m, m). The set RNH k = { m 0 m < k } gives a concrete configuration whose order tye is RN H k. Reflecting it in the second axis we obtain a configuration LNH k LN H k. One can readily check that these neohorton sets have the following remarkable roerties: (i) each set RNH k is centrally symmetric; (ii) for any k < n, RNH n is the disjoint union of n k translated coies RNH k (),..., RNH k ( n k ) of RNH k = RNH k () such that for every i < j, the whole set RNH k (j) lies above and to the right of RNH k (i); (iii) for i < j and x, y RNH k (i), the whole set RNH k (j) lies above the line xy, thus every oint of RNH k (j) sees the oints of RNH k (i) so that if m < m, then m recedes m in counterclockwise order; (iv) for j < i < i, every oint of RNH k (i) sees the oints of RNH k (j) later than the oints of RNH k (i ). These roerties will be used for the roof of Lemma 9 in Section 4 without any artial reference.

6 GY. KÁROLYI, G. TÓTH 3. Order Tyes with the Erdős Hajnal Proerty We say that the order tye T has the Erdős Hajnal roerty, if F T is bounded from above by a olynomial function. Here we exhibit three families of order tyes with this roerty. The notations we aly here slightly deviate from those used in [4]. First, for any k, consider a configuration E = {a, b, c,,..., k } such that the oints,..., k lie inside the triangle abc and the oints b,,..., k, c are in convex osition. E belongs to a unique order tye we denote by E k. Thus, F E (n) = n. In general F Ek is bounded form above by a linear funcion, see [4]. Next, for any k 3, consider a configuration F = {a, b, c,,..., k } such that the oints,..., k lie inside the triangle abc, the oints,..., k lie inside the convex quadrilateral b k c, the oints,..., k are in convex osition, and no line defined by two of them intersects the segment bc. The order tye of F we denote by F k. Finally, let k 4, l, m 0 be arbitrary integers. Two configurations X and Y are said to be mutually avoiding if any line determined by two oints of X has all oints of Y on the same side, and vice versa. Consider a configuration G = {,..., k, q,..., q l, r,...,r m } with the following roerties. The oints,..., k are in convex osition, the oints q,..., q l, r,...,r m lie inside the convex olygon... k, the oints, q,...q l, are in convex osition such that Q = {, q,...q l, } and G \ Q are mutually avoiding, and similarly, 3, r,...r m, 4 are in convex osition such that R = { 3, r,... r m, 4 } and G \ R are mutually avoiding. Deending on the orientation of the convex olygon... k, G belongs one of (at most) two different order tyes, which we denote by G k;l,m. a a 6 5 3 b c b c 3 4 q q q 3 r r 3 4 Figure. Order tyes E 3, F 4, and G 6;3,.

FORBIDDEN ORDER TYPES 7 We will rove that all these order tyes have the Erdős Hajnal roerty. With a small modification of an idea in [4] we have the following result. Theorem 4. Every order tye F k with k 3 has the Erdős Hajnal roerty. Theorem 4 in [4] asserts that the order tyes G k;l,0 also have this roerty. Here we claim the following more general result. Theorem 5. Every order tye G k;l,m, where k 4, l, m 0 and not both l and m are zero, has the Erdős Hajnal roerty. Both roofs utilize a result of Erdős and Szekeres concerning cas and cus. The oints (x, y ),..., (x n, y n ) R with x <... < x n form an n-ca if y y > y 3 y >... > y n y n. x x x 3 x x n x n Similarly, they form an n-cu if y y < y 3 y <... < y n y n. x x x 3 x x n x n Lemma 6. [8] Let f(a,b) denote the smallest integer such that any set of f(a, b) oints in general osition in the lane, no two on a vertical line, contains either an a-ca or a b-cu. Then ( ) a + b 4 f(a, b) = +. a Proof of Theorem 4. We rove that for T = F k, the function F T is bounded from above by a olynomial of degree 3k 5. Let X be any order tye of cardinality X > n ( ) n+k 4 3. k Assuming that Cn X, we rove that F k X. Let X X, then convx has less than n vertices. Triangulating it we find a, b, c X such that more than ( ) n+k 4 3 k oints of X lie inside triangle abc. Denote by P the set of these oints. Define a artial ordering ab on P as follows: For, q P, let ab q if and only if the ray q intersects side bc and the ray q intersects side ac of the triangle. One can readily check that the relation ab is indeed transitive. Partial orders ac and bc can be introduced in a similar manner. Note that any two oints of P are related by exactly one of these three relations. Thus, a reeated alication of Dilworth s theorem [6] gives that there is a subset P of P of size P > ( ) n+k 4 k, which is linearly ordered with resect to one of the three artial orders, say bc.

8 GY. KÁROLYI, G. TÓTH Consider a Cartesian system whose horizontal axis meets the rays ab and ac at equal angles and the oints of P lie in the uer half lane. The first coordinates of the elements of P follow each other according to the linear order bc. Since X, and hence P too, does not contain an n-ca, according to Lemma 6 it must contain a k-cu,..., k. It is clear that {a, b, c,,..., k } F k. Thus, F k X as claimed. Given a family of sets Y, Y,..., Y m, a transversal of this family is an m- element set {y, y,..., y m } such that y i Y i for i =,,...,m. One key to the roof of Theorem 5 is the following same tye lemma due to Bárány and Valtr. Lemma 7. [4] For every integer t there is a ositive c t with the following roerty. Assume that X, X,..., X t are lanar oint sets such that X X X t in general osition. Then there are subsets Y i X i with Y i c t X i, such that all transversals of Y, Y,..., Y t belong to the same order tye. Proof of Theorem 5. Let X be an order tye of cardinality X > c 0 n α, which does not contain C n. We rove that if c 0 = c 0 (k, l, m) and α = α(k, l, m) are sufficiently large, then G k;l,m X. Let X X, and assume that no two oints of X lie on a vertical line. Choose a large enough integer t = t(k, l, m) whose value will be secified later. According to a result of Aronov et al. [], every configuration of N oints contains two mutually avoiding subsets of size at least N/0. By a reeated alication, we can obtain airwise mutually avoiding subsets X, X,...X t, such that X i > c n β holds for every i t with β > α/t. Using Lemma 7, we can find subsets X i X i, X i > c n β such that any transversal of X, X,...X t is of the same order tye. In view of the Erdős Szekeres theorem (Lemma 6), there is a sequence i, i,... i s such that s log 4 t, and any transversal of X i, X i,...x i s is in convex osition. For simlicity, we denote X i j by Y j. Consider now any ordered air (Y i, Y j ), i, j s. Define a binary relation on the oints of Y i. For, q Y i, let q if and only if has smaller x- coordinate than q, and all oints of Y j lie above the line q. It is not hard to see that is a artial ordering. According to Dilworth s theorem, there is either a chain or an antichain of size Y i > c 3 n β/. Suose that C Y i is such a chain (res. antichain). Then all oints of Y j are above (res. below) every line determined by C. Delete all oints of Y i which are not in that chain (res. antichain).

FORBIDDEN ORDER TYPES 9 Proceed analogously for each ordered air (Y i, Y j ). Denote the resulting sets by i Y i, i =,...s. Now we have the family,,... s, such that any transversal of,,... s is in convex osition, in this counterclockwise order, for any air ( i, j ), j is either above, or below every line determined by i, and i > c 4 n γ holds for every i s with γ = β/ t. Define now a four-coloured comlete grah on the vertex set {,...,s} as follows. For any i < j, we know that j is either above, or below every line determined by i, and i is either above, or below every line determined by j. So we have four ossibilities for the air ( i, j ), that determines the color of the edge ij. Call the correonding colours aa, ab, ba, and bb, resectively. By Ramsey s theorem, there is a comlete monochromatic subgrah of size r log 56 s. Suose without loss of generality that its vertices are,...,r. Now we should distinguish four cases. Since reflection in the x-axis interchanges the above and below relations, is will be enough to consider two cases. Case : All edges are coloured with colour aa. Case : All edges are coloured with colour ab. r 3 r r Figure 3. Cases and. Now we assume that t is big enough so that r k. We choose the value of c 0 and α so that {( ) n + l c 4 n γ > max, l ( n + m m )}.

0 GY. KÁROLYI, G. TÓTH X does not contain n oints in convex osition, therefore does not contain an n-cu. It follows from Lemma 6 that in either case contains an (m + )-ca C = { 3, r,..., r m, 4 }. For i =,...k 3, choose a oint i+3 i. In Case, we use the fact that k does not contain an n-cu, therefore it must contain an (l+)-ca C k = {, q,...,q l, }. In Case, we use the fact that k does not contain an n-ca, therefore it must contain an (l + )-cu C k = {, q l,...,q, }. k 5 k 3 4 3 4 5 Figure 4. Finding G k;l,m. In either case, the set C k C { 5,..., k } is a configuration whose order tye is G k;l,m. It is roved that G k;l,m X. 4. Order Tyes with Triangular Convex Hull The following result describes the growth of the function F T (n) for order tyes T whose convex hull has three vertices. Theorem 8. Let T C 3 be an order tye whose convex hull is C 3. (i) If T = E k for some integer k, then F T (n) is bounded by a linear function in n. (ii) If T = F k for some integer k 3, then F T (n) is at least quadratic in n but bounded by a olynomial in n. (iii) If T E k, F k, then F T (n) is exonentially large in n. Proof. Part (i) and the lower bound in (ii) is contained in Theorems 3 and 6, resectively, of the earlier aer [4]. The uer bound in (ii) can be obtained

FORBIDDEN ORDER TYPES with a slight modification of Theorem 4 therein, which we described in the revious section. The key to the third satatement is the following lemma, whose roof we ostone until the end of this section. Lemma 9. Let n be a ositive integer and let T be an order tye of a configuration of 3 oints contained in the convex hull of other 3 oints. If T is contained in both LN H n and RN H n, then T = E 3, or T = F 3. Let T be a configuration of at least 4 oints such that its convex hull is a triangle abc, and the relations T LNH n, T RNH n hold for some ositive integer n. If T 5, then T E = F, or T E = F. If 6 T = k + 3, then Lemma 9 imlies that S E 3 F 3 holds for every 6-element configuration S T. It follows that line q intersects the same two sides, say ac and bc, of triangle abc for each air of two different oints, q T \ {a, b, c}. That is, the elements of T \ {a, b, c} can be ordered as,..., k so that the rays i j and j i intersect sides bc and ac, resectively, for every i < j k. Assume that T i = {a, b, c, i, i+, i+ } F 3 and T i+ = {a, b, c, i+, i+, i+3 } E 3 for some i k 3. Then oints i+, i+, i+3 lie inside triangle i bc so that line i+ i+ intersects sides i b and bc, whereas line i+ i+3 intersects sides i c and bc of the triangle (Fig. ), a contradiction. By symmetry, it is not ossible that T i E 3 and T i+ F 3. Therefore T i must belong to the same order tye, either E 3 or F 3, for every i k 3. Accordingly, T E k or T F k. c a i+ i i+ i+3 b Figure 5. S = {b, c, i, i+, i+, i+3 } E 3 F 3.

GY. KÁROLYI, G. TÓTH Thus we roved that if T E k, F k, then either LN H n or RN H n does not contain T for every integer n. From Lemma it follows that f T (n+) > n and f T has exonential growth as claimed. It remains to rove Lemma 9. If T E 3, F 3, then T is either one of the four order tyes deicted on Fig. 6, or one of the mirror images C, D. c c c c z z z z a x y b a x y b a x y b a x y b A B C D Figure 6. Order tyes of six oints. We must rove that neither of these six order tyes is contained in both LN H n and RN H n. Since A has the searation roerty, it is not conteined in any twin construction. Therefore neither LN H n nor RN H n does contain A. Next we rove that B is not contained in RN H n. Assume that on the contrary, a configuration {a, b, c, x, y, z} B is contained in RNH n. Condider the smallest k such that {x, y, z} is contained in RNH k (i) for some i n k. Both RNH k (i ) and RNH k (i) must contain at least one of x, y, z. Because of the threefold symmetry of B and the central symmetry of RNH n, without any loss of generality we may suose that x, y RNH k (i ) and z RNH k (i). Note that z is inside triangle xyc. Now c RNH k (j) for some j n k+. Here j i, for otherwise any vertical line that searates RNH k (i ) and RNH k (i) would searate {x, y, c} from z. It is equally imossible that j = i, since in that case both x and y would lie below the line cz, so cz would not searate x and y. Finally, were j > i, both x and y would lie left to the line cz, again a contradiction. To see that C is not contained in LN H n, we assume that a configuration {a, b, c, x, y, z} C is contained in LNH n and that k is the smallest integer such that {x, y, z} LNH k (i) for some i n k. Having lost the threefold symmetry, we must distinguish three subcases.

FORBIDDEN ORDER TYPES 3 Case : z LNH k (i ) and x, y LNH k (i). For z is inside triangle xyc, mirroring the revious argument we find that c cannot be in any subset LNH k (j). Case : y LNH k (i ) and x, z LNH k (i). Now we use the fact that y is inside triangle zxb. We arrive at a contradiction as before: there is no lace for the oint b. Case 3: x LNH k (i ) and y, z LNH k (i). Because of the orientation of triangle xyz, the oints x, y, z follow each other from left to right in this order. Since the orientation of both triangles yzb and yzc is clockwise, both b and c must lie under any horizontal line l that searates LNH k (i ) and LNH k (i). Point x sees y, z and c in this order, therefore c must lie in LNH k (i). For triangle abc to contain z, oint a must lie above l. But then line ax cannot searate z and c, a contradiction. Thus we have roved that C is indeed not contained in LN H n. By symmetry, C is not contained in RN H n. A similar argument demonstrating that RN H n does not contain D comletes the roof. We omit the technical details. Acknowledgment This aer was comleted during the secial semester on Discrete and Comutational Geometry held at the EPFL Lausanne, sonsored by the Centre Interfacultaire Bernoulli and the Swiss National Science Foundation. References [] N. Alon, J. Pach, and J. Solymosi, Ramsey-tye theorems with forbidden subgrahs, Combinatorica (00) 55 70 [] B. Aronov, P. Erdős, W. Goddard, D.J. Kleitman, M. Klugerman, J. Pach, and L.J. Schulman, Crossing families, Combinatorica 4 (994) 7-34 [3] I. Bárány and Gy. Károlyi, Problems and results around the Erdős Szekeres convex olygon theorem, in: Discrete and Comutational Geometry, (J. Akiyama et al., eds.), Lecture Notes in Comuter Science 098, Sringer (00) 9 05 [4] I. Bárány and P. Valtr, A ositive fraction Erdős-Szekeres theorem, Discrete Comut. Geom. 9 (998) 335 34 [5] M. Chudnovsky and S. Safra, The Erdős-Hajnal conjecture for bull-free grahs, J. Combin. Theory B 98 (008) 30 30 [6] R.P. Dilworth, A decomosition theorem for artially ordered sets, Annals Math. 5 (950) 6 66 [7] P. Erdős and A. Hajnal, Ramsey-tye theorems, Discrete Al. Math. 5 (989) 37 5

4 GY. KÁROLYI, G. TÓTH [8] P. Erdős and G. Szekeres, A combinatorial roblem in geometry, Comositio Math. (935) 463 470 [9] P. Erdős and G. Szekeres, On some extremum roblems in elementary geometry, Ann. Univ. Sci. Budaest. Eötvös, Sect. Math. 3/4 (960 6) 53 6 [0] J. Fox and B. Sudakov, Induced Ramsey-tye theorems, Adv. Math. 9 (008) 77 800 [] A. Gyárfás, Reflections on a roblem of Erdős and Hajnal, in: The Mathematics of Paul Erdős (R.L. Graham and J. Nešetřil, eds.) Algorithms and Combinatorics 4 Volume II, Sringer (997) 93 98 [] J.D. Horton, Sets with no emty convex 7-gons, Canadian Math. Bull. 6 (983) 48 484 [3] G. Kalai, Private communication, IAS Princeton (000) [4] Gy. Károlyi and J. Solymosi, Erdős Szekeres theorem with forbidden order tyes, J. Combin. Theory A 3 (006) 455 465 [5] Gy. Károlyi and P. Valtr, Point configurations in d-sace without large subsets in convex osition, Discrete Comut. Geom. 30 (003) 77 86 [6] W. Morris and V. Soltan, The Erdős-Szekeres roblem on oints in convex osition a survey, Bull. Amer. Math. Soc. 37 (000) 437 458 [7] J. Nešetřil and P. Valtr, A Ramsey roerty of order tyes, J. Combin. Theory A 8 (998) 88 07 [8] G. Tóth and P. Valtr, The Erdős-Szekeres theorem: uer bounds and related results, in: Combinatorial and Comutational Geometry (J.E. Goodman et al., eds.), Publ. M.S.R.I. 5 (005) 557 568 E-mail address: E-mail address: karolyi@cs.elte.hu Gyula Károlyi geza@renyi.hu Géza Tóth