The Internet Intelligence Authority Bahrain's Internet Ecosystem James Cowie, CTO ITS 2009 Manama, Bahrain
Overview The Study and its Goals Primary Conclusions Who is Renesys? Terminology and Methodology Sizing the Bahrain Internet Ecosystem Evolving Relationships Among Participants Vulnerabilities and Remediation Factors Affecting Future Growth Provider Historical Details Conclusions 2
Goals of The Study Provide a comprehensive overview of the evolving relationships between Bahrain's Internet Service Providers (ISPs) and the international telecommunications carriers serving the Region. Identify key strengths and potential weaknesses of Bahrain's Internet connectivity Provide historical context to help the reader understand the present state of the market. 3
About Renesys Since 2000, Renesys has specialized in objective third-party analysis of Internet markets and their evolution over time. Renesys provides enterprises, regulators, and Internet service providers with both realtime and historical perspectives on the Internet connectivity of critical counterparties Applications include situational awareness, information assurance, security and business intelligence 4
Impact Analysis for Operators, Regulators Realtime routing changes support assessment of impacts from events such as storms, power outages, physical infrastructure failures. 5
Ecosystem Analysis Exposes Internet's Structure, Critical Networks Few operators, and fewer enterprises, have transparency into the Internet connectivity of their partners, providers, customers, and competitors. 6
One-Slide Review of Internet Routing Network Prefix: a block of contiguous address space (example: 128.36.0.0/16) Autonomous System: a registered organization that speaks BGP with other autonomous systems (example: Batelco, ASN 5416) Border Gateway Protocol (BGP): allows ASNs to advertise their available network prefixes, starting from the originating autonomous system, and spreading throughout the planet within 30s, creating the global routing table 7
Renesys Infrastructure 8
Renesys collects routing perspectives Renesys actively seeks out new locations from which to study the Internet Service providers can meet us from anywhere in the world via multihop BGP, or directly at: NAP of the Americas (Miami) PAIX (San Francisco) One Wilshire LAX (Los Angeles) London Internet Exchange Amsterdam Internet Exchange Equinix Tokyo 9
Determining Relationships Each data partner's perspective provides a distinct autonomous system path from that ISP to each network in Bahrain Each path provides supporting evidence that adjacencies exist (or don't exist) between pairs of provider autonomous systems We classify those relationships as transit or peering and sum over all networks, all providers to understand a given market. 10
Example: A Single Route, Interpreted In June, Renesys partner X had this route: 79.171.240.0/24 :: X 7473 8966 35019 39273 30882 To get to this network from X, go to Singapore Telecom (7473), on to Emirates (8966), and then to the BIX (35019), and then to Lightspeed (39273) and finally Benefit Company (30882), who owns the network 11
Strengths and Limitations Analysis of Internet topology is based solely on observation of BGP route changes, active measurement from outside, and other public data sources Renesys did not have access to or visibility into network traffic in any form, nor did we have visibility into provider-internal routing or most private peering arrangements 12
Data Used in the Study Primary data sources were drawn from the global BGP routing table, summarizing many independent perspectives Renesys receives a realtime picture of 300+ ISPs' favored routes to all of the network prefixes in Bahrain This network topology dataset has been recording changes in global routing for all network prefixes on earth, with 1 second granularity, continuously since 2002. 13
Methodology We examined global routes (and, in some cases, performed traceroutes to specific prefixes) to understand the Kingdom's externally visible connectivity Study conclusions are independent observations or inferences drawn by Renesys from external routing data and active measurement, without the participation of the operators or the TRA. 14
Disclaimers This study was commissioned by the TRA, but Renesys was encouraged to make an entirely independent assessment based on our own data and observations of the market. In cases where the data do not support a firm conclusion, I shall attempt to clearly distinguish inference from fact. The opinions presented here are entirely my own, and not those of the TRA, nor those of the Government of Bahrain. 15
Primary Conclusions of the Study 16
Bahrain Internet Ecosystem (Aug 2009) 17
1. In regional terms, Bahrain is still a relatively small Internet market. Bahrain Kuwait Lebanon Qatar Jordan Saudi Arabia UAE Iran Oman Egypt Syria Iraq Yemen Source: Renesys Market Intelligence August 2009 18
But Bahrain's Internet Ecosystem is one of the richest in the region, for its size. Country ASNs Per Capita Networks ASNs 2008 GDP Population Bahrain 1 7 8 12 13 Kuwait Lebanon Qatar Jordan Saudi Arabia UAE Iran Oman Egypt Syria 2 3 4 6 7 5 8 9 10 12 5 4 9 8 2 6 3 12 1 11 5 3 10 7 2 6 1 12 3 11 5 10 6 13 1 3 2 9 4 8 10 9 12 7 4 8 2 11 1 6 19
Clear Evidence of Domestic Competition Steady growth of the number of fixed-line, wireless, and mobile service providers who participate in the Bahrain ecosystem At the start of 2003, there was one autonomous system originating Bahrain prefixes Batelco. Today there are 18, not counting NSA Bahrain. Autonomous Systems are the golden standard of a diverse ecosystem they can diversify their Internet transit to include multiple providers. 20
2. Bahrain's Internet Connectivity is adequate but should be more diverse 188 networks, 18 autonomous systems, 3 international carriers, 3 physical paths King Fahd causeway west, FLAG/FOG north Arguably insufficient provider diversity: Batelco purchases from Flag and Tata. Zain and BIX purchase from Emirates and Tata. Lightspeed purchases Flag indirectly via Batelco. Mena and 2Connect purchase from Tata and BIX. The other 12 autonomous systems buy from BIX. 21
TRA's pressure to further open FLAG to competition makes sense in this context. Competing providers do not have infrastructure to perform direct connections to the international market In that sense, they are not operating fully independently as competing players A true carrier-neutral exchange facility would be the best solution Second-best solution might be for BIX to purchase access to FLAG transit 22
3. Batelco does not offer domestic peering. Batelco is seen to peer with regional providers outside Bahrain, including Qtel (AS8781) and Emirates (AS8966) This allows regional traffic to be exchanged without incurring the high cost of international transit outside the Gulf Region Batelco does not peer domestically with the BIX, nor with any of the operators hosted there Why is this significant? 23
Domestic Traffic, Routed Internationally If a Batelco customer wishes to exchange traffic with a customer of another domestic operator, packets would very likely pass: From the Batelco customer Over Batelco's fixed infrastructure Out of the country to KSA, UAE or even London Through one or more foreign routers Back to Bahrain via Tata, FLAG, or Emirates Into the BIX Back out over Batelco's fixed infrastructure (leased wholesale to the competing operator) Down to the competing operator's customer. 24
Domestic Traffic, Routed Internationally This would seem to provide significant disincentive to purchase competing service! Increased latency (at a guess, 50-250ms) Increased jitter and packet loss Both sides incur higher international transit costs Bahrain has had a functional Internet Exchange for more than five years, in order to avoid situations like this. Domestic operators should exchange traffic settlement-free in order to promote the growth of domestic content and security/stability of domestic Internet traffic. 25
4. Batelco's customers lack transit alternatives. Autonomous systems can purchase transit from multiple providers, and the Internet will use BGP to pick the best inbound path This is the standard way for an enterprise or provider to route around single-provider failure Nearly 40,000 autonomous systems worldwide Zero autonomous systems receive IP transit from Batelco (AS5416). Why not? Is this unusual? 26
National Providers Have ASN Customers Armenian Telephone Company has 10 Cameroon Telecom has 5 Kyrgyz Telecom has 5 New Caledonia PTT has 3 Saudi Arabia STC has 29 Iran DCI has 51 Batelco has none 27
Multiple interpretations of the data All of Batelco's IP transit customers are entirely satisfied and do not wish to pay the extra price to retain an autonomous system number and a second service provider Batelco does not sell IP transit to autonomous systems who are likely to buy from its domestic or international competitors, only wholesale connectivity Either is possible. 28
4. Evolving role of the BIX is unclear. BIX is not a place where all domestic traffic gets exchanged settlement-free (Batelco, Zain, Lightspeed are absent) BIX is not a carrier-neutral facility where domestic providers can meet directly with international carriers who compete for their business BIX is basically a reseller for a few hundred megabits of Tata/Emirates transit, and cannot change the basic dynamics of competitive access to international transit resources 29
This is not the BIX's fault. The BIX has satisfied its original mandates: Provide a home for domestic peering Make it easier for new ASNs to secure international transit (if only indirectly) Satisfy monitoring requirements and lawful intercept capabilities Wholesale connectivity continues to hobble BIX effectiveness. Providers such as 2Connect and Zain have either left the BIX, or diversified away from the BIX with direct purchase of int'l transit. 30
5. Inference of Capacity Constraints in Last Mile to Operators Many BIX participants colocate in the same building (presumably to avoid purchasing wholesale fixed line capacity) Active traceroute studies of BIX members' networks indicate that 95th percentile latencies are significantly higher for nonresident members We interpret this as capacity constraints and network congestion in the last mile between BIX and the operators. 31
ASN provider ASN customer 5416 Batelco 30882 Benefit Co 31452 Zain 35019 BIX 35313 2Connect 35443 Kalaam 35457 Etisalcom BH 35546 Northstar 35568 Nuetel 35729 Viacloud 39015 Mena 39273 Lightspeed 41110 BCN 41303 Ascentech 42931 RTS 44167 icolpl 47380 Kulacom 35019 BIX 5384 Emirates 28.6 154.85 85.25 19.73 991.65 4.38 19.13 33.87 31 17.03 7.28 16.02 188.65 12.79 4.68 15412 Flag 6453 Tata 202.77 326.64 th These are 95358.44 340.34 367.89 percentile worst-case latencies. Generally higher worst332.87 case latencies measured in the 95th percentile window are strongly suggestive of congestion and delay between BIX and the provider in question.
6. Facility Carrier Neutrality is Desirable In a carrier-neutral facility, participants purchase bandwidth directly from carriers The exchange is responsible for space, power, ports, but not international transit Transit prices kept low through open competition In the long run, this should be a primary goal for the TRA, the BIX, and interested operators 33
Will BIX evolve to a more carrier-neutral facility model? Seems unlikely given current mandate and operational model, and limitations on physical capacity Gateway Gulf came online as the study results were being prepared GG and BIX face many of the same challenges: Painful physical connectivity challenges Lack of direct access to international carriers Limited size of the domestic content market Significant international traffic asymmetries 34
7. Start measuring domestic traffic Renesys recommends that TRA work with operators to measure the total volume of Internet traffic that originates and terminates within the Kingdom, and within the Gulf region This is a key metric that reflects the evolution of the domestic and regional market (content and ICT services) Capture ICT investment in domestic market, rather than encouraging flight to offshore information markets, reliance on int'l transit 35
8. Promote investment in domestic content and services Most Internet content consumed in Bahrain comes from the or Europe This creates highly asymmetric traffic ratios More domestic content (and better local caching) will reduce reliance on international transit, improve operators' bargaining power with peers elsewhere in the world Local streaming media, e-commerce, payment processing, would all help to improve balance over time 36
Top 100 Websites Accessed by Bahrain Internet Consumers, August 2009 Source: Alexa.com Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Site http://google.com.bh http://live.com http://youtube.com http://google.com http://yahoo.com http://facebook.com http://msn.com http://maktoob.com http://kooora.com http://travian.ae http://bahrainforums.com http://blogger.com http://tagged.com http://microsoft.com http://wikipedia.org http://4shared.com http://gamezer.com Primary IP 74.125.77.104 207.46.30.34 64.15.120.233 74.125.45.100 209.191.93.53 69.63.176.140 207.68.172.246 74.54.154.48 216.93.181.137 92.51.158.104 207.210.66.170 74.124.127.191 67.221.174.30 207.46.232.182 208.80.152.2 208.88.227.170 208.43.29.244 Country DE CA VG Rank 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 18 19 20 21 22 23 24 25 http://flickr.com http://alwasatnews.com http://mozook.com http://indiatimes.com http://bp.blogspot.com http://bramjnet.com http://te3p.com http://3roos.com 68.142.214.24 74.53.119.169 208.43.69.74 203.199.93.69 unknown 208.64.26.42 208.43.69.85 72.35.81.133 IN unknown 68 26 http://inetmail.com.bh 193.188.97.108 BH 76 FR SA AT unknown SA CA CA DE CA 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 http://dailymotion.com http://bdr130.net http://wordpress.com http://akhbar-alkhaleej.com http://artyria.ae http://imdb.com http://graaam.com http://jro7i.com http://metacafe.com http://hawaaworld.com http://mexat.com http://0zz0.com http://brg8.com http://clicksor.com http://rapidshare.com http://ask.com http://orkut.com http://g9g.com http://friendster.com http://6rb.com http://alayam.com http://conduit.com http://rediff.com http://mediafire.com 195.8.215.136 89.144.99.81 76.74.254.126 64.226.254.55 83.137.113.99 207.171.166.140 208.43.81.104 unknown 72.32.120.222 212.162.151.65 208.115.42.250 64.15.129.80 72.46.153.178 66.48.81.155 195.122.131.14 66.235.120.101 74.125.65.85 75.126.212.102 209.11.168.112 67.205.81.115 66.132.220.36 98.142.106.40 204.2.177.43 38.114.196.10 69 70 71 72 73 74 75 Site http://gulf-daily-news.com http://jeddahbikers.com http://shiavoice.com http://photobucket.com http://doubleclick.com http://jsoftj.com http://imageshack.us http://google.co.in http://zshare.net http://bing.com http://jeeran.com http://bahrainevents.com http://alwaqt.com http://m5zn.com http://classesinternational.com http://6rbtop.com http://arabseyes.com Primary IP 64.226.254.35 67.225.167.166 87.96.162.44 209.17.70.11 216.73.93.8 208.43.78.207 208.94.2.98 74.125.127.104 216.155.135.202 64.4.8.147 70.98.189.92 174.133.64.106 209.51.158.162 67.220.200.226 67.217.100.249 72.55.191.190 72.46.153.146 http://anonymous.com.bh 193.188.112.44 BH http://amazon.com http://netlog.com http://z5x.net http://bbc.co.uk http://vmn.net http://myegy.com http://bitefight.org 72.21.207.65 194.60.206.60 212.187.241.144 212.58.224.138 69.50.138.195 92.241.168.97 87.106.180.118 BE GB GB RU DE http://fsupport.gov.bh 89.31.192.194 BH 66.48.78.201 216.39.57.106 208.64.28.98 75.126.128.130 208.96.49.204 98.129.138.249 72.52.250.71 174.132.118.28 74.86.48.18 97.74.26.1 192.150.18.117 66.96.248.165 66.135.205.13 70.84.164.202 208.43.247.190 69.5.88.231 64.151.87.249 64.38.59.221 32.107.37.84 72.32.84.240 87.233.147.140 174.37.120.248 216.239.59.104 209.85.229.104 CA SK NL http://paypopup.com http://xtendmedia.com http://bahrain2day.com http://kaznova.com http://hihi2.com http://goal.com http://hodood.com http://lakii.com http://onemanga.com http://adsbychannel.com http://adobe.com http://umm.biz http://ebay.com http://maktoobblog.com http://bo7.net http://megaupload.com http://startimes2.com http://alamuae.com http://mbc.net http://zain.com http://mininova.org http://b4bh.com http://google.ae http://google.co.uk Country SE CA
9. Significant physical vulnerabilities persist. Physical infrastructure failures continue to threaten international connectivity Such problems can be routed around with difficulty but the alternative paths experience significant congestion. We shall examine three classes of physical failures that affect Bahrain's international transit, and show how their impact is observed in the routing table. 38
Sea Cable Corridor (Fog and Falcon) Causeway (SFO) Source: TRA 39
9.1. Causeway failures 29 June 2009: King Fahd Causeway Bahrain loses connectivity to Riyadh BIX and all downstream lose Tata transit All traffic diverts to Emirates over FOG cable Significant congestion results For reasons unknown, no operator other than Batelco chose to purchase FLAG transit as an alternative route Batelco experiences 4 minutes of instability, and then fails over to its alternative paths to Tata Restoration 8 hours later 40
Causeway Cut: Visible in Routing 41
Causeway Cut: Visible in Routing 42
9.2. FOG (Fiber Optic Gulf) Cable Failure February 2009: FOG Cable Cut Zain, BIX and all downstream lose Emirates transit Traffic diverts westbound to Tata over Causeway Significant congestion results BIX has more westbound capacity, not as severe But impact lasted for nearly two weeks Batelco not as significantly affected by the outage; transit preferences through FLAG unaffected 43
FOG Cut: BIX shifts to 100% Tata for a week 44
FOG Cut: Zain shifts to 100% Tata for a week 45
FOG Cut: 2Connect prefers direct Tata, avoids congested BIX for a week 46
FOG Cut: Batelco has FLAG, is largely unaffected 47
Percentage unstable across providers Vertical stacking shows correlated failure modes that reveal common physical layer dependencies. Emirates connections are more unstable through Spring/Summer 2009 (FOG cuts) 48
Worst-case events for key BH providers Worst-case hours (out of 5,000) indicate correlation among rare events. BIX and Zain are more exposed than Batelco because they lack FLAG transit for redundant UAE connectivity. 49
FOG Cable seems particularly vulnerable More than 30 days of reduced capacity apparent in first half of 2009 Event signatures in routing data suggest that similar impacts were felt Jan 24 Feb 2, 2009 Feb 12-24, 2009 Mar 30 Apr 6, 2009 Apr 13-18, 2009 Need to supplement FOG in order to guarantee connectivity to UAE (and SMW3/4 transit) 50
Emirates Telecom (AS8966) 51
9.3. FLAG/Falcon, SMW3/4 cables Falcon and/or Sea-Me-We cables are obviously critical for long-distance global interconnection to Europe, Asia SMW4 cut 30 Jan 2008 Flag cut 30 Jan and 2 Feb 2008 Same thing happened again 19 Dec 2008 Under these circumstances, one can only wait for restoration Eastbound BH connectivity was retained 52
Sea-Me-We-4/FLAG Cuts on 30 Jan 2008 53
SMW4/Flag Cuts 30 Jan 2008 Other countries were even more badly affected because of lack of provider diversity Bahrain suffered 10% outage and significant congestion, lasting for weeks 54
Countries impacted, December 2008 55
Individual Provider Histories 56
Batelco (AS5416) Jan 2001: Already buying from Teleglobe (Tata, AS8297 and AS6453) and Savvis (AS3561) when Renesys historical routing data coverage begins. Aug 2002: Turn down AS8297 as Teleglobe retired the ASN Oct 2003: Briefly add transit to FLAG (AS15412) before dropping them again in December 2003 Jan 2004: Add transit via Softbank IDC (AS4694; aka Japan Telecom IDC, aka Cable and Wireless IDC) as a heavily prepended (7x) backup route. Feb 2004: Add Cable and Wireless (AS1273). Jun 2004: Drop Savvis (AS3561). Mar 2006: Add transit via FLAG (AS15412). Jul 2006: Drop Softbank IDC (AS4694). May 2007: Drop Cable and Wireless (AS1273). Jul 2009: Continuing transit relationships via Tata (AS6453) and FLAG (AS15412). 57
Batelco (AS5416) 58
Batelco (AS5416) 59
Bahrain Internet Exchange (AS35019) Mar 2004: Internet exchange license granted by TRA. Jun 2005: BIX comes online, single-homed to Tata (AS6453). Jul 2005: Kalaam Telecom (AS35443) and 2Connect (AS35313) join. (2Connect will also buy a limited amount of backup transit direct to Tata; see below.) Sep 2005: Etisalcom Bahrain (AS35457) and Northstar (AS35546) join. Dec 2005: Viacloud (AS35729) and Mena Broadband (AS39015) join. (Mena will also get transit to Tata; see below.) Jan 2006: Lightspeed Telecom (AS39273) joins. Feb 2006: Nuetel(Amwaj, AS35568) joins. Nuetel also gets occasional satellite transit from IABG Teleport (AS29259) from April 2006 through November 2006. Jul 2006: Ascentech (AS41303) joins. Sep 2006: BCN (AS41110) joins. 60
Bahrain Internet Exchange (AS35019) Jul 2007: RTS (AS42931) joins. Dec 2007: BIX adds second transit to the EMIX (AS8960). 2Connect stops using the BIX for transit. May 2008: Kulacom (AS47380) and icolpl (AS44167) join. MTCVodafone (Zain) also stops using the BIX for transit. Jul 2008: 2Connect resumes using the BIX for transit. Oct 2008: Bahrain Central Informatics (AS48109) joins. May 2009: BIX adds additional transit from Emirates (AS8966). June 2009: BIX drops first EMIX transit (AS8960). August 2009: Lightspeed Telecom (AS39273) stops transiting the BIX, and Gateway Gulf (AS44876) joins. 61
Bahrain Internet Exchange (AS35019) 62
Zain (AS31452) Aug 2003: MTC-Vodafone licensed to provide Internet services. Jun 2004: MTC (AS31452) joins the Internet with satellite transit via Transfer, Ltd (aka Horizon Satellite Services, AS30729). Dec 2004: Add second, terrestrial transit connection to Emirates (AS8961). Mar 2005: Add third transit connection via Horizon Satellite Services (AS30981), drop Transfer, Ltd May 2006: Join the BIX (AS35019) and begin transiting some traffic through them. Oct 2006: Drop Emirates (AS8961) as a transit provider. Sep 2007: Add Tata (AS6453) as a transit provider. May 2008: Stop transiting the BIX entirely, relying on Tata and Horizon for all connectivity. Sep 2008: Turn off Horizon Satellite Services. Start advertising 62.209.16.0/20 (WiMax) on 8 September. Jan 2009: Restore transit via Emirates (AS8961). 63
Zain (AS31452) 64
Mena (AS39015) Nov 2004: Mena licensed to provide Internet services. Dec 2005: Join the BIX (AS35019). Nov 2008: Add second transit connection to Tata (AS6453). Today about 80% of Mena's prefixes are advertised only through Tata, and the remaining 20% are advertised through BIX. It's rare to see prefixes available through both paths. 65
Mena (AS39015) 66
2Connect (AS35313) Sep 2004: Licensed to provide Internet services. Jul 2005: Joined the BIX (AS35019). Feb 2006: Added a limited amount of direct transit to Tata (AS6453). Dec 2007: Dropped BIX transit, but still present at the BIX. Jul 2008: Restored BIX transit. 67
2Connect (AS35313) 68
Lightspeed Communications (AS39273) Jan 2006: Lightspeed comes online at the BIX. Mar 2009: Start providing transit to The Benefit Company (AS30882) Aug 2009: Leave the BIX, start buying FLAG transit (via Batelco), and start originating The Benefit Company's address space directly. Transit diversity has decreased substantially; Lightspeed should consider acquiring a second connection to Tata for maximum east-west diversity, replicating Batelco's own transit strategy. 69
Emirates Telecom (AS8966) 70
Conclusions 71
Bahrain is well-positioned for the future. Increased domestic competition has laid the groundwork for a rich, resilient Internet ecosystem There are still significant concerns about physical diversity, last-mile connectivity, and open access to international carriers Batelco and the BIX, in particular, must continue to evolve in order to maximize the growth potential of the ICT sector over the next decade. 72
Thank You! http://www.renesys.com 73