How To Create A Toecdn (Open Edge Content Delivery Network) From Scratch On A Microsoft Ipad Or Ipad (For Free) On A Pc Or Ipa (For A Free) With A Free Ipad) On An Ip

Transcription

1 The Open Edge Content Delivery Network

2 LATEX 2ε layout from Copyright c Fredrik Danerklint bitcoin: 1toecdnToGp4QJCK5QBHiMBnLDqgufXVy The concept of The Open Edge Content Delivery Network is hereby released under Public Domain First edition, December 2013

3 1 Preface The Open Edge Content Delivery Network, TOECDN for short, is a concept of how to distribute static content. In the beginning there was a plan (and still is) to change the whole entertainment industry as known today. To change the whole entertainment industry as such would be quiet easy to do. It is the distribution of the content which is the hardest part. Let s see how much trouble distributing static content can give us. Assume that we have a release for a blockbuster movie, that s gonna be distributed through a streaming service. This to households at the same time, in the same city. Each household is going to require at least 40 Mbit/s of bandwidth (blu-ray quality) to be able to watch without any kind of buffering. If we divide the city into 25 different areas, were we in each area put cache-servers providing 20 Gbit/s of bandwidth, this could actually work. To summarise, this would consume 400 Gbit/s of total bandwidth, only to these households and only for one movie in one city! If you have more, multiply accordingly.

4 4 Preface Today s business strategy for Content Delivery Network industry is built upon that they will be the one to distribute the content (in the future) to consumers. Some CDN:s will even put their servers within a Internet Service Provider (core) network, for free, if the ISP are consuming a lot of bandwidth from them. To help a ISP even further with reducing their bandwidth, they can have something called transparent caching. This will only work on a few specific sites and it does not scale that well (limitation of the hardware used for this). What they are now discussing is to how to exchange content between each other as CDN:s. Exchanging data between three or more parties is something which we are already doing today, via Internet Exchange Points. IXP:s should be aware that when the concept of TOECDN is gaining success with ISP:s and Content Providers, your traffic will decrease! This is also true for Tier 1 and Tier 2 ISP:s. And all current Content Delivery Networks.

5 5 The purpose of making the concept of TOECDN is: (1) The main goal has been to allow the end users/customer/consumers of an ISP to be able to have their own cache-servers at their residence. (2) The next thing we wanted to get rid of was the concept of today s mirrors. A mirror is a server which duplicate another server s content. This is to allow users to download the content more closely to were they are located. A mirror must have all the content available and also calculate how much space on their harddrive(s) it will consume, in the future, even if the consumers are only interested in 10% of the content. (3) The third thing to solve is the geographic location of an request. Today everybody is using a database which contains the country of origin for an ip-address. TOECDN has the location from which city the request is coming from. (4) Force every ISP to have their Internet Routing Registry records up to date. Using an already public database, we add one line (a remark) into this database to be able to get TOECDN start working. (5) Be able to build cache hierarchy or mesh networks between ISP:s and/or IXP:s, to load (new) content faster, when a cache-server has a miss in their cache. (6) Allow the smallest ISP to implement their own cache-servers within their network. (7) Utilise an open concept released under Public Domain to make sure that anyone could use these features.

6

7 2 An overview of the concept Before we begin, let s take an overview of how TOECDN is supposed to work and how the different safety nets of TOECDN are working. What we are trying to access in these different test cases are: You can find the source code for the DNS specific part at which has working examples for both an ISP and for those who is providing (static) content. 2.1 yoursite.example yoursite.example is not interested in running their own DNS Server for TOECDN, so they are using toecdn.net for that. Toecdn.net will only check to see if the request comes from a known ISP or not and answer appropriate to that. In their bind.conf file for their domain, they have made the following changes: static.yoursite.example. IN CNAME static.yoursite.example.toecdn.net. *.static.yoursite.example. IN AAAA 2001:0db8:517e::42

8 8 An overview of the concept They have an httpd-server running at the ip-address 2001:0db8:517e::42. They have also configured their httpd-server so that it add the following header to allow the cache-servers to actually cache their content: For status-code 200: Cache-Control: proxy-revalidate,s-maxage=0,must-revalidate,max-age=0 For status-code 404: Cache-Control: no-cache 2.2 isp.example The ip addresses for the ISP anycast cache server(s) are: 2001:0db8:156::1:cace and 2001:0db8:156::2:cace. The second address is used for the cache server for an customer of the ISP to know where to load the content from. Our ISP is running a specific dns-backend for TOECDN at their subdomain tc.isp.example. After they have verified that their dns-backend is working properly, they update their routing set in their IRR database with this: REMARK: ToEcDn: tc.isp.example Now a Content Provider knows if a ISP can handle TOECDN or not from that routing set. Our dns-server, cache-server and customer is physically located in Karlskrona, Sweden. (kaa.k.se)

9 2.3 Example: Example: 1 This example will show what happens when a ISP is not participating in TOECDN. dns-resolver@isp trying to lookup ip-address for: static.yoursite.example yoursite.example s dns-server toecdn.net s dns-server 1. Question: ANY static.yoursite.example 2. Answer: CNAME static.yoursite.example.toecdn.net 3. Question: ANY static.yoursite.example.toecdn.net 4. Answer: CNAME xx.static.yoursite.example 5. Question: ANY xx.static.yoursite.example 6. Answer: AAAA 2001:0db8:517e::42 Note that when toecdn.net does not find that the ISP is listed in the database (gather from the routing sets), it will answer back with the prefix of xx as an indication of this (step 4).

10 10 An overview of the concept 2.4 Example: 2 This example will show what happens when a ISP is participating in TOECDN but does not have any cache server(s) online (for one reason or another). 1. yoursite.example s dns-server dns-resolver@isp trying to lookup ip-address for: static.yoursite.example toecdn.net s dns-server tc.isp.example s dns-server running with a toecdn-specific backend 1. Question: ANY static.yoursite.example 2. Answer: CNAME static.yoursite.example.toecdn.net 3. Question: ANY static.yoursite.example.toecdn.net 4. Answer: CNAME static.yoursite.example.tc.isp.example 5. Question: ANY static.yoursite.example.tc.isp.example 6. Answer: CNAME kaa.k.se.static.yoursite.example 7. Question: ANY kaa.k.se.static.yoursite.example 8. Answer: AAAA 2001:0db8:517e::42 The different between this example and the previously example is the answer in step 4 and the new step 6. Step 4 just tell the resolver that you should try to ask the dns sever at the ISP for an answer. Since the ISP in this example don t have any cache server(s) online (step 6), we have to give something back as answer, so the user has some server it can connect to. We simply cannot just say static.yoursite.example since that will create an lookup-loop back to us.

11 2.5 Example: 3 11 So we add something which can make the answer somehow unique. The solution to solve this problem is to add the UN locode of the city where the dns server is located! (you can see this in step 6). The CP can now use this information to let the user connect to a server closer to the user, if they have any. 2.5 Example: 3 In this example we will see how the content is served from a cache-server to our browser. DNS lookup for static.yoursite.example tc.isp.example s dns-server running with a toecdn-specific backend User accessing URL dns-resolver@isp trying to lookup ip-address for: static.yoursite.example toecdn.net s dns-server 2. yoursite.example s dns-server User residence 16. Connection to origin httpd-server isp.example s cache-server *.static.yoursite.example s httpd-server dns-resolver@isp cache-server is asking for: kaa.k.se.static.yoursite.example

12 12 An overview of the concept First part - dns lookup: 1. Browser asking dns-resolver@isp for static.yoursite.example 2. Question: ANY static.yoursite.example 3. Answer: CNAME static.yoursite.example.toecdn.net 4. Question: ANY static.yoursite.example.toecdn.net 5. Answer: CNAME static.yoursite.example.tc.isp.example 6. Question: ANY static.yoursite.example.tc.isp.example 7-8. Answer: AAAA 2001:0db8:156::1:cace Second part - making a connection to cache-server and origin httpd-server 9. Browser open a connection to 2001:0db8:156::1:cace 10. Cache-server has a miss so its need to lookup and get the content from origin. From example2 we make the lookup unique as we include our UN locode, kaa.k.se. 11. Question: ANY kaa.k.se.static.yoursite.example Answer: AAAA 2001:0db8:517e:: Cache-server can now connect to 2001:0db8:517e:: Cache-server gets the content from the httpd-server 16. and the browser gets it from the cache-server! Third part - cache-server at residence. We need three things for making this work. Which ip-address to listen to, 2001:0db8:156::1:cace. Which ip-address to ISP cache-server, 2001:0db8:156::2:cace for making outgoing connection to and our UN locode, kaa.k.se. If we cannot connect to 2001:0db8:156::2:cace, we can try to lookup kaa.k.se.static.yoursite.example and connect to that as a fallback. 9. Browser open a connection to 2001:0db8:156::1:cace 10. Cache-server has a miss so it is making a connection to 2001:0db8:156::2:cace. 11. ISP s cache-server is doing point 10 to 15 as in the second part. 12. Cache-server is getting the content from ISP s cache-server 13. and the browser gets it from the cache-server! Closer than this to the edge is hard to come by.