Optimization of File Allocation for Video Sharing Servers

Transcription

1 Optmzaton of Fle Allocaton for Vdeo Sharng Servers Emad Mohamed Abd Elrahman Abousabea, Hossam Aff To cte ths verson: Emad Mohamed Abd Elrahman Abousabea, Hossam Aff. Optmzaton of Fle Allocaton for Vdeo Sharng Servers. NTMS 29, 29, pp.emad Abd-Elrahman and Hossam Aff. <hal-68298> HAL Id: hal https: //hal-nsttut-mnes-telecom.archves-ouvertes.fr/hal Submtted on 27 Mar 212 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 Optmzaton of Fle Allocaton for Vdeo Sharng Servers Emad Abd-Elrahman and Hossam Aff Wreless Networks and Multmeda Servces Department, Telecom SudPars (ex. INT), France. 9, rue Charles Fourer, 9111 Evry Cedex, France. {Emad.AbdElrahman, Abstract Ths paper focuses on one of the most used short vdeo servers on the Internet, t s YouTube. YouTube has the thrd rank of the nternet stes related to ts traffc transactons. In ths work, frstly, we study the effect of the huge numbers of vewers, hts, and fles on the vdeo sharng servers behavours by analyzng some recent statstcs about YouTube. Then, we propose an optmzaton for fle allocaton procedures n general and then we apply the algorthm to some examples of YouTube vdeos. Ths soluton mproves the number of hts related to those knds of servers over the Internet. Fnally, we try to optmze the revenue from the fle allocaton and propose a hybrd soluton for the fle hostng or server cachng systems. Index Terms Servers hts; Socal networks; Fle allocaton optmzatons. I. INTRODUCTION OWADAYS, many servers use short vdeos as a Nbusness model and beneft from a hgh degree of nterest because of ther ablty to easly dffuse these clps to end. Users can share ther own vdeos, download avalable vdeos and also create ther own profles to know the ratng of ther vdeos evaluated by others. Many servers lke YouTube [1], Google [6], Yahoo [12], Dalymoton [13] and others have a good rankng as presented by Alexa [8].Ths evaluaton shows the rankng of content dependng on uploads and downloads and on the number of vdeos uploaded by others. Accordng to the last statstcs showed by Alexa, YouTube has a great and ncreasng nterest as t was ranked thrd ste out of the ranked 5 stes wth the hghest traffc hts over the Internet. We wll focus n ths paper on ths server as a good case study for vdeo sharng servers. Although Yahoo has an advanced rank than YouTube (the second ste by Alexa), YouTube s more famous for short vdeos transfer. Dalymoton also occupes the rank 68 by the same measurement ste Alexa. In ths paper, we try to analyse the network archtecture of these servces and present several enhancements that provde an optmzaton n mportant parameters such as response tme and storage amount. We show that these optmzatons, used at a very large scale, provde really mportant mprovement n the global effcency of the servce as seen by the user and by the provder. The rest of ths paper s organzed as follows; secton II ntroduces the hstory of YouTube and some mportant statstcs, secton III dscusses the socal networkng concept and ts data centres desgn, secton IV presents our proposal for fle allocaton optmzaton, secton V shows our algorthm evaluaton, secton VI draws our attenton towards the problem of fle hostng on YouTube and ts proposed solutons, secton VII presents related work and fnally the concluson for our work and ts future drectons are presented n secton VIII. II. HISTORY OF YOUTUBE YouTube s four years old, and t s already the thrd most vsted webste n the world n 29. YouTube was founded n February 25 and t became so mmensely popular n a short perod of tme. It allows people to easly upload and share vdeo clps on and across the Internet through webstes, moble devces, blogs, and emal. Everyone can watch vdeos on YouTube, upload, download and also create ts own profle on ths server to pursue ther vdeos statstcs. People can see frst-hand accounts of current events and fnd vdeos about ther hobbes and nterests. As more people capture specal moments on vdeo, YouTube s empowerng them to become the broadcasters of tomorrow [14]. In YouTube's hstory, USA n 27 regstered the largest number of hts for ths ste, more than 15,5, clcks n the process of measurng the hts of ths server related to USA only [2]. In the press of December 28, YouTube s the leader for onlne vdeo communty that allows people to dscover, watch and share orgnally created vdeos. So, now YouTube has more than 1 mllon vewers, whch represents two out of three Internet who watched onlne vdeo. Overall, nearly 15 mllon U.S. Net watched an average of 96 vdeos that month. YouTube fans watched some 5.9 bllon vdeos n December 28[15].By the end of 28, YouTube accounted for 2.13 of all UK Internet vsts compared to 2.11 for Lve Mal. Durng the same week, YouTube was the thrd most vsted webste n the UK behnd Google UK and Facebook, whle Lve Mal ranked fourth. And n the recent news for 29, YouTube.com accounted for 1 out of every 3 U.S. onlne vdeos vewed n January. In Germany, 28 mllon onlne vdeo vewers watch more than 3 bllon vdeos n December 28, the statstcs n January 29 ndcatng that 28.5 mllon German Internet vewed a vdeo onlne n December 28, up 1 percent versus the prevous year [7]. The fnal press n USA ndcates that YouTube Attracts 1 Mllon U.S. vewers [5]. In bref, the uploaded rate to YouTube was sx hours of vdeo every mnute n 27. Then t grew to eght hours per mnute, then 1 hours per mnute, then 13 hours per mnute. In 28, t became 15 hours of vdeo uploaded every /9/$ IEEE Authorzed lcensed use lmted to: Telecom and Management Sud Pars. Downloaded on January 21, 21 at 7:38 from IEEE Xplore. Restrctons apply.

3 mnute. Now, 2 hours of vdeo are uploaded to YouTube every mnute, and ths s a huge rate and fact the developng of YouTube to become the best onlne vdeo home. They are dreamng that ths ste wll get 24 hours vdeo uploaded per mnute; that means a full day of vdeo uploaded every mnute [19]. Accordng to Google's Inc. n Sydney, a report sad that; 1 hours of vdeo s uploaded every second to the company's vdeo sharng ste [1]. Actually, there s no precse statstcs about the number of or number of hts to YouTube servers avalable over nternet but the majorty of ths nformaton form the press released on (Table III) shows some recent statstcs for the grand countres (the most vewed fles durng ths month May 29) n each country. We used Google Trends & Analytcs [3, 4] to draw some statstcs about YouTube servers, hts, uploads and downloads, that wll help us magne the structure of YouTube networks. We also suggest some changes to the structure of ths huge socal network so as to gan a lot of optmzng bandwdth utlzaton especally on the nternatonal lnes between dfferent countres and dfferent contnents (as zones). III. SOCIAL NETWORKING A. Defnton A socal network s a specal structure made of nodes representng people or organzatons that are ted by one or more specfc type of relatons such as values, vsons, deas or just frendshps. Really, the concept of socal networkng has been developed more rapdly than the concept of nternet tself durng the last years. B. Data Centers Desgn In the past, an enterprse mght respond to the need for addtonal capacty and performance wth tradtonal data center nfrastructure optons (Fg. 1). For example, the enterprses buld out the current centralzed data center; buld addtonal regonal centers when they sgn contracts for hosted facltes. Snce geography plays an mportant role n delverng vdeo to global vewers, the desgn for data centers changes to decentralzed one (Fg. 2). Hence, the cost n dfferent world regons must be examned. It wll provde a valuable baselne for enterprses lke Google or YouTube to compare aganst alternatves such as Web acceleraton servces. Therefore, enterprses may consder the expensve opton of dstrbutng hardware n regons of the world that are close to and potental markets. For example, decentralzed desgn s the best soluton to buld data centers n dfferent zones lke; Asa, Afrca or wth the much heavy loads areas. The fles movements n ths case may be not expensve f the nfrastructure already exsts all over the world. If the enterprses ncreasngly replace ther decentralzed data archtecture wth large centralzed data center nfrastructure, the capacty becomes an ssue. If the data center reaches 95 utlzaton, the enterprse must fnd a way to scale ts operaton to gan more productvty from ts current nfrastructure [11]. We recommend the decentralzed archtecture to avod the server burden problems and obtan a unform dstrbuton of data centers and not the tradtonal localzed desgn. IV. THE PROPOSAL FOR FILE ALLOCATION OPTIMIZATION We present n ths secton our optmzaton. We assume that; we have a certan number of zones ( from 1 to N where N s the max. no. of zones) and a number of servers j (j from 1 to M where M s the max. no. of servers). The zones correspond to the socal network consumers and the servers are area servers. Our goal s to reach a mnmum cost and to reduce the number of hts on the man servers. So, we try to have a dstrbuton formula that wll lead us to optmze the content locatons on the servers n dfferent zones accordng to mnmum cost between zones and servers. A. Mnmum Cost Let D j be the cost between zone () and server (j) where vares from 1 to N and j vares from 1 to M. Ths corresponds to a user from zone consultng a content from server j. The total cost when consultng the vdeos n dfferent zones would be (C = D j * H j ) where H j s the number of hts for a fle comng from zone to server j. we compute mn (C = D j * H j ) to defne the best allocaton for any fle. Example, f we have 6 zones and 6 servers (cluster) on prncpal of one server per zone then: For zone 1, where the access comes from zone 1: Mn C 1 = (D 11 *H 11 ) +( D 12 *H 12 ) +(D 13 *H 13 ) + ( D 14 *H 14 ) + ( D 15 *H 15 ) +( D 16 *H 16 ) For zone 2, where the access comes from zone 2: Mn C 2 = (D 21 *H 21 ) + ( D 22 *H 22 ) + (D 23 *H 23 )+( D 24 *H 24 )+ ( D 25 *H 25 ) +( D 26 *H 26 ) For zone 3, where the access comes from zone 3: Mn C 3 = (D 31 *H 31 )+ ( D 32 *H 32 ) + (D 33 *H 33 )+( D 34 *H 34 ) + ( D 35 *H 35 ) +( D 36 *H 36 ) For zone 4, where the access comes from zone 4: Mn C 4 = (D 41 *H 41 )+ ( D 42 *H 42 ) + (D 43 *H 43 )+( D 44 *H 44 ) + ( D 45 *H 45 ) +( D 46 *H 46 ) For zone 5, where the access comes from zone 5: Mn C 5 = (D 51 *H 51 )+ ( D 52 *H 52 ) + (D 53 *H 53 )+( D 54 *H 54 ) + ( D 55 *H 55 ) +( D 56 *H 56 ) For zone 6, where the access comes from zone 6: Mn C 6 = (D 61 *H 61 )+ ( D 62 *H 62 ) + (D 63 *H 63 )+( D 64 *H 64 ) + ( D 65 *H 65 ) +( D 66 *H 66 ) So, the mnmum value of C 1 to C 6 wll gve us an approxmate locaton for the best allocaton for ths fle demands so as to save bandwdth and tme for vewng or downloadng. The evaluaton results wll lead us to logcally move ths fle to the regon from whch the hghest demands are comng. We can follow the steps of the proposed algorthm n Matlab n (Table I). Authorzed lcensed use lmted to: Telecom and Management Sud Pars. Downloaded on January 21, 21 at 7:38 from IEEE Xplore. Restrctons apply.

4 B. The Correlaton between Demographc Informaton and Web Usage In ths part, we try to fnd a relaton between the network topology and the cost calculated between zones for any fle. We have yet calculated the cost related to zone (1) by C 1 and make the same for zone 2 to 6. Then, we can decde the best allocaton for ths fle accordng to these values but based on the relaton between zones: 1). Frst Assumpton: Full Mesh Topology If C 1 < (C2 to C 6 ) then the best locaton of ths fle s the servers n zone (1), ths means that the mnmum value C wll defne the best locaton of that fle. Actually, the full mesh desgn suffers from the N 2 problem where, the number of lnks needed for ths desgn equal N (N-1)/2, and no one can say that there s a warranty for full mesh desgn on the nternet (see Fg. 3 left-half). 2). Second Assumpton: Not-full Mesh Topology If the network s partally meshed (see Fg. 3 rght-half) then, the prevous calculatons and decsons wll be dfferent, and we must take nto account the cost of the ntermedate zones. For example, f zone (1) s not drectly connected to zone (2), but connected through zone (3), then, whle makng our calculatons, we must add the cost between zone 1&3 and 2&3 for the calculaton of zone (1). We apply the algorthm n table I on the sx fles selected n table II fle by fle and we take nto consderaton the dfferent locatons of the fle. Therefore, we run the algorthm 6 tmes for each fle by assumng the movements of the fle from zone to another and optmze the best locaton accordng to the mnmum cost equaton (mn (C = D j * H j )).In Fg. 4, we llustrate the results for the fles dstrbuton. We notce that the hts of the sx fles n the dfferent zones play the bg role n the mnmum cost calculatons. For fle 1, ts best locaton s the servers n zone 1 where the mnmum cost of that fle nvestgated by the algorthm. For fle 2, t must be moved from ts locaton n zone 2 to zone 3 where the algorthm gave the mnmum cost. For the rest of fles shown n Fg. 4; fle 3 moves from zone 3 to zone 2, fle 4 moves from zone 4 to zone 3, fle 5 remans n the same zone 5 for achevng mnmum cost and fnally fle 6 moves to zone 4. We conclude that, we can apply that algorthm on any fle for whch we have some statstcs to defne ts best locaton on the vdeo sharng servers. TABLE II EXAMPLES OF SOME FILES CHOSEN FROM YOUTUBE AND RELATED HITS Fles/ s/ Access YouTub e Fle Fle Fle Fle Fle Fle Fle Fle Fg. 3: Full Mesh desgn for 6 zones & Partally Meshed desgn Algorthm TABLE I THE PROPOSED ALGORITHM. Input N (number of zones) Input M (number of servers) Input h j (number of hts come from zone to sever j for specfc fle) Input d j (the assumed cost between zone and server j) Total cost C = hj * dj // where from 1 to N and j from 1 to M If C mn than C +1 to C N Then allocate ths fle n zone () Else moves ths fle to the Mn (C +1 to C N) value locaton End If Return the best locaton for ths fle (best ) End V. THE PROPOSAL EVALUATION In ths secton, we focus on the evaluaton of our algorthm. We choose some fles randomly from YouTube ste that rated as top vewed durng ths perod. We select 6 fles from dfferent zones as lsted n table II, where each fle has ts own statstcs related to hts and from dfferent zones. These statstcs are estmated by the percentage of YouTube for each fle accordng to the fle profle on YouTube ste and the statstcs from Alexa [8] for YouTube /zone utlzaton Fle Fle Fle Fle Fle Fle Fle Fle Fle Fle Fg. 4: Fles dstrbuton by ther hts and the cost related to zones. Authorzed lcensed use lmted to: Telecom and Management Sud Pars. Downloaded on January 21, 21 at 7:38 from IEEE Xplore. Restrctons apply.

5 VI. THE FILE HOSTING PROBLEMS ON YOUTUBE We clam that YouTube popularty wll cause a huge storage problem because the number of fles wll ncrease wthout any revenue from ths hostng. A recent report [17] about YouTube vdeo growth curve assures that YouTube currently streams more than 3 bllon vdeos per month worldwde. Ths huge number of vdeos drew our attenton towards studyng the problems of nflaton of the server database and how the company can beneft from the hostng of those fles. The challenge s how YouTube wll treat ths dffculty wthout affectng the onlne vdeo traffc marketng (there s a forecastng that; by the end of the year 29, YouTube wll lose about 47 mllon Dollars and the company wll need to change ts strateges [18]). A lot of analyses on the contnung growth of YouTube s lbrary versus the cost of storng fles were made. Snce the storage of vdeo fles that are not vewed wll grow and should t be YouTube responsblty to keep these vdeo records for the future and f so can the sponsors support addtonal cost for the long tme hostng of those fles? We beleve that, Google needs to study the phenomena of ths growth and thnk how they wll fnd a good soluton for managng ths problem. YouTube could generate revenues by changng ts strategy towards ganng money that wll not rely only on advertsements but also from hostng fles. We thnk that Google wll gan from the nvestment that wll return from fles hostng on YouTube servers. A. The Proposed Solutons for Vdeo Hostng Costs In ths part we have two problems to solve; the frst one s the long term hostng of fles that become old and the hostng cost n relaton to the revenue from hostng: 1). Long Term Hostng and the Hybrd Algorthm We can add two flags numbers related to each added fle on the server; The frst one s the hstory flag (F h ) that can regster the hstory of ths fle by countng the number of days ths fle allocated or uploaded to the server. The other flag s (F v ) whch represent the number of vewers of that fle. By calculatng the rato (F v /F h ) n a certan perod of tme, we wll have an ndcaton that ths fle deserves to reman on the server n case the rato s hgh or to be omtted from the server database n the other case. Ths rato expresses the Fle Tme-To-Lve FTTL on the server, whch s gven by COUNTRY HITS/DAY FOR THE MOST VIEWED FILE FTTL = (F v /F h ). Most Webs cachng systems depend manly on the tme perod whch the fles spent on the servers for the decson of contnue hostng or deletng (Least Recently Used (LRU) dea). Others depend on the number of vews for those fles (Frequency Based) approaches. But n our proposal we used the two parameters (tmes & numbers) for calculatng FTTL. So, we can consder ths algorthm as a hybrd algorthm for manage cachng or storng fles. 2).The Relaton between Fle Hostng Cost and ts Revenue In ths paragraph, we study the behavour of fle hts on ther hostng servers and the revenue returned from hostng ths fle or movng to another zone server accordng to the fle allocaton algorthm n table I. The followng parameters wll be used n optmzaton of the total revenue from hostng fles; H j : Numbers of hts/day for a fle on server j, accessed from n zone. D j : Cost of httng a fle on server j, from user n zone. C j = H j. Dj : Total cost/day for a fle put on server j. H j = H j : Number of hts/day for a fle on server j. C j C j = : Average cost/day for a fle put on server j. j Hj R : Average revenue generated by a fle put on server j. R j = H j. R j : Total revenue generated /day for a fle put on server j. The formula to compute the Revenue Cost Rato (RCR); RCR = R j Rj. Hj = Cj Hj. Ths rato wll gve us a good ndcaton for the revenue from hostng ths fle n terms of ts hostng cost wth relaton to the fle hts. Fnally we can calculate the net revenue by the followng equaton; R j - C j = H j. j R - D j H j. Dj And ths revenue s consdered as a good ndcaton also for the decson of hostng the fle or not. TABLE III STATISTICS & EVALUATIONS FOR THE MOST FAMOUS COUNTRIES BY YOUTUBE. OF USERS ACCESS YOUTUBE(X) INTERNET USERS (MILION)(N) YOUTUBE VIEWERS/ MONTH TOTAL TIME(MIN) (AV = 22.9 M/D/U) Australa UK Canada Ireland Brazl France Japan Inda USA Germany Span Russa PROBABILITY OF YOUTUBE USERS(X/N) Authorzed lcensed use lmted to: Telecom and Management Sud Pars. Downloaded on January 21, 21 at 7:38 from IEEE Xplore. Restrctons apply.

6 B. Traffc and User Socal Behavour From table III, we can see that; the traffc of YouTube has a great effect n attractng the vewers. We notced n one hand, the average tme for each user spent per day s 22.9 mnutes. Ths tme s not a small perod f we make comparsons wth other stes lke news or mals stes. Therefore, all nternet researchers expect ths tme wll ncrease dramatcally n the near fve years. On the other hand, a lot of countres whch have a large percentage of accessng YouTube lke USA not by default gve us an ndcaton that t s the country that has the maxmum percent of utlzaton n terms of the probablty of YouTube. We need to take nto account the total number of nternet as a base for that calculaton as the results appeared n the last column of the table III above. The socal relaton between YouTube vewers and vdeos s a sgnfcant relaton but, we can not put t under any mathematcal equaton. The number of vstors to YouTube as t appeared n the table III has no relaton f we compared t wth the last column of the table (the probablty of YouTube (x/n)). For example, f we compare USA wth Ireland we can fnd that; the vewers probablty for Ireland s 25:1.4 to USA. Although the number of nternet n USA s 22.1 M: 2.4 M to Ireland whch almost rato 1:1. VII. RELATED WORK Many solutons have been proposed for fle allocaton for vdeos and other heavy loaded servers and data over the Internet by cashng fles lke AKAMAI systems [11], whch facltes a lot for vdeo delvery and streamng cashng soluton. But, the research stll tres to fnd soluton desgn for servers and fles allocaton procedure wth low cost. Many measurements and statstcs, lke the study n [9] have been carred out to show the grand utlsaton of socal networks and the ncreasng numbers uploaded or downloaded vdeos through nternet usng famous stes lke YouTube or Dalymoton. Those latter are the most famous sts. When you come to understand ther traffc pattern, YouTube s a sgnfcant web ste even among all web 2. stes. Ths s due to the fact that content of YouTube s vdeo that consumes much more bandwdth than text, pcture and audo. Some statstcs regardng the network of the Unversty of Calgary and YouTube would be useful. The unversty has 28 students and 53 faculty and staff. The data for ths study was collected n 85 consecutve days n sprng 27. It turns out that YouTube traffc consttutes 4.6 of the whole nternet traffc of the unversty. YouTube s the most popular vdeo-sharng web ste and t s the source of 6 of the vdeos watched over the nternet. 1,, vdeos are downloaded to watch every day and 65, new ones are uploaded. There s another work whch presented a systematc and n-depth measurement study on the statstcs of YouTube vdeos [1]. When authors analyzed these statstcs, they found that YouTube vdeos have notceably dfferent statstcs from tradtonal streamng vdeos, n aspects from vdeo length to access pattern. They also studed some new features that have not been examned by prevous measurement studes: the growth trend and actve lfe span of vdeos. In [16], the authors tred to analyze the vdeo and the user characterstcs for dfferent geographcal regons, concentratng manly on Latn Amerca. They developed an effcent way for collectng data about vdeos and. Based on the collected data, they showed that there exsts a relatonshp between geography and the socal network features avalable n YouTube. They presented evdence that ndcates that geography creates a localty space n YouTube, whch could be used to explore nfrastructure mprovements, such as cachng mechansm and content dstrbuton networks. VIII. CONCLUSION The feld of socal network desgn and short vdeos delvery lke YouTube becomes an mportant research doman. In ths paper, we tred to show the hstory and analyze some statstcs related to one of the most popular vdeo delvery; YouTube.com. We also focused on the socal network desgn for such types of servers and presented an optmzaton for fle allocaton that gave us two contrbutons. The frst contrbuton moves fles from server to server accordng to the mnmum cost between zones and servers. The second contrbuton lnks hts on some servers wth fle allocaton and geographcal dstrbuton of servers. We also optmzed the revenues from fle hostng and fle movements. As a future work, we hope to optmze the allocaton of fles and ther dstrbuton n case of Peer to Peer networks under complex condtons lke the hdden or out of work of one node n the logcal network for Peer to Peer desgn. IX. REFERENCES [1] YouTube Members: [2] YouTube Press: [3] Google Trends: [4] Google Analytcs: [5] 1-mllon-us-vewers. [6] Google Blog: [7] ComScore Press: [8] Alexa: [9] P.Gll, M.Arltt, Z.L, A.Mahant, YouTube Traffc Characterzaton: A Vew from the Edge, ACM Internet Measurement Conference (IMC) San Dego, CA, USA October 27. [1] X. Cheng, C. Dale, J. Lu, Statstcs and Socal Network of YouTube. IWQoS th Internatonal Workshop on Vdeos, Qualty of Servce, IEEE 28. [11] AKAMAI : [12] Yahoo: [13] Daly Moton: [14] [15] PC World: [16] F.Duarte, F.Benevenuto, V.Almeda, J.Almeda, Geographcal Characterzaton of YouTube: a Latn Amercan Vew, Web Conference, 27. LA-WEB 27. Latn Amercan, Oct Nov.2 27 Page(s): [17] YouTube currently streams more than 3 bllon vdeos per month worldwde, [18] 7MllonIn29Analysts.php [19] 2 Hours of Vdeo Uploaded Every Mnute, Authorzed lcensed use lmted to: Telecom and Management Sud Pars. Downloaded on January 21, 21 at 7:38 from IEEE Xplore. Restrctons apply.