ANALYSIS OF THE WEB, PROCESSOR SPEED AND BANDWIDTH GROWTH: IMPACT ON SEARCH ENGINE DESIGN

Transcription

1 ANALYSIS OF THE WEB, PROCESSOR SPEED AND BANDWIDTH GROWTH: IMPACT ON SEARCH ENGINE DESIGN K. Satya Sai Prakash Network Systems Laboratory IIT Madras, Chennai India Phone: ssai@acm.org S. V. Raghavan Network Systems Laboratory IIT Madras, Chennai India Phone: svr@cs.iitm.ernet.in ABSTRACT World Wide Web (web) is perceived as an unstructured and uncontrolled system. This paper explores the web growth, processor speedup, and bandwidth growth over the time and proves that the web in its entirety is not an uncontrolled system that explodes indefinitely. Critical observation and analysis of the web, processor speedup, and bandwidth growth, prompted us to conjecture that the Internet information processing index (I) is constant. Global tendencies in WWW/Internet indicate that huge data is getting accumulated on the web. User base too is growing at a rapid pace apart from the bandwidth and processor speeds. In the present web scenario, processor speed (S), bandwidth (B), and the volume of the web data (V) are the three vital parameters. We made careful study and conjectured that I = k (B * S)/V The above relation provides a control over the web content organization and the access. This study makes an invaluable contribution in the design of Search Engines that need to skim through the huge volumes of web data (part of which dynamically changes), design of audio, video-rendering systems over Internet and in establishing ISPs and the client network architecture designs. KEYWORDS Moore s Law, Neilsons Law, Search Engine, Internet 1. INTRODUCTION Since the inception of the concept of Internet, the number of systems that are coming into the purview of the Internet is growing exponentially. World s computing and communication environment is under the hood of the web. Web is expanding exponentially with the content created by the increasing number of users. All possible information is posted on the web today. Be it a riddle, game, report, news item, recipe and what not; think of an item, it can be found on the web. Person or a thing can be searched and contacted over web. id has become an alias to every individual. At this juncture it is important to understand the advancements in the silicon industry, so that we know the computing system s processing ability and communication systems speedup [4]. Understanding the global tendencies of Internet/WWW is very essential to design the infrastructure for a clientele or ISP. This also helps in designing real-time interactive application networks. Certainly it plays a

2 role in designing the new generation search engines that need to index all the web data to yield recent and relevant results as a minimal possible result-set with minimal/optimal infrastructure. Gordon Moore made a detailed observation about the processor speedup and gave his law [1][2], as The number of transistors integrated on leading edge circuits would continue to double every 18 months until the fundamental physical limits are reached. Figure 1. Moore s Law Neilson s law [4] states that a high-end user's connection speed grows by 5% per year. Parkinson s law states that data expands to fill the space available for storage and the storage capacity is doubling every 18 months. Figure 2. Neilson s Law Also in [6] the authors observed that the web is doubling every 14 months. Web Pages (millions) Figure 3. Web Growth Data is fit into the curve y = 12.83x x 2 + 1E+8x 1E+11 with R 2 =.9989

3 2. CONJECTURE The Internet information processing index (I) is constant. In this context we have three vital parameters V (Volume of the Web data), B (bandwidth over the Internet), S (Processor Speed). It can be seen that, I B (S and V constant), I S (B and V constant) and I 1/V (B and S constant) Hence I = k B * S / V, k is a constant of proportionality. From the observations made in the previous section, we noted that the annual growth of B, S and V are 5%, 66.67% and 85.71% respectively. Then k is about {I 1 = B*S/V, with k = 1, then I 2 = (r 1 B+B)*(r 2 S+S)/(r 3 V+V) where r 1, r 2, r 3, are annual growth rates of B, S and V respectively. Then I 2 = [(1+r 1 )*(1+r 2 )/(1+r 3 )] B*S/V, i.e. I 2 = k I 1, where k = (1+r 1 )*(1+r 2 )/(1+r 3 ). Substituting r 1 =.5, r 2 =.667 and r 3 =.857, we get k = 1.35} In general, I n = [B(1+r 1 ) n ]*[S(1+r 2 ) n ]/V(1+r 3 ) n, I n = k (B*S)/V. That implies, k = (1+r 1 ) n *(1+r 2 ) n /(1+r 3 ) n, where n is number of years. This explains the improved rendering of the web data generally experienced by the end user, when there is infrastructure enhancement. More effort is being made to study the other possible relations that could exist between three vital web parameters, like, I = k 1 B S/V Where B S is the number of bytes received at the available bandwidth B that are processed by the processor with speed S. Also the non-linear relation ship between the parameters is given as I = k 2 B * S /V Memory (M) growth is not taken into consideration in view of its rapid growth in comparison with bandwidth growth. Equilibrium of data processing system depends on min {B, M}. 3. CASESTUDY GOOGLE Google s initial infrastructure as a research product was described in [14]. In 1997, they started with 4 machines downloading and analyzing 26 million pages. Today it runs on 1 Linux-cluster indexing about 3 billion documents and responding to 25 million queries a day. Google is keeping pace with the growing web in indexing the web. The following plot illustrates that the computing systems growth is at a constant rate at Google. No. of Machines Figure 4. Google s Computing Resource Growth Curve fitting exercise yielded that y = x 4E+6, with R 2 = 1 First derivative of the above curve gives us the rate of change. dy/dx = d/dx(1999.2x 4E+6) =

4 Hence Google is procuring 2 Linux boxes a year to keep pace with the growing web and user queries. Google s document indexing capability grows exponentially since It can be seen in the following plot. Web Pages (millions) The above plot fits into the curve, y = x x 2 2E+8x + 1E+11 with R 2 = 1 Figure 5. Google s Web Indexing Rate Figure 6 shows the increase in the Google s popularity in terms of increased queries per day. Queris Per Day (millions) y = x x 2-3E+7x + 2E+1 R 2 = Figure 6. Google s Query Hits With reference to the above conjecture take a look at how the Search Engines have performed over 3 years [5]. The following pattern is obtained and observed. Figure 7. Google s Performance Sm Summer, F Fall, W Winter, Sp Spring (No survey was reported by NPD in Summer 98).

5 The average successful search rate is 8% and the variation is about 5% that is consistent with the above stated conjecture. 4. OBSERVATIONS AND DISCUSSION Observing the growth of the web, processor speed and bandwidth and calculating as per the above formulation we get the following: When the Bandwidth (B) and Processor Speed (S) are Kbps and KHz and the web data is in the order of Mega bits. So we can see that B*S/V is O(1). As B and S leap into Mbps and MHz, web data is exploding into Tera bits again conforming to O(1). From this intuition, we conclude that I = k. (since B * S/V is tending towards 1). It is seen from the plot (Fig. 8) that the growth is consistent with the observations made earlier B (Mbps) S(MHz) V(Millions of pages) Figure 8. Growth of Processor Speed, Bandwidth and Web Detailed best-fit exercises yield the following plots for processor speedup, bandwidth growth, and web growth. Processor Growth: 16 y =.2517x x 3 + 6E+6x 2-8E+9x + 4E R 2 = Figure 9. Growth of Processor Speed

6 Bandwidth Growth: y = x x + 1E+8 R 2 = Figure 1. Growth of Bandwidth Web Growth: y = x x 3 + 3E+7x 2-4E+1x + 2E+13 R 2 = Figure 11. Growth of Web With bandwidth (B), Processor speed (S) and Memory (M), web data processing and rendering are done at the following rate. Let each update size be u b bytes. No. of updates per day: N No. of cycles per update to process: u c per update Also, each query size: q b bytes No. of queries per day: M No. of cycles per query to process: q c per query Then the system is under control only if, N * u b + M * q b min {B, M} and N * u c + M * q c S, Where N = N/24*6*6 and M = M/24*6*6. Here updates indicate data processing and queries indicate data rendering. Conjecture can be looked with reference to the subsystems, as the inherent processing speed, bandwidth vary at large from a sub system to subsystem. Consider the following typical client server communication across Internet.

7 Web Server B S B C Client S C S S B I Internet S S Web Server speed B S Web Server bandwidth B I Internet bandwidth S C Client speed B C Client bandwidth Figure 12. C - S Communication across Internet Here we can find subsystems like Server, Client and Internet with varying bandwidths and processor speeds. The data processing index is given by Data Processing Index (I D ) = Data Rendered/ Total Data (in KB) 5. SARVAGNA NEXT GENERATION SEARCH ENGINE In this section we give an over view of the prototype new generation search engine we are building. We compare and contrast the design and architecture of Sarvagna with Google. SARVAGNA is a Scalable, Available, Reliable, Versatile, Adaptive, Global, Novel, and Accurate new generation Search engine. It is built in accordance with the generic architecture [1] and design standards [11]. Its data collection, processing and rendering are based on push based protocols proposed in [9][12]. Complete functional and implementation details of SARVAGNA will be made available in a technical report. Figure 13. SARVGNA User Interface

8 Table 1. Architecture, Design and Implementation Issues Issue Google Sarvagna Architecture Single Unit Decoupled Units (IIE & RRE) Infrastructure 1 Linux cluster Itanium II Web Indexing Employs Robots. (PULL) PUSH based Protocols Recency 28days Periodicity Instantaneous Relevancy Pigeon ranking and Link Popularity K q -K f based Technique Implementation C/C++ Visual Studio.Net Aim of this brief section is to introduce the reader to a next generation search engine that is architecturally different from the existing search engine crop and whose design is adaptive to the web dynamics. 6. CONCLUSION Though web is a growing entity, we can perceive that a controlling law is governing the web growth, processor speedup and the bandwidth growth. Global tendency of Internet/WWW is to explode. So technological innovations, advancements in electronics and enhanced processor speed and bandwidth can not reduce the total access time of the web information because of the simultaneous rapid growth of the web. At the same time information growth is kept under check with the infrastructure enhancement. This has a pounding effect on Search Engine Technology, ISP Design and Access Network Design. We brought out a simple relation that is possibly governing the web information growth, processor speed growth and bandwidth growth. Future work is aimed at, Deriving an accurate value for the proportionality constant, k Studying the improvement of Recency and Relevancy of Search Engine based on this system. Studying the possible non-linear relationships between S, B and V. REFERENCES 1. Dr. Gordon E. Moore, April 1965, "Cramming More Components Onto Integrated Circuits", Electronics, 38(8) Special Report, September 199, "Gigabit Network Test-beds", IEEE Computer, 29(9), pp Brian E. Brewington and George Cybenko, 2, "How Dynamic is the Web?", www9/computer Networks Journal, 33(1-6), pp Michalis Faloutsos, Petros Faloutsos and Christos Faloutsos. "On Power-Law Relationships of the Internet Topology", ACM SIGCOMM, pp: , Jacob W Green, Hyper Dog: Up to date Web Monitoring through Meta Computers, MS Report, Baltimore, Maryland, October 2 9. K. Satya Sai Prakash and S. V. Raghavan, Web Recency Maintenance Protocol, Proceedings of 4 th International Workshop on distributed Computing, LNCS 2571, Springer 2, pp: K. Satya Sai Prakash and S. V. Raghavan, DIAPANGSE: Distributed Intelligent Agent based Parallel Architecture for Next Generation Search Engines, In the proceeding s of 5 th International Conference on Information Technology (CIT 22), Bhubaneshwar, India, Tata McGraw Hill, pp: K.Satya Sai Prakash and S. V. Raghavan, Workload Characterization and Performance Analysis of R 2 Protocols, Technical Report (TR-NSL-IITMadras-SAISVR-I). 12.K. Satya Sai Prakash and S. V. Raghavan, User Relevancy Improvisation Protocol, Technical Report (TR-NSL- IITMadras-SAISVR-II) Sergy Brin and Lawrence Page, April 1998, The Anatomy of Large Scale Hyper textual Web Search Engine, 7 th World Wide Web Conference, Brisbane, Australia. URL: http: //www7.scu.edu.au/programme/fullpapers/1921/com1921.htm