IJIRST International Journal for Innovative Research in Science & Technology Volume 2 Issue 11 April 2016 ISSN (online):

Transcription

1 IJIRST International Journal for Innovative Research in Science & Technology Volume 2 Issue 11 April 2016 ISSN (online): Web Grabber Prof. Shilpa Nimbre Professor Research, Kopri, Thane (E) , India Divyesh Ahir Student Research, Kopri, Thane (E) , India Gopal Patel Student Research,Kopri, Thane (E) , India Suresh Nadar Student Research, Kopri, Thane (E) , India Abstract The proposed project is a high-speed, multi-threading website download and viewing program. The program can quickly extract website and/or some web pages of a site including HTML, graphics, Java Applets, sound and other user definable files, and saves all the downloaded or extracted data in the hard drive. After downloading, all links within the web pages are recreated. It creates a complete hard drive copy of the site that you can view at your own place without being connected to the Internet. Web Grabber duplicate the original directory format of a site so that it easy to download and easy to move from a site to another server. In the event a person loses his Internet connection while downloading a site, the Resume Session allows us to pick up a session. If we wish to update a previously downloaded site, the Update Session feature allows us to revisit a site using new search parameters to make sure we have the most current files. Keywords: Applets, multi-threading, HTML I. INTRODUCTION Internet brings the world in our hands and now a day it is very easy to learn anything using internet connection. If we are using a website frequently each time we have to switch on the internet connection to access the site and it also take time to load a website again which is depend on speed of Internet connection. Surfing in online is a very useful one, but sometimes it will make us to waste lots of time. We can check the site without internet connection by simply downloading the entire site. Crawlers are also known as Spiders. It is an automated program specifically run by the search engine system. Spider visits a web site, read the content on the actual site, the Meta tags of that sites and also collect as well as extract the links that the site connects. The spider then gives all extracted information back to a central repository, where the data is indexed. It will visit each link of the website and index those web pages too. The spider will frequently come back to the sites to test out for any information that has updated. The frequency with which this happens is determined by the moderators of the search engine. A crawler is like a book where it contains all the records with the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may possibly index up to a million pages a day. Web Grabber is like a Web Crawler which restricted to surf only one server. Using Web Grabber, we can download the entire website and access the website without internet connection which save lots of time. The purpose of the System is that download the page or entire site easily with the images and other files as pages for offline viewing. Simply make a copy of the website in local disk, and can browse the site. The proposed project is a high-speed, multi-threading website download and viewing program. By sending multiple simultaneous requests to surf, it download entire website or some of the web pages including HTML, images, Java Applets, sound and other user files that are on the requested server, and saves all the files in the hard drive in their native format. II. EXISTING SYSTEM Distributed Ontology-Driven Focused Crawling. A focused crawler is a web crawler that collects Web pages that satisfy some preset specific condition or property, by prioritizing the crawl frontier and managing the hyperlink exploration process. Few of the predicates may be based on simple, deterministic and surface properties. URL Rule Based Focused Crawler Crawlers usually perform some type of URL normalization in order to avoid crawling the same resource more than once. URL normalization also called as URL canonicalization. It refers to the process of standardizing and modifying a URL in a proper All rights reserved by 48

2 manner. There are various types of normalization that may be performed including transformation or conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component. Efficient Focused crawling based on Best First Search. A focused crawler implements a strategy that associates a grade with each link in the web pages it has downloaded. The links are sorted according to the grades and inserted in a queue. A best first search is implemented by popping the next page to analyze from the front of the queue. This strategy ensures that the crawler preferentially pursues promising crawl paths. III. PROPOSED SYSTEM Types of Crawlers Focused Web Crawler: - Focused Crawler is the Web crawler that tries to download pages with some sort of similarities and also that are related to each other. It extract data or documents which are relevant and specific to the given topic. It is also known as a Topic Crawler. The focused crawler determines the following Relevancy, Way forward. It decide how long or how far the given page is relevant to the particular topic and how to proceed forward. The advantages of focused crawler is that it is economically feasible in terms of hardware and network resources and also it can reduce the considered amount of network traffic. The search exposure of focused web crawler is also huge. Distributed Crawler: - Distributed web crawling is a distributed computing technique. Many crawlers are working to distribute and contribute in the process of web crawling, in order to have the most freshness of the web. A main server manages the synchronization and communication of the nodes, as it is geographically distributed. It basically uses Page rank algorithm for its increased efficiency and quality search. The benefit of distributed web crawler is that it is robust against system crashes and other events, and can be adapted to various crawling applications. Incremental Crawler Architecture Fig. 1: Architecture All rights reserved by 49

3 Fig. 2: Algorithm The crawler visits the web till the collection has a considerable number of pages, and then it will stops visiting pages. When there is some update or necessary to refresh the collection, the crawler create a new collection using the same process as above, and then crawler replaces the old collection with this new collection. This type of crawler is known as Periodic crawler. The crawler may keep surfing pages after the collection reaches its predefined target size, to incrementally update or refresh the local collection. By this incremental update, the crawler refreshes existing pages and replaces less-important or less-priority web pages with new and more-important pages. Crawler operated in this mode is known as Incremental crawler The URLs in CollUrls are selected by the Priority Module. The Priority Module continuously scans through AllUrls and the Collection to make the refinement decision. When a page not in CollUrls turns out to be more important than a page within CollUrls, the Priority Module schedules for replacement of the less-important page in CollUrls with that more-important page. The URL for this new page is placed on the top of CollUrls, so that the Update Module can crawl the page immediately. Also, the Priority Module discards the less-important page from the Collection to make space for the new page while the Priority Module refines the Collection; the Update Module maintains the Collection fresh (update decision). It constantly extracts the top entry from CollUrls, requests the Crawl Module to crawl the page and puts the crawled URL back into CollUrls. To estimate how often a particular page changes, the Update Module saves the checksum of the last crawl page and compares that checksum with the one from the current crawl. The Update Module from the above comparison can tell whether the page has changed or not. The Crawl Module crawls a page and updates /saves the page in the Collection, based on the request from the Update Module. All rights reserved by 50

4 IV. WORKING Activity Diagrams Fig. 3: Grabber Process Fig. 4: User Process All rights reserved by 51

5 Class Diagrams Fig. 5: Class Diagram Admin module The Administrator can do the following actions: Admin Actions 1) Enter into the site 2) Enter the URL 3) Identifies the Site 4) Identifies the Layers 5) Start Download 6) Saves the Downloaded site Features a) The Purpose of this Tool is to Grab the complete Website and Download and View it in from the hard disk without intervention of Inter Net. b) Web Grabber Dramatically increases the productivity by providing powerful, fully-automated Web Data extraction. Users can rapidly find, capture, and store any information from the website. c) Web Grabber collects both static and dynamic contents from the web pages, and its fully-customizable, to ensure fast and simple extraction from any target website. Advantages a) Web grabber will help business men extract and collect the market figures, product pricing data, or real estate data. b) Web grabber will help book lovers extract the information about books, including their titles, authors, descriptions, ISBNs, images and prices from online book sellers. c) Web grabber will help the journalists extract the important news and articles from news sites. d) Web grabber is useful for the all types of people for extracting the data with images, charts and other things. e) Web grabber is useful for the people seeking a job extract, job postings from online job websites, he can find a job faster and minimum inconveniences. V. CONCLUSION In this study, we get to know what web grabber is, how it works, and what are the advantages are available in this system. This system is very useful for all types of people for extracting the data, files, images etc. on any format from any website. We can grab single page or total site using this software. In Web grabber we can download with a code of that page. This is advantageous while designing a project, when anybody wants to go through the similar features project. Web grabber software installation is easy on any type of system and it will easily install in system. Only we have to specify the URL, and this smart utility explores, extract the entire web site and downloads all the images, data and files automatically, that meet our specifications to a specified folder. This will save lots of amount when we want to download large amount of data in the web site. All rights reserved by 52

6 ACKNOWLEDGMENT No project is ever complete without the guidance of those expert how have already traded this past before and hence become master of it and as a result, our leader. So we would like to take this opportunity to take all those individuals how have helped us in visualizing this project. We express our deep gratitude to our project guide Prof. Nilima Patil and co-guide Prof. Shilpa Nimbre for providing timely assistant to our query and guidance that she gave owing to her experience in this field for past many year. She had indeed been a lighthouse for us in this journey. We would also take this opportunity to thank our project coordinator Prof. Sulochana Madachane for her guidance in selecting this project and also for providing us all this details on proper presentation of this project. We extend our sincerity appreciation to all our Professors from K.C. COLLEGE OF ENGINEERING & MANAGEMENT STUDIES & RESEARCH for their valuable inside and tip during the development phase of the project. Their contributions have been valuable in so many ways that we find it difficult to acknowledge of them individual. We also great full to our HOD Prof. Asmita Deshmukh for extending her help directly and indirectly through various medium in our project work. REFERENCES [1] Java, [2] Herbert Schildt, Java Complete Reference, 5th edition [3] N. Eiron and K. S. McCurley, Analysis of Anchor Text for Web Search, IBM Almaden Research Center. [4] Finding what people want: Experiences with the WebCrawler, Pinkerton B., In Proceedings of the First World Wide Web Conference, Geneva, Switzerland, [5] Computer weekly web site, Apr [6] Different googles yahoo - WEB stats domain overview for keyword.[online].available: [7] Junghoo Cho, Hector Garcia-Molina, and Lawrence Page. Efficient crawling through URL ordering. In Proceedings of the 7th World-Wide Web Conference, [8] Lawrence Page and Sergey Brin. The anatomy of a large-scale hyper textual web search engine. In Proceedings of the 7th World-Wide Web Conference, All rights reserved by 53