Performance Analysis of Book Recommendation System on Hadoop Platform

Transcription

1 Performance Analysis of Book Recommendation System on Hadoop Platform Sugandha Bhatia #1, Surbhi Sehgal #2, Seema Sharma #3 Department of Computer Science & Engineering, Amity School of Engineering & Technology, Amity University, Noida, India. Abstract- Recommendation engines are computational intensive and hence ideal for Hadoop Platform. This research paper aims at building a book recommendation engine which uses data mining for recommending books. It will give its users the ability to upload and download engineering books as well as novels which will be used to draw out conclusions about the stream of a user and the genre of the books liked by that user. It will analyze the user behavior by making use of content based filtering and will apply association rules in data mining for displaying individualized recommendation system of books. This database will then be transferred to the Hadoop HDFS so that a comparison between the read and write efficiency between the local system and Hadoop can be made. Keywords - Association rules, Collaborative filtering, Content based filtering, Data mining, HDFS, Hadoop, Map Reduce, Recommendation engine. 1. INTRODUCTION Hadoop is a large scale distributed batch processing infrastructure. It allows for the distributed processing of large data sets called big data across clusters of computers using Map Reduce and HDFS file system.hadoop can be used with data mining applications. It makes possible the storage of unstructured data i.e. images in a structured form by making use if it s file system and other projects that work with Hadoop like HBase, Hive, and Pig etc. In this book recommendation engine the focus has been on performance analysis of the application on the local system and then compare it with the performance of the recommendation system when it is migrated to Hadoop. Hadoop provides a mechanism to analyze the performance by making use of its inbuilt plug-in so no additional resource is required. In the book recommendation engine, books will be displayed according to the readers preferences in a hierarchical way to categorize readers interest in different genres, the users pattern of downloading different engineering books and to form an effective set of rules based on that. New books will be appropriately presented according to users needs. Based on users interest and books properties, a book recommendation system will be created. Furthermore, it gives the facility to upload and download the books and save the books in the wish list if the book is unavailable or user wants to download it later. Star rating of each book is provided along with which is based on the concept of data mining. User is allowed to rate the book while downloading it or adding to his wish list. The whole dataset is then transferred to the Hadoop HDFS so that a comparison between local and Hadoop system can be made and the use Hadoop in the recommendation system can be justified 2. UNDERSTANDING HADOOP The Apache Hadoop software library is a framework that allows for the processing of big data across clusters of computers using simple programming models in a distributed manner. It can scale from a single server to thousands of machines, each of which offers a local processing and storage. Rather than relying on hardware components to deliver high availability the Apache Hadoop software library is designed to detect and handle failures at the application layer so it delivers a highly available service on top of a cluster of computers each of which may be prone to failures. [1] Hadoop makes it possible to run applications on systems with thousands of nodes involving terabytes of data. Its distributed file system called HDFS facilitates rapid data transfer rates among nodes and allows the system to continue to work interrupted in case of node failure. When a record has to be searched in terabytes of data, it will take a lot of time to retrieve the record but using Hadoop one can work in parallel and make the computation as well as searching faster. Hadoop framework can be used for log analysis, marketing analytics, data mining, image processing, web crawling etc. 2.1 HDFS IN HADOOP HDFS is a distributed, scalable, and portable file system which is written in JAVA for the Hadoop framework. The only source of input to the Hadoop framework is through its HDFS. HDFS stores large files of approximately 64MB each. It achieves reliability by replicating the data among multiple hosts. The NameNode is the central part of an HDFS file system. It keeps the directory tree of all the files in the file system and tracks the presence of file data across the cluster. Hadoop file system includes the secondary name node which connects with the primary name node to build snapshots of the primary namenode s directory. The DataNode is responsible for storing data in the HDFS. The job tracker schedules the map or reduces jobs to task trackers with an awareness of the data location. Similarly, the task tracker accepts task Map, Reduce and shuffle from JobTracker. On start-up, a DataNode connects to the NameNode. It then responds to requests from the NameNode for file system ISSN: Page 1037

2 operations. Client applications can talk directly to a DataNode once the location of data has been provided by the Name Node. The JobTracker then schedules the Map Reduce jobs to the TaskTracker. MapReduce operations allocated to the Task Tracker instances near a DataNode that talk directly to the DataNode to access the files. TaskTracker instances can be deployed on the same servers that host DataNode instances so that MapReduce operations are performed close to the data. Fig. 2.1 HDFS operations 2.2 MAP REDUCE IN HADOOP Hadoop makes use of Map Reduce programming model for processing large datasets and distributed computing on clusters of computers. Map Reduce take the advantage of locality of data for processing data on or near the storage assets to decrease transmission of data. It is a two step process namely Map and Reduce. In the Map process, the master node takes the input from the HDFS file system and divides it into smaller sub-problems and distributes them to worker nodes. The worker node may divide it again and process the smaller problem, and pass the answer back to its master node. Mapping process is performed by Mappers using key-value pair. In the Reduce process, the master node collects the solution to all the sub-problems and combines them in some way to form the solution to the original undivided problem. Reduction is performed by Reducers. The only mode of intercommunication between the nodes is during the shuffling process. matching it with user profile to give a future recommendation. The amount of data that is there in the Book recommendation system is a big problem. To display search results from a million users and to choose similar interest of a million user adds to latency in the search and degrades the performance of the Recommendation system. As recommendation systems are computational intensive, a vast amount of data needs to be searched when search results have to be displayed. It becomes difficult to store such a large amount of data. Furthermore, storing books in a relational form is a big problem as books and images are an unstructured form of data. Books and images needs to be stored as links in the relational database and retrieving results from links adds to more latency in the recommendation system. Book Recommendation Systems help users in managing their reading list by learning their preferences. There are two categories of such system, one which gives a list of recommendation based on user profile in a library automation system and others which tells user specifically what should he read next according to the current requirements. There are recommendation systems like whichbook.net, what should I read next, lazy library, library think etc., each of which uses a particular strategy to fulfill current requirements of the user. Current recommendation methods mainly use two approaches: collaborative filtering method [7] and content-based method [8]. The collaborative filtering method is based on how other customers have rated a book on average. Thus, the collaborative filtering method can be used to recommend books without effective book information, if sufficient user feedbacks (ratings) are available. However, this approach is not suitable for recommending unpopular books that readers do not rate. On the other hand, the content-based method recommends books highly evaluated by the recommendation systems using a user s feedback. The content-based method is useful for finding unpopular books when other users feedback is unavailable but effective book information is available. For example, in case a user has a book about Algorithm and Data Structures and the book is rated by him, the content-based recommendation can be applied. However, this approach may not be suitable for finding new category books because this method depends on each user s interests. These two approaches are better choice for promoting book purchase, but they are not always suitable for book reuse. [3] Fig. 2.1 Map Reduce mechanism 3. CURRENT RECOMMENDATION SYSTEMS The Book Recommendation Systems implemented so far have either used similar user interest or item characteristic for 4 PROPOSED SYSTEM Book recommendation system has been developed rapidly due to the Web technology, which provides a new way to acquire the reader s demands. However, existing recommendation systems don t analyze the recommendation information and can't supply enough information for readers to decide whether to recommend a book or not. Some systems also lack a feedback mechanism for readers which would not satisfy the needs of the readers. In order to solve these problems, a user s interest based book recommendation system has been proposed. The recommendation pages will contain all the ISSN: Page 1038

3 essential and expanding book information for readers to refer to. Readers can rate the book of their choice, and the star rating data from different user with similar patterns will be analyzed by the recommendation system to make scientific recommendation decision. The application of the recommendation system shows that both the recommended book utilization and readers' satisfaction were greatly increased. Fig.4.1: Proposed Model of book recommendation service The data input to the book recommendation system is the engineering books as well as novels. The books input to the book recommendation system through the database which is maintained in database. Data preprocessing describes any type of processing performed on raw data to prepare it for another processing procedure. Hadoop performs pre processing by automatically clustering the whole data into clusters depending upon the kind of data. Association rules are if/then statements that help uncover relationships between seemingly unrelated data in a relational database or other information repository. Recommendation is based on the user selection. If a customer selects an engineering book of a particular stream then he will be appropriately recommended other engineering books of the same domain. Example: In case a customer selects Digital image processing book by Gonzalez woods then the customer will be appropriately presented other books of Digital image processing by the other authors. Similarly, if a person selects a novel to download, then the customer will be recommended books by the same author and books of the same genres. Finally, the performance of the local system is compared with that on the Hadoop platform. The book recommendation system will be using single node Hadoop cluster to store data in HDFS. Hadoop will allow the distributed processing of large data sets across clusters of computers using Map Reduce and HDFS file system. In this book recommendation engine the focus will be on performance analysis of the application on the local system and then compare it with the performance of the recommendation system when data is read or written from HDFS. A near approximation of Big Data is desired for best results. The larger the data the better will be the performance of the recommendation system on Hadoop.Hadoop provides a mechanism to analyze the performance by entering the URL of the NameNode and JobTracker dashboard in the web browser. The NameNode runs of the port and the JobTracker runs on the port number by default.in the book recommendation engine, books will be displayed according to the readers preferences in a hierarchical way to categorize readers interest in different genres, the users pattern of downloading different engineering books and to form an effective set of rules based on that. New books will be appropriately presented and classified in order to satisfy the needs of readers. Furthermore, it gives the facility to upload and download the books and save the books in the wish list if the book is unavailable or user wants to download it later. Star rating of each book is provided along with which is based on the concept of data mining. User is allowed to rate the book while downloading it or adding to his wish list. Appropriate recommendations will be presented simultaneously.some of the proposed features of the Book Recommendation System are 1) Performance Analysis of the application on Hadoop 2)Easy management of the unstructured data by using Hadoop 3)Cluster formation implicitly done by Hadoop 4)Content based filtering 5) Advance Book Recommendation 6) Customized web pages 7)Explicit data collection by asking a user to rate a book and create a list of books that he likes 8)Implicit data collection by observing the books that user view and by keeping the record of the books that user download 9)Analyzing performance of the application on local machine and on Hadoop platform.10) Ability to upload and download a book. 4.1 MODEL OF BOOK RECOMMENDATION SYSTEM User Interface User Information Book Information Data Mining Profiles Database Antecedent Of rule Recommendation Module Subsequent of rule Book Database Subsequent Of rule Book Information Fig. 4.1: Model of Book Recommendation System The model of book recommendation service is made up of data mining, rule database, user interface, rule matching, and book database. Fig. 4.1 shows the methodology adopted in the paper. Firstly the user gives input to the book recommendation system which is in the form of either user details or the name of the book with or without author name. The input given by the user is then processed by the book recommendation system module and the desired books are fetched from the database and displayed to the user. Now the user can download the book of their choice after rating the book. Once ISSN: Page 1039

4 the user gives his star rating, then the association rules are implemented based on the user rating to display him other related books of the same author or other user with similar rating patterns. The association rules produced by data mining module are used to analyze user s interest. The data mining technology is applied on the profiles database where the star rating and downloading patterns of all the users are stored. Recommendation module finds the appropriate match from the books database based on the analysis of the profile database according to user s information, and then output some book recommendation information by searching in book database according to the subsequent of selective rule, which may of user interest. Recommendations are also made on the basis of the same author and same domain of the book selected by the user. At the backend the user s data in the database is analyzed in order have better understanding of the user s interests. Every time when the new user rates a particular book, its average rating is calculated and stored in the database. So our book recommendation system uses the combination of both content base and collaborative filtering to have better understanding of user s interest DATABASE DESIGN The database design of application is as below. Table: Books_Details and Novel Book_id Auto Number Gives the Book Id of the book Book_Name Text Gives the name of the book along with author Name Text Gives the book name Author Text Gives the book author Stream Text Gives the book stream Domain Text Gives the book domain Star_Rating Number Gives the star rating of the book Book_image Text Gives the book image Book_pdf Text Gives the book pdf Table 4.1: Books_Details and Novel Table: User_Details Username Text Gives the username Password Text Gives the password _address Text Gives the id of user birth_date Text Gives the birth date of user Table 4.2: User_Details Table: Profiles User_Id Text Gives the User Id of user Downloads Text Gives the books downloaded by user Uploads Text Gives the books uploaded by user List Text Gives the list of books in MyList Star_Rating Text Gives the start rating of book Table 4.3 Then the dataset is uploaded to the Hadoop HDFS by converting the data in the datasheet into the CSV format and then uploading it to the HDFS. 5. EXPERIMENTS AND ANALYSIS To validate the performance of the proposed Book Recommendation System and analysis based on Hadoop, this paper implements the platform and design several scenarios for testing. The scenarios conclude data upload, data query, and data statistics. The data upload is operated on HDFS, while the data statistics are observed by the Job Tracker and NameNode dashboard which runs in the web browser. This paper simulates a Book Recommendation System on Hadoop platform. The platform maintains multiple items pertaining to users like customer name, customer password, book downloaded, book uploaded, star rating etc. The data is stored in database. Database includes more than 2000 of books and the images corresponding to the books. A database of 1000 customers has been made This paper deploys single node Hadoop cluster with CPU of Intel core i5, 2.93GHz, memory of 2 MB and hard drive of 500G. The OS is Windows 7 (64bit), the version of Hadoop is hadoop with a hadoop plugin for eclipse Europa 3.3. As Hadoop runs only on Linux systems so Cygwin has been used which provides a Linux like environment and runs within windows.the tomcat is apache-tomcat , and the Java is jdk 1.6.0_17. Meantime, MsAccess is chosen as reference RDBMS. Same operation is conducted in database on identical environment to make a comparison. Counter Map Reduce Total HDFS 1.83 MB MB bytes read HDFS MB 1.57 MB bytes written Local bytes MB 1.93 MB read Local bytes 1.95 MB 1.93 MB 3.88 MB written Fig. 5.1: Map Reduce results ISSN: Page 1040

5 Fig. 5.1 gives a comparison of running an application on Hadoop and the local system. When the application is run on the Local system the number of bytes requires to read is 1.93 MB while when read operation is performed from HDFS system it is 1.83 MB. There is a difference of ( = 0.1) MB which is approximately 5 %. This is when there is a data of around 4 MB. If the data increases the performance increases further. Similarly, the number of bytes required to write in case of local system is 3.88 MB while the number of bytes required to write in case of HDFS is 1.57 MB. There is a difference of ( =2.31) MB. The efficiency improves by %. This is the case when the system has around 4 MB of data. If this data increase the efficiency increases further. This is the reason for using Hadoop platform for Big Data. Book Recommendation System can be an apt application which has Big Data as there can be millions of users and millions of users thus demanding the use of Hadoop to improve efficiency. 6. CONCLUSION This recommendation system can meet the book demands of readers to the greatest extent; meanwhile it can recommend books to the user based on his interest. It has applied the association rules in data mining for book downloads, and designed an individualized recommendation system on book download and upload and rating patterns of the user. Simulation results show that this mining of data of books rating by the user to predict his interest has good results. All data about books and users are analyzed to identify patterns through data mining. After related user group, having similarity with target user, is classified by collaborative filtering. MapReduce and Hadoop are expected to be useful for data intensive applications. However, there is a limited understanding of the trade-off of different design choices in Hadoop application such as file system, streaming mode, networking, replication etc. Future works include bringing a near approximation of Big Data by increasing the number of books as well as the users so that the whole system has a better performance and the abilities of Hadoop can be put to the best of use. Hadoop works best with application whose data is put on the cloud because it s very difficult to keep and maintain petabytes of data on a hard disk.the next goal will be to transfer the recommendation system to cloud. REFERENCES [1] Hadoop. < 2009> [2]HDFS architecture.< docs/current/hdfsdesign.html, 2009> [3]Kuroiwa Takanori ; Bhalla Subhash, Dynamic Personalization for Book Recommendation System Using Web Services and Virtual Library Enhancements, Computer &Information Technology, 2007, 7th IEEE International Conference on Computer and Information Technology, Publication Year: 2007, Page(s): [4] T. White, Hadoop: The Definitive Guide, O Reilly Media, Yahoo! Press, June 5, [5] Konstantin Shvachko ; Hairong Kuang ;Sanjay Radia; Robert Chansler ; The Hadoop distributed File System, Yahoo! Sunnyvale, California USA, /10/$ IEEE, [6]Jin Shan ; Fan Chunfeng ; Meng Yun ; Wu Xiaohai ; Chen Qingzhang, The design of a new book auto recommendation system based on readers' interest, 2011 International Conference on Electrical and Control Engineering (ICECE), Publication Year: 2011, Page(s): [7]G.Linden;B.Smith; J.York. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing, pages 76 80, Jan/Feb [8]R.J. Mooney, L. Roy. Content based book recommending using learning for text categorization, Proceedings of the 5th ACM conference on Digital Libraries, pages , [9]Zhag,Guang-Qian ; Sun,Wei, User Preferences to attributes of books for personalized recommendation, 2012 IEEE 3rd International Conference on Software Engineering and Service Science (ICSESS),Publication Year: 2012, Page(s): [10]Maurya,M. ; Mahajan,S., Performance analysis of Map Reduce programs on Hadoop cluster,2012 World Congress on Information and Communication Technologies (WICT), Publication Year: 2012, Page(s): [11] Xu, Weijia ; Luo, Wei ; Woodward, Nicholas, Analysis and Optimization of Data Import with Hadoop, 2012 IEEE 26th International Conference on Parallel and Distributed Processing Symposium Workshops & PhD Forum (IPDPSW), Publication Year: 2012, Page(s): [12]Hadoop Overview. < ISSN: Page 1041