Implementing Advanced Cleaning and End-User Interpretability Technologies in Web Log Mining

Transcription

1 109 mplementing Advanced Cleaning and End-User nterpretability Technologies in Web Log Mining Zidrina Pabarskaite School of Computing nformation Systems and Mathematics, South Bank University, 103 Borough Road, London SE OM, UK Abstract. Two new approaches to web log mining are presented. Novel advanced cleaning improves web log mining results. mproved filtering removes pages with no links from other pages. n the data visualisation phase, technical representations of web pages are replaced by user attractive text interpretations. Experiments with the real world problems showed that the proposed techniques significantly increase the quality and usefulness of web log mining results. Keywords. Web log mining, web mining, data cleaning, knowledge representation and visualization. 1. ntroduction The web mining analysis process has evolved over the past decade. A wide range of techniques of web log mining appeared during this time [l], [6], [9]. However, some problems remain and have received little or no attention [7]. Many methods only work well on web sites having simple designs and are of limited use. The complexity of most web sites, the type and appropriate cleaning of the data set, the manner and the sample of the data used, have profound effects on the performance of web log data mining. Typical web log cleaning methodologies removes images and pictures. The analysis concerns the investigation of medidmultimedia files but there are masses of other irrelevant files, which stay untouched during all the analysis process. These can be internal web administrator actions, special purpose files, etc. The complexity of many web sites can influence the data cleaning process and impact the final results as well. t is characteristic of human nature that people want to see simple and understandable structures. Web log analysis tools do not always produce results which are easily understood by investigators. Scientist in [3] pointed the need to develop tools to help better understand mined knowledge. Mining reports generated by software applications in the final stage should be least complicated and more accessible to the majority who are interested in this information. This is the problem of visualization of results. Most of the web data is used not by technicians or web site developers, but by the people making business decisions. n this context, it is important to present results understandable to business decision makers. The new approach presented here proposes to solve visualization problem. This is done by using text tags viewed on the web site and used to define links to other pages. Using this approach, firstly, the request to the web server is send asking for the HTML page code. At the next stage, text tags are extracted. These text tags replace the classical technical URL representation used by the majority of web analysis tools. By doing this, text interpretations can be easily traced and understood by any person who is doing analysis. 2. Complex Web site structures Before data can be analysed it should pass through several pre-processing stages such as data cleaning, feature selection, transformation and etc. [6]. The pre-processing can take up to 80% of the time spend analysing the data. The quality of the final results strongly depends on cleaning process. Existing web log techniques perform rather superficial data cleaning. Web sites are created using different tools and design methodologies, therefore different types of web logs are generated. A cleaning process suitable for one type of web log data, can be irrelevant for another type and produced inaccurate results. For instance, the frame is a structure of the web site (see Fig. 1). t represents certain types of pages and consists of a number of other pages. When a request is submitted, the web browser detects the frame and additionally requests frame 24th nt. Conf. lnformafion Technology nterfaces /T 2002, June 24-27, 2002, Cavtat, Croatia

2 110 related pages. At the same time, the web server logs a number of requests into the log file instead of simply logging just one transaction. This problem has not been seriously investigated by researchers even thought it commonly arises when data is collected from frame designed web sites. This issue prompts the need of removing irrelevant frame pages from web log files. Such cleaning is important because it influences the size of the data set, the speed and the accuracy of the final results. During cleaning process, the size of the data sample is reduced by a factor of 3 to 10 times. /iidex.cfa <frameset... > <frame name="topba" src="topba.cfa"...> <frameset... > <frame name="leitmenu" src="leftnenu.cfa"...> <frame name="m;lir" src="notrecognised.cfa"...> </frameset> </frameset> ii /index.cfa name="topbar" src="topbar.cfa" ( /topbar.cfa 1 <frameset... > <frame name="leftmenu" src="lehenu.cfa"...> <frame name="main" src="notrecognised.cfa"... > name="leftmenu" name="main" src="notrecognised.cfa" =igure 1. The architecture of frame pages The picture (see Fig. 1) presents a typical us frames while developing a web site. The picture shows the HTML code, where index.cfa presents a complex structure of the web page. t consists of a new frame <frameset...>. Every frame has its own name <frame name...>. The first frame contains a page with the name topbar.cfa, the second frame splits into two different web pages: 1eftmenu.cfa and notrecognised.cfa. When the page index.cfa is requested, all three other pages are downloaded as compound parts of this frame. At the same time web server logs request of index.cfa, /topbar.cfa, 1eftmenu.cfa and notrecognised.cfa into the log file. t is very likely that just one page, for example /notrecognised.cfa, presents the desired information. 3. An advanced cleaning technique Web log cleaning is not as easy as it may appear. Most web mining techniques removes image files with extensions GF and JPG (if analysis does not involve image examining) [8], [51. Files with these extensions do not refer to separate web pages. They are part of some web page and are usually downloaded together with the requested page file. However, web servers monitor and log all requested files, including imagdmedia files. Experimenting with web log data cleaning provides evidence that there is much work to be done in this area. Besides images, which can be removed by most web mining techniques, there are number of other irrelevant files. These pages have a negative impact on the analysis. For example, the administrator might log into a site to make some updates or modifications, the web server will log this action as well. The web server cannot distinguish internal from external user. Table 1 presents a list of file types referred to as irrelevant. This list was gathered from experimental studies analyzing web log data. Table 1. Discovered irrelevant web pages URLfile tlpe GF JPG C0 SWF JS XT css PFR admin Description Picturehmage file Picturehmage file An icon so that when visitors to your site boohmark or add it to their 'yavourites" a nifry little icon shows up instead of that drab default explorer icon Flash -animation files Java script (additional extensions) file Text file for search engines. t usually contains info about the site for robots HTML style sheet, style description, lots of different pages can have the same style description Picture Frames Administration actions The web logs from the advertising company site was used as a source of experimental studies (for details, see Section 5). n the beginning, standard web log mining tools were applied. n the later stage, it was noticed that the web log mining results were not meaningful to the advertising client. Exploring the web site design technology, it was discovered that the web site was created using frames (for details, see Section

3 111 Links Table soum Page Text fleftmermcfa content ciin neknucfa /members/mddex c6n ABOUT US flefhnenu cfa fleknu cfa lawardsmdex cfm hlshop/indexcfm AWARDS BOOKSHOP Aethnenu cfa icareerdmdex ciin CAREERS flehucfa i~scusslodlndex ciin DSCUSSON Aeitmnu cfd aboubxljoin html HOW TO JON Aeknu cfd lpmsonsimddex cfm PA PENSONS Aeknu cfd Aefhnenu cfa kmczslindci cfm /e&tdetulr cfm MEMBERS ONLY MY DETALS heknu cfa inewsiindev cfm NEWS 4 Record to Links Table Page Analvzer <a hrehcontent cfm></a> flethnenu cfa imntent cfin ltopbawfa mdex cfa /userdefinit~on cfa lresourcziindexciin /resowcz/menuciin hkshopmdexciin ltcokshop/menu.ch Page to be Analyed log file and retrieved from the web server. A page analyzer engine processes bodies of retrieved pages further. The analyzer parses HTML source and stores detected link in the links table (see Fig. 3). The procedure is repeated for all web pages in the web log. The links table in Fig. 3 consists of the source information (the first column) pointing from where the visitor of the web site got to the page located in the second contoured column. These are names of pages accessible Page page Request from the source pages. The third column is the text of user friendly web page names presented on the web site. A Web Server nternet similar page retrieving method was presented in [4]. The authors used special web agents to get links from the World Figure 2. Retrieving the source code from web Wide Web for indexing search engines. pages However, here, the links which have 2). As described earlier, frames do not refer to been retrieved are used for cleaning exact pages. t is difficult to detect frames purposes. Moreover, text tags retrieved the HTML code has their role in presentation results. FLTER F page found in link table it is preserved, ovmse -removed Add page as such link emt content cfm /eventsirecession-menu html leventsiconfaence html iservicesicpd-index c h lservicesltraimngicpdiloghome cfm iservicesitraimnglcpd~menu cfm ieventsirecession-menu html iservicesicpd-index cfm Figure 3. Retrieved just those web pages which are accessible from other pages because frame pages do not have characteristic file extension (e.g., like pictures extensions jpg, gif). The below proposed technique consist of two frame recognition stages Retrieving HTML code from web server The first stage of advanced web log filtering consists of retrieving information from the web server. Web log file serve as a source of web pages to be downloaded. n the developed framework, page names are taken from the web 3.2. Filtering Some pages can be found in both web log file and link table (see Fig. 3). However, the origin of the pages in the link table is different. Link table contains only pages the user can click on. However, there are some text tag problems. Sometimes, links can be the image or JavaScript file. No text tag can be retrieved using these links. n these cases, third column in JavaScript object-based scripting language used for visual enhancement.

4 112 Table 2. Top pages of the generated by Webtrends web mining tool, here irrelevant page is indicated as RP, and link page as LP Views Total no Type /ipaccrreers/ 19,760 RP L 17,988 RP /ronrenf. chn 13,453. LP /iaacareet~s/fnrtfile/lisfcute~ories.rfiir 3,492 LP /rmuform.rfm 3,347 RP /secr~~~neil;searchfmrrrcret.cfnr 2,607 RP /ipacareer.r/solash. cfm 2,496 RP /aboutmembers/ 2,312 RP /careers/ 2,296 RP /careers/menu. chn 2,181 RP /iuacureers/lzome mainkiver.cfin 2,091 LP /rad/ 1,886 lrp /ipacureers/~~elconre. chi 1,868 LP /tliunkvou.cfni 1,742 RP /aknree acrion.chn 1,725 RP Lserviceshaiii, crfm 1,440 LP /cod/lonin!. drli 1,434 RP /ser~~ices/~lvnninicmmu rhn 1,413 RP /ctd/add!. chi1 1,406 RP hem ber.~mau/searchouiions. cfin 1,370 LP /neu;s/ 1,353 RP /services/ 1,350 RP /nens/meizu.cfm 1,343 RP /services/trainir~.e/cr~~~~home.cf?z 1,313 LP the Links table may contain a gap. n some other cases, text tags can be completely meaningless, for instance click here, etc. From link pages it is possible to get to the other pages. n other words, user can click on a link page and get to the destination page. Link table s second column contains all possible pages user can get to (click on). Following this, the page is relevant if it exists in the links table. Pages that do not exist in the links table must be removed (see Fig. 2). Using the methodology developed in this research, cleaning covers not just frame pages but all irrelevant pages (for example, pictures, flash animation, etc.) and leaves only essential pages for further analysis and experimental studies. 4. Visualization web log patterns The essence of nformation Visualization is referred to the creation of an internal model or image in the mind of a user. Hence, information visualization is an activity that humankind is engaged in all the time [2]. The scheme of the process of knowledge discovery by Fayyad et al. [7] does not involves information visualization. However, visualization should give a clear view of the rules, classes or recommendations discovered in the final data analysis stage. For example, information presented for a web mining analyst might not be understandable by a business analyst. For example, a technical interpretation of the page index.htm1 is commonly understood by web site developers. The expression index.htm1 might be more easily understood by a business analyst as Homepage. The implemented engine (see Fig. 4) takes technical web page names from the cleaned Links table and replaces them with textual information. 5. Results The web site of the media advertisement company was used to show the effectiveness of the proposed methodology. The web site was created using frame pages. Standard cleaning technology was not effective. For the end-user, the results produced by Webtrends were poor and of limited use (see Table 2). Table shows top pages viewed by visitors to the web site. The first column of the Table 2 represents names of the web pages (views). The second column indicates percentage of a viewed page out of all views. The indication of relevancy of the page was added to the last column by authors (reminder, if it has not link, it is not suitable for mining process). The size of log file contained records gathered over the period from to After applying the proposed cleaning techniques, the size of the data was reduced to records (about 22% of the original data set was left). The results showed that in 72% of the cases used in the study the proposed frame cleaning technique outperformed standard cleaning methodology. After the data was cleaned from irrelevant records, visualization technology was applied and produced the results as can be seen in the last replacing engine stage, see Figure 4. For example, /content.cfm is replaced with text,. /events/recession-menu.html with RECESSON, /events/conference.html with Essential Advice for Budget Planning 2002, /services/cpd-index.cfm with CPD ZONE, /news/index.cfm with NEWS. This modification is pretty attractive and allows the analyst to trace the mined web site immediately. WebTrends is the web traffic analysis softwart: package for web administrators and marketing managers everywhere.

5 113 fieitmenu cfa /lehenu.cfa /leftmenu cfa /lefhnenu.cfa ileftmenu cfa /leftmenu cfa Links Table iawarddiudes cfm ibookshopiindex c h icareerdindex cfm /dscussionjlndex cfm iaboutudjom html ipensionsiindex.cfm /sewwedindex c h /editdetails.ch ABOUT US AWARDS BOOKSHOP CAREERS DSCUSSON HOW TO JON PA PENSONS MEMBERS ONLY MY DETALS NEWS contenwin Replacing technical vemon hy user fnendly interpretation interpretahon is ieventsconference hhnl ieventslconference.html /semces/cpd indexcfm /newsiindes.c& User friendly Web Log RECE~SON Essenhal Advlce fm Budget Planrung 2002" "Essentd Advlce for Budget Planrung 2002" "Essenhal Advice for Budget Planrung 2002" CPD ZONE NEWS Figure 4. User friendly interpretation of web pages 6. Concluding remarks Two new techniques for enhancing web log mining are presented. A novel framework for performing advanced web log data cleaning was presented. The advantages of this technique were enlarged. This new framework was implemented and tested on a real data set of web logs from the advertisement industry. The results were compared with the results produced using standard cleaning techniques. The last stage of data mining is visualization. The engine implemented in this research, automatically retrieves text from HTML sources and presents the results understandable to the domain user. Experiments with the real world problems and real world users showed that the proposed techniques significantly increase the attractiveness and usefulness of web mining results. 7. References Retrieval. n Database and expert systems applications Vienna: EEE Computer Society. [5] Faulstich, L., et al. A Data Mining Analyzing the Navigational Behaviour of Web Users. n Workshop on Machine Learning User Modelling of the ACA'99 nternational Conf Creta, Greece. [6] Faulstich, L.C., et al. WUM: A Tool for Web Utilization Analysis. n EDBT Workshop WebDB' , Spain. [7] Fayyad, U., et al. KDD for Science Data Analysis: ssues and Examples. n Knowledge discovery & data mining [8] Jain, N., et al., Web Mining: Pattern Discovery from World Wide Web Transactions. 1997, University of Minesota: Minneapolis. [9] Pitkow, J., et al. WEBVZ: a tool for World- Wide Web access log analysis. n First nternational World Wide Web Conference. 1994, Switzerland. [l] Balabanovic, M., et al., An Adaptive Agent for Automated Web Browsing. Visual Communication and mage Representation, Vol. 6(4). [2] Banissi, E., nformation Visualization. Encyclopedla of computer science and technology, Vol. 42(27). [3] Cooley, R., et al. Web Mining: nformation and Pattern Discovery on the World Wide Web. n Tools with artificial intelligence [4] Deogun, J.S., et al. Structural Abstractions of Hypertext Documents for Web-Based