It is designed to resist the spam in the Internet. It can provide the convenience to the user and save the bandwidth of the network.

Transcription

1 1. Abstract: Our filter program is a JavaTM 2 SDK, Standard Edition Version (J2SE) based application, which can be running on the machine that has installed JDK It can integrate with a JavaServer Pages (JSP) web-based user register system to prove filtering service for the user. This filter program can filter the base on Bayesian Filtering, which is one of the popular approaches of using statistic method to identify spam. The Bayesian algorithm is proved that it has 99.5 percent of accuracy to filter . It is designed to resist the spam in the Internet. It can provide the convenience to the user and save the bandwidth of the network. 2. Goal: Develop a Bayesian filter to prove the filtering service to the user. The user didn t need to install anything, just need to register an account for this filtering service. It can make our system be used in anywhere and easy enough to use. 1/14

2 3. Background: spam is the most common form of internet spamming. It involves sending unsolicited commercial messages to any recipients. Unlike legitimate commercial , spam is generally sent without the permission of the recipients, and frequently contains various tricks, for example, virus or untrue things. In these few years, becomes the indispensable way to communicate with each others. Each person may receive lots of s for each day and quite the large numbers of these s are spam. Therefore, the users need to spend their time to read the useless things. In this situation, the filtering becomes the necessary service when using . To restrict the development of spam, lots of the filtering approach have appeared in the world. The statistical filtering is one of the approach that perform language classification to identify an is spam or not. It has higher accuracy than other filtering approach. It uses statistics and probability to determine the likelihood of a message being spam, along with the machine learning methods. Bayesian filtering is a popular implementation of statistical filtering. In the market, there are lots of statistical filters have Bayesian filtering function, they have 99.5~99.99 percent of accuracy to filter . It can help the user to solve the problem of spamming. 2/14

3 4. Bayesian algorithm: 4.1 Process to develop a Bayesian filter: As the Bayesian algorithm shows, before the Bayesian filter can filter the , we need to build the dataset for the Bayesian filter through machine learning. Once we have the dataset, we can start to filter the . Therefore, the two main processes to develop a Bayesian filter are: 1. Build the dataset 2. Filter the The build dataset workflow is as follows, you can see the detail example at Page 1-4 in the documentation of implementing the Bayesian filter. 1. Prepare the sample corpus s Spam. Nonspam. 2. Build the tokenizing table of the s 3. Update the spam or nonspam counters in the dataset 4. Update the world value (P value) of each tokens Figure1: The build dataset workflow 3/14

4 The filter workflow is as follows, you can see the detail example at Page 5-8 in the documentation of implementing the Bayesian filter. Incoming 1. Build the tokenizing table of the 2. Build the decision matrix base on the dataset DataSet 3. Calculate the S value If S < 0.9 If S >= 0.9 Nonspam Spam 4.2 Formulas comparison: Figure2: The filter workflow Since our filter base on Bayesian algorithm, so the formula which be used to calculate P value and the Bayesian formula which be used to calculate the S value are important for our project. We find 3 different P value formulas and 4 Bayesian formulas to analysis which one is most suitable for our project. The detail comparison can see at Page 9-18 in the documentation of implementing the Bayesian filter. 4/14

5 5. System architecture: In the following figure, the Bayesian filter program locates at the Proxy server. The user register program locates at the web server. If a user wants to use our filter service, he can register a service account at our website. Then he can login our filter server by any reader which support POP3 Protocol, for example, Microsoft Outlook. Figure3: Bayesian filter architecture 5/14

6 5.1 Proxy server operation workflow: User login to the proxy server Proxy server connects to the user relative POP3 server Proxy server download the mails from POP3 server If the is spam, add SPAM identifier to the subject header. Proxy server return these s to the user after filtering 5.2 Bayesian filter: Locate at the proxy server When proxy server downloads a mail from POP3 server, it will filter this s several headers and content. These headers will be filtered: From To Subject Date Content-Type 5.3 Web server: Web server is responsible for saving the user register system and the registered user information database If a PC is powerful, the web server and proxy server can run at the same PC 5.4 Communication protocol: The client user, proxy server and POP3 server communicate with POP3 protocol 6/14

7 6. Function lists: 6.1 Proxy server: Build Dataset -- Build the dataset for using by the Bayesian filter. We can regularly increase the number of the sample corpus s and use this function to re-build the dataset. It can make the filtering result become more and more accuracy. Filtering -- Responsible for retrieving the headers and content of an ; tokenizing and calculating the spam probability of an Web server: For user: Register the user can register the service account. Login the user can login to the user register system. Modify the user can modify the information of their account. For administrator: Login the administrator can login to the user register system. Manage the users account the administrator can add, delete and modify the users account. 7/14

8 7. System operation example: This example is concerning about the operation of this Bayesian filter program. 7.1 This is our user register website. The user can register his account in this website. Each account contains only one ID, password and host address. If the user want to use this service on 2 boxes, he need to register 2 accounts in our website. The user can modify his account information and the administrators can manager the user accounts. Figure4: register website index page Figure5: register page 8/14

9 7.2 After the user registers an account, he can use Outlook to login the proxy server to retrieve the new incoming . Figure6: user use Outlook to login 7.3 When the user connects to our server, our server will check whether his account and password are correct. If it is correct, the proxy server will help the user to connect to his relative server and start to retrieve the new incoming s. Figure7: The proxy server login to the user s relative account 9/14

10 7.4 When each incoming goes through our filter, its headers & content will be filtered. The filter follows the workflow of filtering an to identify it is spam or not. Figure8: Each incoming will be filtered 7.5 If the incoming is spam, SPAM will be added to the subject header of this . That is the whole operation example of this system. Figure9: SPAM will be added to the subject header of spam 10/14

11 8. Development Environment: 8.1 Hardware: Pentium 4 3GHz, 1GB RAM, 120GB Hard Disk 8.2 Operating System: Microsoft Windows XP Professional with Service Pack Software: Apache Tomcat 5 J2SE5.0 Development Kit (JDK) MS-Access 2003 MS-Outlook 2003 JCreator 3.50 Macromedia Dreamweaver Programming Language: J2SE 5.0 (JAVA 2 Standard Edition) JSP (JavaServer Pages) SQL 11/14

12 9. Installation guide: Install JavaTM 2 SDK, Standard Edition Version 1.5.0_06 to proxy server Install Apache Tomcat 5 to web sever Copy Bayesian filter program to the proxy server Copy user register system program to the web server Modify the Config.java file of the Bayesian filter program. Set the variable of databaselocation to the database location of the DataSet. Set the variable of databasename to the database s name of DataSet. Set the variable of userdatabaselocation to the database location of the user register system. Set the variable of userdatabasename to the database s name of the user register system. After modify the Config.java file we need to re-compile it Run the proxy server and web server. 12/14

13 10. Conclusion: In this project, we create a Bayesian algorithm to filter the spam successfully. In the development of process of this system, we have learned and done several things as follows: 1. Learn the Bayesian filtering algorithm 2. Fine the material and different formulas 3. Analysis each formula s characteristic and accuracy and determine which one is suitable for out project 4. Do a Bayesian filter instance manually 5. Design the system architecture. At first, we want to develop a client side of Bayesian filter program. If the users need to use our filtering service, they need to install our program. Therefore we feel it is inconvenience for the user and decide to change the design to server side. 6. Finally we implement the Bayesian filter system base on this instance and evaluation work that we have done before. For the user, our Bayesian filer system can help them to save the time since they don t need to spend their precious time to identify the spam manually. It can help to decrease the obsession of spam for user. For other areas, it can restrict the development of the spam for long term and then save the network bandwidth and computer processing time. It also can decrease the chance that users are misled by the wrong information in the spam. 13/14

14 11. Job distribution: The user part of web server: Roger, Wong Kuan The admin part of web server: Victor, Choi Wan Chong Work together: Proxy server Search the material and different formulas of Bayesian filter Documentation about the implementation of Bayesian filter Demo Presentation PowerPoint Report 14/14