Web Client-Server Architecture To Support Advanced Text Search

Transcription

1 Project Electronic Cub Reporter Web Client-Server Architecture To Support Advanced Text Search By Haotian Sun MSc in Advanced Software Engineering Supervisor: Rob Gaizauskas August 2005 This report is submitted in partial fulfillment of the requirement for the degree of MSc in Advanced Software Engineering i

2 Declaration All sentences or passages quoted in this dissertation from other people s work have been specifically acknowledged by clear cross-referencing to author, work and page(s). Any illustrations which are not the work of the author of this dissertation have been used with the explicit permission of the originator and are specifically acknowledged. I understand that failure to do this amounts to plagiarism and will be considered grounds for failure in this dissertation and the degree examination as a whole. Name: Haotian Sun Signature: Date: ii

3 Abstract This project is about designing and building a web-based client-server system to support advanced searching over the UK Press Association archive, which is ~20GB of newswire stories constituting the entire PA E-Archive from 1994 till the present. The new system will support journalists in the task of gathering and writing background for breaking news stories. This report includes a literature review which was made to gain the appropriate amount of information on web-based database-driven systems and Web Server-side technologies which will allow the implementation of the web application. Also a requirement analysis was made to determine the requirements of the system to be produced. The system integrates the conventional search engine technology with novel summarization and natural language analysis tools. The technologies used in the implementation of this system include JSP, Java Beans and Servlets, a MySQL relational database, Apache Lucene Searching Engine, a database connection pooling technique and the Tomcat Web Server. Also a new special method was introduced to enhance the performance of this large database-driven web application. iii

4 Acknowledgements Thanks to Rob Gaizauskas, for his valuable guidance, help and feedback. Thanks to Horacio Saggion for his cooperation on data and technical support. Thanks to Emma Barker for her requirements analyses. Thanks to Martin Cooke for his useful e-commerce reading material. Finally, thanks to my parents, my girlfriend and my friends for their love and support. iv

5 Contents Chapter 1: Introduction What is the Cubreporter Project? Aims and Objectives Looking Ahead Chapter 2: Literature Review Overview of Web-based Database Systems System Architecture Survey Tiered Architecture Available Technologies Web Application Models The J2EE Model The MVC Design Model Possible Architecture Scenarios Pure JSP implementing MVC JSP & Servlets implementing MVC JSP, Servlets & EJB implementing MVC Review of the PA Digital Text Library Chapter 3: Requirements and Analysis Functional Requirements Search Function Data Rendering User Personalization System Administration Non-Functional Requirements Constraints Chapter 4: Design Overview of Technology choices Presentation Tier Design Business Tier Design Database Tier Design Chapter 5: Implementation and Testing Overview Implementation Testing Functional Testing Performance Testing Chapter 6: Results and Discussion Findings Goals Achieved Further Work Needed Chapter 7: Conclusion References Web References Appendix 1 Selected Code: Appendix 2 Screenshots: v

6 Chapter 1 Introduction Chapter 1: Introduction 1.1 What is the Cubreporter Project? Suppose a journalist from UK Press Association (PA) went to gather news for the Election He would write background for breaking news and send it back to the headquarters via wireless equipment. As a sophisticated journalist, he should know how many times Tony Blair won the election but he might forget how many seats Labour got in the last election. So he opened his browser and linked to the CubReporter website, where he could log into the advanced searching system to ask for help from PA database which contains approximate 20GB of newswire stories from 1993 till now. What he could enter for search is either a keyword like election Labour seats 2001 or a complete question like How many seats did Labour Party get in 2001? Then the system will show him all the stories related to his query. In short, the journalist would pick up what needed in an efficient way and customize to save them in his record portfolio for further tracking. The above is a scene showing the purpose of the Cubreporter project. Obviously, this project is a web-based client-server and database-driven system. The key issue in this project is how to realize a high performance searching function across the PA s existing large database via web pages. The CubReporter Project is a large and complex project funded by the UK EPSRC for 3 years, and it is the collaboration between the researchers in the Departments of Computer Science and Journalism together with the Press Association Group. The project has been studied by the other researchers for more than two years, and the content introduced here is actually the late implementation of the project. Since the project was a team work, my task was to help the CubReporter researchers in the NLP Group to realize functions of the system. Table 1.1 shows the role of each member in this project. Role Member Principal Investigator Pro. Robert Gaizauskas Requirements Analyzer Dr. Emma Barker Technique Supporter Dr. Horacio Saggion Database Designer Dr. Angelo Dalli System Developer Mr. Haotian Sun Table 1.1 In short, the CubReporter Project is a research project which investigates how language technologies might help journalists to access information in a news archive in the context of a background writing task, as introduced by Gaizauskas et al.(2005). 1.2 Aims and Objectives Currently, the UK Press Association Group has a system called PA Digital Text Library (WWW1) to support journalists in the task of gathering and writing background for breaking news stories. The problem of this system is that it is suffering - 1 -

7 Chapter 1 Introduction from low performance of web pages loading and provides limited functions for document searching. The new system under development should have many advantages against the existing one. In short, it will integrate as many functions as possible in order to convenient the journalists daily job from the PA Group. For example, we may use the existing system to make a search for some kind of stories or even use some searching engine (e.g. Google) to get some information, but obviously, these available approaches are either limited by functions or inexact results. What we are doing is to help the PA Group to reorganize their data archive and develop a new system that is not only practical but also has powerful multiple functions. This project aims to design and build a web-based client-server application to support searching over the PA archive, as indexed and analyzed by the language analysis tools. This includes the following aspects: Designing and writing web server side "business logic" code to access databases and assemble content for dynamic page creation Developing a flexible presentation layer framework for rendering content in the client browser Developing a appropriate session management framework to support collection of search results during iterative search and navigation of search history 1.3 Looking Ahead Firstly, a literature review was undertaken in order to gain the appropriate information and knowledge on the available technologies and processes that could be used to develop the application. Since the application is a web-based system, a system architecture survey for web applications is carried out, where tiered architecture is introduced and after that there is a more detailed description for each tier with the available technologies that can be used during the development of each tier. In addition to this, there is a review of available web application models and an introduction to Model-View-Controller (MVC) design model. After that, possible architecture scenarios are raised, with the focus on the combination with JSP technology. Finally there is a review of the PA Electronic Text Library Website, where some similar functions of the new system are introduced. In Chapter 3 Requirements & Analysis, a full list of requirements is presented. These requirements include both functional requirements and non functional requirements and some constraints to the system. Furthermore, a detailed analysis of these requirements was made in order to understand them and to make the correct decisions during design and implementation. After that, in Chapter 4 Design, an overview of chosen technologies is firstly introduced and then the designs for each tier are illustrated with detailed discussion and appropriate figures. In Chapter 5 Implementation & Testing, there is a detailed description of the implementation process, including some more in-depth review of the significant functions of the application, like index-based searching. A functional testing was followed to evaluate - 2 -

8 Chapter 1 Introduction the quality of the system developed. Moreover, a testing script was designed to simulate several real cases for evaluating the novel method that enhances the performance of the search function. In Chapter 6, all the finding and results achieved during the process of development in this project were discussed from both the team view and individual point. Lastly, a summary of all the work completed in this project was made in the Chapter 7, and also a review of the project

9 Chapter 2 Literature Review Chapter 2: Literature Review 2.1 Overview of Web-based Database Systems Over the past few years, most enterprises have become used to managing their information via electronic database. Now, due to the popularity and accessibility of the World Wide Web, people are tending to retrieve information from web-based application, either by companies or individuals. This trend results in the use of databases for storage of all types of information. Since databases provide the ability to store information in an organized fashion with easy access and retrieval, more and more web-based systems are relying on the database server to support the storage of certain information. Finally, with the application of distributed system theory, the different parts of such a system can run on multiple computers, which provide us an easy and convenient way to enhance development and maintainability. 2.2 System Architecture Survey The Cub Reporter project is a collaboration of researchers introduced in Chapter 1, who built the language analysis tools and studied journalists' information seeking behavior. My job was to work together with them to carry out the system. The critical thing implemented firstly was to decide the system architecture so that we could identify each one s task and work together and efficiently Tiered Architecture Having a review of current web site systems, most of them are based on a tiered architecture, especially the ones used in e-commerce and large web-based applications. The tiered architecture is also called distributed architecture which means that systems are composed of programs running on multiple hosts, such as a web server and a database server. A tier above is one of those hosts but it can be virtual distributed applications running on a single host. So a tier may be a logical, as well as a physical layer of a system. In this project, the word tier refers to logical layers. Generally, there are several tiers that represent different logics such as the presentation logic which shows how information is presented to the clients and the business logic which means the collection of objects and methods that are different from business to business. The advantage of such a design is to allow separation of concerns, which means coding paradigms and required skill sets are both different. Typically, there are following kinds of tired system: - 4 -

10 Chapter 2 Literature Review One-tier systems can be either applications running on a physical computer or isolated systems which disconnect to a network but can function correctly from logical view. Applications like word processors and image viewers belong to this category. The major advantages of one-tier systems are their simplicity and high performance. Since there is no need to communicate with remote services through a network, the application can have the maximum performance allowed by its hardware. Two-tier systems run on two computers, composed of a client application and a server application. For example, a typical web application developed by CGI runs on a client s web browser and a web server. But as mentioned before, a tier can be a logical layer of a system, so it is possible to have a two-tier application running on one machine with a division of two layers. Thus, there are usually three parts in a two-tier system: a client, a server and a protocol. The role of the protocol, which in most cases is the HTTP protocol, is used to send and request data between the client and the server. The advantage of this architecture is to separate presentation logic from business logic, which means the presentation logic is on the client side, while the business logic is on the server side. So the majority of the data processing can be done on the server and the client just needs to receive and present the data derived from the server side. However, this kind of architecture has little potential for resource sharing, which is a big problem for large database-driven system. Three-tier systems have one more layer and usually represent presentation logic, business logic and data logic which means how to ensure the persistence, security and of data respectively. This architecture allows concurrent data access via transactions. It is common to have more than one client to retrieve data simultaneously, so it can avoid the data corruption. For instance, a server requests or stores data in the database server after it receives a request from a client and then it gives information returned from the database back to the client. N-tier systems are those systems having more than three tiers. This type provides great performance capabilities, enhancing the amount of services it can provide and allowing greater maintainability. The disadvantage of an n-tier system is that it is inefficient in most cases, expensive and more complex. In short, the most widely used architecture is a three-tier type, which offers a separation of presentation logic, business logic and data logic. This architecture is argued and proved by Chaffee (2000) as the best solution for a web-based database-driven system. So it is necessary to discuss the available technologies to employ next Available Technologies Since three-tier architecture enhances the separation of presentation logic, business logic and data logic, we can discuss the technologies that could be employed from the - 5 -

11 Chapter 2 Literature Review view of each tier. Presentation Tier Presentation tier means how the web pages interact with the clients. Since most of the content rendered to the end users is generated from the database, this project is a dynamic web system. The current available and popular technologies to produce dynamic content in a web page are Active Server Pages, Java Sever Pages and Velocity Template Language. Active Server Pages (ASPs) are the contribution to the development of dynamic pages from Microsoft. This technology has experienced two generations, classic ASPs and recent ASP.NET. Classic ASPs use scripting languages on the server side and have support for VBScript and Jscript which are built in the web pages. New ASP.NET supports any.net compliant language, such as C#, Java. Since code in ASP.NET is compiled rather than interpreted by the ISA server, it runs faster than classic ASPs. Though both of these technologies are widely used and easy to learn, they are heavily dependent on the Microsoft s server and framework. Because this project is being developed under an open source environment, they are not the proper solutions. Java Server Pages (JSPs) are another technology provided by Sun for developing web pages that include dynamic content. They enable web developers and designers to rapidly develop and easily maintain, information-rich, dynamic Web pages. Unlike a plain HTML page, a JSP page can change its content based on any number of variable items, including the identity of the user, the user s browser type, information provided by the user, and selections made by the user. [4] A JSP file has a.jsp extension and contains both standard markup language elements (e.g. HTML tags) and special JSP elements that allow the server to insert dynamic content in the page. For instance, the following shows a piece of JSP code. <HTML> <HEAD><TITLE>Hello World! </TITLE></HEAD> <BODY> <% %> int i=0; while(i++<10){ <p>do Loop # <%out.println(i); %> </p> </BODY> </HTML> The codes enclosed in <% %> tags are embedded java codes, which exactly obey the programming rules in a typical java class. When a JSP is requested for the first time, it needs to be compiled by the virtual machine on the server side firstly, which allows the server to handle JSPs much faster in the future

12 Chapter 2 Literature Review Since JSP is based on Java technology and open to use, it is supported by many open source web servers, such as Apache Tomcat, and can implement complex web applications via the collaboration with Java Beans and Java Servlets (discussed later). Velocity is one of the Apache Jakarta Projects, which is a Java-based template engine. It permits anyone to use a simple yet powerful template language to reference objects defined in Java code. [WWW 1] Velocity pages are actually written in the form of HTML with embedded Velocity code and end with a.vm extension. The following shows an example of Velocity code style. <HTML> <BODY> #set ($foo = Veloocity ) Hello $foo World! </BODY> </HTML> The bold code above is the embedded Velocity code and the result is a web page that prints Hello Velocity World!. Generally speaking, references begin with $ and are used to get something, and directives begin with # and are used to do something. Code in this style is called Velocity Template Language (VTL). The most attraction Velocity has is that it fully enhances the Model-View-Controller (MVC) pattern (talked later) to organize web sites, which is regarded as a clear way for web designers and Java programmers to work in parallel. This means that web page designers can focus solely on creating a nice site, and programmers can focus solely on writing top-notch code. Since Java code is separated by Velocity from the web pages, it is easier to maintain the web site over its lifespan. So Velocity is a valid alternative to JSPs. However, JSPs plus JavaBeans can also apply MVC design pattern to web sites. According to Martin Cook (2005), the only problem JSPs can have is that there may be a mix between presentation tier and business tier if a system is large and complex. But obviously this does not mean JSPs are not suitable to use in presentation. Considering the valuable experiences we have in JSPs and Java technologies and the limited time on the development, JSPs are enough to carry out the presentation tier with the proper cooperation with Java Beans and Servlets. So JSP is preferred technology to carry out the presentation tier in this project. Business Tier The Business Tier shows us the collection of objects and methods that are different from business to business, which means how to represent your business logic to be performed in a certain way in your project. In this tier, the available technologies are JavaBeans and Java Servlets. (1) Java Beans JavaBeans (Enterprise Java Beans) are a platform-independent architecture for Java, which is one of the most interesting and powerful ways to cooperate with JSPs

13 Chapter 2 Literature Review JavaBeans are reusable Java classes whose methods and variables follow specific naming conventions to give them added abilities. They can be embedded directly in a JSP page using the <jsp:usebean> tag. A JavaBean component can perform a well-defined task and make its resulting information available to the JSP page through simple accessor methods. The difference between a JavaBeans component and a normal third party class used by the generated servlet is that the web server can give JavaBeans embedded in page special treatment. For instance, a server can automatically set a bean s properties using the parameter values in the clients request. In other words, if the request includes a name parameter and the server detects through introspection (a technique where the methods and variables of a Java class can be programmatically determined at runtime) that the bean has a name property and a setname(string name) method, the server can automatically can setname() with the value of the name parameter. So there s no need for calling getparameter(). JavaBeans are embedded in a JSP page using the <jsp: usebean>action, which has the following syntax where case is sensitive and the quotes are mandatory. <jsp:usebean id= name scope= page request session application class= classname type= typename > </jsp:usebean> The setting of the attributes in the tag is explained in the table 2.1 as following: Attributes Specifications Examples Id Define the name of the bean. This is the key under which the bean is saved if its scope extends id= potentialuser id= newstory beyond the page. If a bean instance saved under this name already exists in the given scope, this instance is used within this page. Otherwise a new bean is created. Scope Specify the scope of the bean s visibility. The default value is page which means the variable is scope= session scope= request created essentially as an instance variable. If the value is request, the variable is stored as a request attribute; if it is session, the bean is stored in the user s session; if application, the bean is stored in the servlet context. Class Specify the class name of the class= bean, which is used when com.cubreporter.user.userbean initially constructing the bean

14 Chapter 2 Literature Review The class name must be fully qualified. Type Specify the type of the bean as it should be held by the system, type= com.cubreporter.user.userinterface used for casting when the object is retrieved form the request, session, or context. The type value should be the fully qualified name of a superclass or an interface of the actual class. Table 2.1 Furthermore, the <jsp:setproperty>action provides the ability for request parameters to automatically set properties on the beans embedded within a page. This allows beans to access the parameters of the request without having to call getparameter(). Though there are four ways to use this feature, the common usage is as following: <jsp:setproperty name= beanname property= * /> where any parameter with the same name and type as a property on the given bean should be used to set that property on the bean. For example, if a bean, which represents the user, has a setusername(string username) method and the request has a parameter username with a value of Rob, the server will automatically calluserbean.setusername( Rob ) at the beginning of the request handling. The parameter is ignored if the parameter value can not be converted to the proper type. or <jsp:setproperty name= beanname property= paramname /> where the given property should be set if there s a request parameter with the same name and type. For instance, <jsp:setproperty name= beanname property= username />. Finally, the <jsp.getproperty>action provides a mechanism for retrieving property values without using Java code in the page. Its usage is as following: <jsp:getproperty name= beanname property= paramname /> This means to include at this location the value of the given property on the given bean

15 Chapter 2 Literature Review (2) Java Servlets The Java Servlet API allows a software developer to add dynamic content to a web server using the Java platform. It has the ability to maintain state after many server transactions, which is done using HTTP Cookies, session variables or URL rewriting. Java Servlets is a generic server extension, which is actually a Java class that can be loaded dynamically to expand the functionality of a server. Since servlets are written in the highly portable Java language and follow a standard framework, they provide a means to create sophisticated server extensions in a server and operating system independent way. Java Servlet API makes use of the Java standard extension classes in two packages javax.servlet and javax.servlet.http. The javax.servlet package contains classes and interfaces to support generic, protocol-independent servlets. These classes are extended by the classes in the javax.servlet.http package to add HTTP-specific functionality. Every servlet must implement the javax.servlet.servlet interface. Most servlets implement this interface by extending one of two special classes: javax.servlet.genericservlet or javax.servlet.http.httpservlet. A protocol-independent servlet should be a subclass of GenericServlet, while an HTTP servlet should be a subclass of HttpServlet, which is itself a subclass of GenericServlet with added specific HTTP functionality. Unlike a regular Java class, a servlet does not have a main() method. Instead, certain methods of a servlet are invoked by the server in the process of handling requests. Each time the server dispatches a request to a servlet, it invokes the servlet s service method. A generic servlet should override its service() method to handle requests as appropriate for the servlet. The service() method accepts two parameters: a request object and a response object. The request object tells the servlet about the request, while the response object is used to return a response. In contrast, an HTTP servlet usually does not override the service() method. Instead, it overrides doget() to handle GET requests and dopost() to handle POST request. An HTTP servlet can override either or both of these methods. The service() method of HttpServlet handles the setup and dispatching to all the doxxx() methods, which is why it usually should not be overridden.( Hunter & Crawford, 2001) The remainder in the above two packages are largely support classes. For example, HttpServletResquest and HttpServletResponse classes in javax.servlet.http package provide access to HTTP requests and responses. The javax.servlet.http package also contains an HttpSession class that provides built-in session tracking functionality and a Cookie class that allows me to quickly set up and process HTTP cookies. So the session management for the user in the project could also be implemented by a servlet. Apart from the technologies mentioned above, the search engine is the most important part in the business tier. Apache Lucene is the preferred search engine because our

16 Chapter 2 Literature Review application requires a full-text search and Lucene technology is an open source, high-performance, full-featured text search engine (WWW2). Lucene APIs has many powerful features, which include a number of accurate and efficient search algorithms. For instance, the ranked searching (best results returned first), the field searching (e.g., title, author), many powerful query types(e.g. phrase queries, wildcard queries, proximity queries, range queries and more), the date range search, sorting by any field, multiple-index searching with merged results etc. The fundamental concepts in Lucene are index, document, field and term. An index contains a sequence of documents; a document is a sequence of fields; a field is a named sequence of terms; a term is a string. The index files are based on the enterprise s own data and generated by the enterprisers themselves. We can make use of these index files for efficient full text search. Data Tier Today, the most popular and efficient way to store data in a web application is using database or XML. The original PA data archive contains more than 8.5 million stories totaling 20GB of data. The raw corpus has been processed and encoded in XML following a strict Document Type Definition (DTD) specification which includes elements such as story date, category, topic, and structural information such as headlines, bylines, and paragraphs. One example is shown in Figure 2.1. Obviously, in order to access the XML-formatted data, it is necessary to have a view of available XML parsers. Since the later work will make a transfer from the XML data to a relational database, available technologies for operations on databases are also introduced. <?xml version="1.0" encoding="iso "?> <!DOCTYPE HSA SYSTEM "../../../../dtds/hsa.dtd"> <HSA DATE=" " DAY="01" YEAR="1994" MONTH="01" ID="HSA1868" PRIORITY="4" CATEGORY="HHH" COUNT="222" MSGINFO="PA" TOPIC="1 SHOWBIZ Ward" TIMEDATE=" JAN 94"> <HEADLINE>I'M NOTHING LIKE MY SEXY ROLE, SAYS SOPHIE</HEADLINE> <BODY> <BYLINE>By Rob Scully, Press Association Showbusiness Correspondent</BYLINE> <PARAGRAPH NRO="1">Actress Sophie Ward, who plays a sex siren for her latest TV role, insists she is nothing like that in real life. </PARAGRAPH> <PARAGRAPH NRO="2">Sophie, 28, plays Eden in a new Ruth Rendell chiller, A Dark Adapted Eye, starting tomorrow night on BBC1.</PARAGRAPH> <PARAGRAPH NRO="3">``It was fun to play a glamorous part because I am not like that and it is quite hard work... it sometimes took three hours just to get the hair right,'' she said.</paragraph> <PARAGRAPH NRO="4">end mf </PARAGRAPH> </BODY> </HSA> Figure 2.1 Corpus Encoding

17 Chapter 2 Literature Review XML is the abbreviation for Extensible Markup Language, which is simply text and portable data. The data in a XML file is stored in custom tags within the markup language, something like HTML. It has advantages for storing complicated information unambiguously and is easily readable by both humans and machines. XML is fully supported by Java, so there are many APIs that can be employed to parse an XML file, such as SAX and JDOM. As seen from the Figure 2.1, the first line indicates that this document is marked up in XML. The second line specifies validation and root. The document starts from the third line, where the only root element HSA is followed by several attributes, such as the date of this story and the category code HHH which stands for UK Home News only. The next is the story s headline that is in the same level with its body. The body has two kinds of children, which are byline and paragraph respectively. Each paragraph has its own number attribute that could be used for search. (1) XML Parser In order to retrieve certain information from a XML file, an XML parser is needed and handles the important task of taking a raw XML document as input and making sense of the document. Since there are no hard and fast rules to select a XML parser, it is not an easy task to choose the right parser for your application. According to Brett McLaughlin (2001), there are two main criteria which are typically used. The first thing is the speed of the parser. As XML documents are frequently used and their complexity grows, the speed of an XML parser becomes extremely important to the overall performance of an application. The second is conformity to the XML specification. This means you have to be careful when selecting a parser since some XML parsers may not conform to finer points of the XML specification in order to squeeze out additional speed. Balancing these factors, JDOM is an ideal choice to employ according to Cook (2005), because it provides high performance of SAX (e.g. rapid parsing and output) and document model of DOM without memory problems (e.g. random-access). Though XML has many advantages to store data, there are some very important disadvantages. For example, XML files require a large amount of storage compared to databases and it has bad performance if the source data is quite large when searching objects. Because the original PA s archive saved in the form of XML is ~20GB, which is too large and affects the performance of searching, we decided to transfer original PA s archive to store in the database. This is the reason why we decided to reorganize the current PA archive and store it in a relational database. (2) Relational Database

18 Chapter 2 Literature Review A relational database is a database system in which the database is organized and accessed according to the relationships between data items without the need for any consideration of physical orientation and relationship (WWW4). It stores all its data inside tables, and nothing more. All operations on data are done on the tables themselves or produce another table as the result. Relational database systems are widely used among the industry because of their built in function of organizing the data into tables. There are many well-known relational database management systems today, such as MySQL and Microsoft SQL Server. All these products have a number of advanced features to provide easy access to data and high performance to retrieve data in cases where search functions are applied to an application, especially multiple factor searches. In order to communicate with the relational database systems, Structured Query Language (SQL) is used as the standard language to perform certain tasks, such as a search or an upgrade task. Generally, the standard SQL commands such as "Select", "Insert", "Update", "Delete", "Create", and "Drop" can be used to accomplish almost everything that one needs to do with a relational database, though there may be some additional ones that are proprietary extensions to different database management systems. When a database is created, all the tables in the database must be created using the SQL syntax, where various rules of SQL must be followed. Meanwhile, the relationships between the tables must be set up and the column types must be declared. The filling of data into the tables can either be done manually through a command prompt or can be done with embedded SQL code in Java or other types of programs. Lastly, data retrieval from a database can also be done manually through a command prompt or through embedded SQL code in some type of program, always using the SQL command SELECT and some proper syntax. In order to communicate with the database using java, all needed is the JDBC API (Java Database Connectivity). The JDBC API is the industry standard for database-independent connectivity between the Java programming language and a wide range of databases (WWW5). It is just like a bridge between application and database server and provides access to an SQL-based database. The JDBC API makes it possible to do three things: Establish a connection with a database or access any tabular data source Send SQL statements Process the results With JDBC technology, we can make the application development easy and economical. For instance, programmers can save a lot of their time and effort because they do not need to care about complex data access tasks which are hidden in JDBC API. Furthermore, there is no need for configuration by the client, since we can define a JDBC driver

19 Chapter 2 Literature Review Another thing I had to care about in this project is opening and closing database connections. One way to handle this is to have each usage of the database open and close a connection explicitly. One of the problems of doing it in this way is requiring every class that talks to the database to have enough information available to establish a connection. Moreover, opening and closing database connections are just about the most expensive thing an application can do. For short queries, it can take much longer to open the connection than to perform the actual database retrieval. So it is very necessary to have another way to handle the database connection. Connection pooling is a technique that was pioneered by database vendors to allow multiple clients to share a cached set of connection objects that provide access to a database resource. A connection pool class should be able to perform the following actions: The connections are allocated in advance Available connections should be managed New connections are ready to be allocated Wait for a connection to be available The connections should be closed when not required Normally, I should write my own connection pooling servlet. But luckily, Apache Turbine was found as a third-part support, whose database connection pooling facility can be made use of in this project. Apache Turbine is a Servlet based framework that allows experienced Java developers to quickly build secure web applications (WWW3). Turbine allows us to personalize the web sites and to use user logins to restrict access to parts of the application. Turbine provides good support for working with JDBC through the Torque Object Relational Tool and its connection pooling facilities. Before the Turbine facilities can be used, I need to set up my application for Turbine and create a property file with some specific initialization values for the database pool in it. More details will be introduced later. 2.3 Web Application Models Since this project is actually a web-based application, it is important to review some popular and useful web application models so that an appropriate model can be chosen The J2EE Model The Java 2 Enterprise Edition (J2EE) Model is actually a three tiered web application model, which includes Client Tier, Middle Tier and Enterprise Information System (EIS) Tier. J2EE is a compilation of various Java APIs, including JSP, Java Servlets,

20 Chapter 2 Literature Review Enterprise JavaBeans (EJB) and JDBC. These APIs stand for different component technologies, and are managed by what the J2EE documents name containers. A web container provides the runtime environment for servlets and JSP components, translating requests and responses into standard Java objects. Similarly, an EJB container manages EJB components. One important thing we should keep in mind is that an EJB is quite different from a Java Bean. On the one hand, a JavaBeans component is a regular Java class, which follows a few simple naming conventions and can be used by any other Java class. On the other hand, an Enterprise JavaBean must be developed in compliance with a whole set of strict rules and works only in the environment provided by an EJB container. Furthermore, components in the two types of containers can use other APIs (e.g. JDBC) to access databases. Figure 2.2 shows an overview of all the components and their relationship, which was introduced by Pawlan (2001). The client tier includes browsers as well as some GUI applications. A browser usually uses HTTP to communicate with the web container. Client services are supported by the middle tier through the web container and the EJB container. Many applications can be implemented solely as web container components. However in some complex applications, the web components just act as an interface to the application logic implemented by EJB components. Both of these components can access databases via other J2EE APIs like JDBC. The Enterprise Information System (EIS) tier holds the application s business data, which is usually comprised of one or several relational database management servers. Client Tier Middle Tier EIS Tier Figure 2.2 J2EE Model

21 Chapter 2 Literature Review The MVC Design Model The Model-View-Controller (MVC) design model was first introduced by Xerox in a number of papers published in the late 1980s in conjunction with the Smalltalk language. And since then it has been widely used in all popular programming languages because of its advantages on reducing effort needed in software development. As its name states, this design pattern is based on the three entities, the Model, the View, and the Controller. The Model represents pure business data and the rules for how the data is displayed or the user interface for modifying the data. The View, on the other hand, knows all about the user interface details. It also knows about the public Model interface for reading its data, so that it can render it correctly, and it knows about the Controller interface, so it can ask the Controller to modify the Model (Bergsten 2002). Before MVC, user interface designs tended to mix these entities together. MVC decouples them to increase flexibility and reuse and enforces the separation of responsibilities into different tiers. The figure 2.3 shows a brief view on how the different pieces communicate in a MVC model. Response Request View Controller Forward Event Data State change Model XML database Figure 2.3 MVC Model In practice, there are several ways to assign MVC model to the different types of J2EE components, which depends on the scenario, the types of clients supported, and whether or not EJB is used. The next subsection describes possible architecture scenarios where JSP plays an important role. 2.4 Possible Architecture Scenarios In a J2EE application, there are three approaches to assign MVC roles to the different components. Because the architecture is critical to the system and different way has its own advantages, it is very important to choose the right scenario before implementation. The following shows the different architectures in a J2EE application implemented by different components

22 Chapter 2 Literature Review Pure JSP implementing MVC In this approach, separate JSP pages are used for presentation (the View) and request processing (the Controller) and place all business logic in beans (the Model). As shown in Figure 2.4, Controller pages initialize the beans, and then View pages generate the response by reading their properties. A pure JSP approach is a suitable model for testing new ideas and prototyping because it is the fastest way to reach a visible result. However, when you evaluate the prototype by this approach, you will find that the pure JSP application is hard to maintain since it may be a nightmare to redesign and reuse. View Search.html ResultList.jsp Model JavaBeans database Controller Find.jsp Delete.jsp Redirect Figure JSP & Servlets implementing MVC Today, the combination of JSP and servlets is a powerful tool for developing well-structured applications which are easy to maintain and reuse. When both of them are combined, the MVC roles are assigned as shown in Figure 2.5. In this case, all requests are sent to the servlet as a Controller with a parameter or a part of the URI path which indicates what to be done. As in the pure JSP scenario, Java Beans are used to represent the Model. The Controller servlet takes an appropriate JSP page to generate a response to the user (the View) depending on the outcome of the processing. For example, if a request to add a new document is executed successfully, the servlet will pick a JSP page that shows the list of the updated documents. If the request fails, the servlet will take another JSP page which shows why it failed

23 Chapter 2 Literature Review View Search.html ResultList.jsp Model JavaBeans database Controller Java Servlets Figure JSP, Servlets & EJB implementing MVC Nowadays, an application based on Enterprise JavaBeans (EJB) is regarded as an overhead development. This is because it is the most complex structure and brings us transaction management and a client type independent component model. This structure also enforces the MVC design pattern, and makes an application easy to extend and maintain. Compared with the JSP & Servlets scenario, there are another two components: entity EJBs and session EJBs. An entity EJB represents a specific piece of business data, such as a customer. Each entity EJB has a unique identity, and all clients that need access to the entity represented by the EJB use the same bean instance. Session EJBs are intended to handle business logic and are used only by the client who created them. As seen in Figure 2.6, a user sends a request to a servlet as in the JSP&servlet scenario. However, on one hand, it asks an EJB session bean to process the request instead of the servlet itself. Therefore the Controller role spans the servlet and the EJB session bean. On the other hand, the Model role is shared by the EJB entity beans and the web-tier JavaBeans components. As Bergsten (2002) stated in his book Typically,JavaBeans component in the web tier mirror the data maintained by EJB entity beans to avoid expensive communication between the web tier and the EJB tier. The session EJBs may update a number of the EJB entity beans as a result of processing the request. The JavaBeans components in the web tier get notified so they can refresh their state, and are then used in a JSP page to generate a response

24 Chapter 2 Literature Review View Search.html ResultList.jsp Model JavaBeans Entity EJBs database Servlets Session EJBs Controller Figure Review of the PA Digital Text Library The PA Group is currently running a digital text library for their journalists to collect stories. In order to be familiar with the new system requirements to a high level, it is necessary to play with their existing system, and know about their habits in using the system. Here are three main interfaces from the PA Digital Text Library. As seen in Figure 2.7, basically, there should be some fields for users to enter their queries according to different conditions. Furthermore, users are offered to customize their search data range and sort style. Figure

25 Chapter 2 Literature Review Figure 2.8 shows the matched result page, which include a certain number of items from the total results and a page separation function. Each result item shows the basic information of a document, which includes document id, topic, a short paragraph of the content, the group, the word count and the date of the document. It also provides a function for users to save their preferred documents in their online portfolio. Figure 2.8 In Figure 2.9, the complete information of a document is shown with the highlight of the query. Figure

26 Chapter 2 Literature Review Apart from the review of the interface and its functions, it is known that the current PA online Digital Text Library is powered by PHP technology with XML database. According to our experience on using the existing system, it was found that the system has an unreliable performance, such as a very slow response to the client s request if the network traffic is busy

27 Chapter 3 Requirements and Analysis Chapter 3: Requirements and Analysis This project aims to design and build a web-based client-server application to support searching over the PA archive, as indexed and analyzed by the language analysis tools. The original requirements of this project involve the following aspects. (1) Designing and writing web server side "business logic" code to access databases and assemble content for dynamic page creation (2) Developing a flexible presentation layer framework for rendering content in the client browser (3) Developing a appropriate session management framework to support collection of search results during iterative search and navigation of search history In order to understand these requirements in a more detailed way, it is necessary to divide the requirements into functional requirements and non-requirements with some additional constraints. 3.1 Functional Requirements Search Function The advanced search engine facility is the heart of this project, which offers the main functions of the system. The search functions should be divided into two steps, the first of which is called generic search while the second is called refined search. In generic search, the input query could be any word(s), sentences or questions. Users can specify a search calendar range (e.g. after or before a certain date, or between two specified days), and sort the results by date or weighting. In refined search, users can select their search type in a more detailed way according to their queries in the random search. For instance, a list of the related entities and events to their keywords, or even some similar events should be offered for users to consider about and make a further decision Data Rendering The search result returned from database should be presented in a certain form according to the document structure. The information included in one item should indicate the document ID, title, headline, byline (if any), word count, the date and the first paragraph of content if there is more than one. Each results list page should present no more than ten result items. If the search results are more than ten, the system should also offer a page separation

28 Chapter 3 Requirements and Analysis function that allows user to go to the next page, the previous page or a specified page. Furthermore, each result list page should also indicate the following information: the number of the results, the current page, and the number of items shown in the current page. When a user finds the document(s) he wants, he can open the full document in a new window by clicking on its title. The full document window will show the complete document content with all the other information included in a document structure. Since the system offers a tracking function that users can review their preferred documents in their portfolios, there should be a save button attached to each item in both the result list page and the full content page User Personalization Valid users should use a login interface to enter the system with a username and password. The system should offer users a function to modify their password and update their personal information. The system allows users to review their search record and save preferred results in their portfolio for future use System Administration According to the client s requirement, the system will only be available to the authorized subscribes and users can not register themselves. So there should an administrator function for the system. This function will allow the system administrator to allocate an account to a valid user. The functions that an administrator can take include create, delete or en/disable a user. 3.2 Non-Functional Requirements System Loading Performance Since the database in this system is quite large, the search result may take much memory in the server side. It is very necessary to estimate the system loading performance according to the number of simultaneous queries from different users. Here the web server used for testing has a 6GB physical memory and it is supposed that there would be approximately 4GB memory always free and ready to use for this system. The number of the potential users should also be estimated because the system will be only used by the journalists (around 100 people out of 1400) from the PA. It is supposed that we have 5 people send queries to server at the same according to the real situation in a news collection