Automatic Word Lookup Service and Client Tool for SAIKAM Online Dictionary

Transcription

1 NII Journal No.1 ( ) 研 究 論 文 Automatic Word Lookup Service and Client Tool for SAIKAM Online Dictionary Vuthichai AMPORNARAMVETH National Institute of Informatics Akiko AIZAWA National Institute of Informatics Tasanee METHAPISIT Thammasat University ABSTRACT SAIKAM is an online Japanese-Thai dictionary development project seeking for collaborative efforts from developers on the Internet. The project has been developed as a web site providing Japanese Thai dictionary lookup service and, at the same time, allowing a group of devoted users who registered themselves as SAIKAM developer to update and improve the content of the dictionary. All these services are accessible via standard WWW browser software. According to feedbacks from SAIKAM users, however, two new services were recently integrated into SAIKAM system-saikam client tool, and Automatic Dictionary Lookup for Web Pages. These tools are aimed to expand the scope of SAIKAM utilization to offline users, and improve productivity in dictionary editing. [Keywords] Internet, Electronic Dictionary, Collaborative Work 1 Introduction SAIKAM [1] is an Internet-based Japanese Thai dictionary development project. In addition to word definition lookup service, as available in most ordinary electronic dictionaries, SAIKAM also allows any high-spirited users who registered himself as a dictionary developers to update dictionary content, or word definitions, online. This process is necessary due to the fact that SAIKAM's initial word definitions were obtained automatically by matching Japanese words from a Japanese English dictionary to Thai words from a Thai English dictionary by looking for similarity in their English definitions. As a result, the initial dictionary data is still far from complete, nor accurate. It is expected that, with efforts contributed by volunteers on the Internet, the definitions will be getting better throughout the online word-editing process. The final goal of SAIKAM project is to achieve a medium size public-domain Japanese Thai dictionary and make it available to everybody. Currently, the most recent snapshot of the dictionary data is already accessible on the WWW. Beside basic dictionary lookup service, SAIKAM also provides additional value-added services by utilizing the dictionary data such as Usage Sample Search allowing users to search for sample Japanese sentences from Japanese texts corpus collected from the Internet. Both word and pattern searches are supported. A pattern can be specified as a combination of specific words and the word s part of speech. To be discussed in this paper, Automatic Dictionary Lookup Service for Web Page was recently introduced into SAIKAM system. This service functions as a transparent proxy server for browsing Japanese web pages. The proxy server is equipped with a Japanese text parsing tool which identifies list of Japanese words contained in the document. When Japanese web pages are browsed through this proxy server, the user will be presented with original page content with list of Japanese words and Thai definitions appearing at the bottom of the document page. This allows the user to read Japanse web pages more comfortably without the need to consult dictionary for each word meaning one by one. In addition to WWW browser interface, SAIKAM recently started to provide a client tool for users to run on his/her own PC. Any users with MS Windows operating system can download SAIKAM client and use it as an electronic Japanese Thai dictionary on his PC. Implementation of this client and the Automatic Word Lookup feature mentioned above will be the main focus of this paper. 2 SAIKAM System It is necessary to explain briefly first the basic components 1

2 Automatic Word Lookup Service and Client Tool for SAIKAM Online Dictionary SAIKAM Client Client Side Web Browser B A Client Bridge DB Engine CGI Interfaces Web Server Figure 1: SAIKAM System Components Server Side Add-On Services of SAIKAM system. On the server side, SAIKAM consists of four major components (as shown in figure 1): - Database Engine: An SQL database engine maintaining all dictionary tables :- List of Japanese words, Thai words, word definitions, and statistical data. Users profile and access logs are also kept here. WWW Interface: A set of HTML documents, CGI scripts in Perl programming language, and Java applets to provide trilingual WWW interfaces [2] for dictionary browsing and editing. Value-Added Services: Set of components providing automatic word lookup and sample search features. Client Bridge: A daemon process waiting for connection from SAIKAM client tool, and performing data synchronization between SAIKAM client and the main server. Discussion on SAIKAM server, web-based user interfaces, and the usage sample search were already made in [1], [2] and [3]. In this paper, we will focus on the implementation of Automatic Word Lookup of Web Pages and SAIKAM Client which were introduced recently in order to promote the usage of SAIKAM service. 3 Problems and Motivation Initially, SAIKAM provided only WWW browser-based interface for users to browse and edit word definitions. This turned out to be inconvenient for many target users due to the fact that Internet connection must be brought up prior to the access. Also the cost of Internet connections in the users' country-of-resident (Japan and Thailand) are prohibitive for the purpose of dictionary lookup and editing. Another important issue is the speed of data transfer between SAIKAM server and the WWW browser on client machine. Many users lost interest in word editing because long time delay brought unpleasant editing experience. As a result, it is desirable to have a SAIKAM client tool which can serve as an electronic dictionary and an offline editing frontend. The client tool can not only reduce the cost of Internet connection, but also improve the user satisfaction with speedier response. The automatic word definition lookup service was introduced to save Japanese web pages surfers time in looking up definitions of words from the dictionary. Because the web page content and the dictionary service already reside on the Internet, it must be wiser to let the machine perform tedious dictionary lookup task for users. In the following sections, implementations of these two services are discussed. Browser Web Server SAIKAM Proxy Server URL Rewrite Web Loader Definition Lookup Word Parser Figure 2: Document Flow in SAIKAM Automatic Dictionary Lookup Service 4 Automatic Dictionary The block diagram of the automatic dictionary lookup service for web page is shown in figure 2. The process of accessing this service can be described as follows :- 1. The user first opens SAIKAM Automatic Dictionary Lookup Service web page, and enters the URL of a Japanese web page he/she wants to browse. (figure 3) 2. SAIKAM redirects the browser to SAIKAM Automatic Dictionary Lookup proxy server, the URL specified by the user is passed to the proxy as an argument. 3. SAIKAM proxy retrieves the document from web server of the given URL. Once the document is completely loaded, the proxy then parses the document for known Japanese words, and returns the original document with word definitions in Thai attached to the browser. 2

3 NII Journal No.1 ( ) Figure 3: Gateway to Automatic Dictionary Lookup service Japanese text parsing is done by first filtering out all English characters and symbols. The remaining Japanese text strings are then chop into lists of Japanese words using Chasen tool [4]. After that, the system tries to merge consecutive Japanese words together to see if they can form a compound, and find the longest compound existing in SAIKAM dictionary database. 4. During the parsing and filtering process, any URLs appearing inside anchor tags <A>, or framed window, are modified to make sure that further document requests will pass through the proxy server as the user navigates through the links as well. By rewriting the URLs, there is no need for the users to change proxy settings on his local WWW browser in order to access this service. Since Thai character codes cannot be mixed with Japanese characters in the original HTML document, all Thai definitions are displayed as graphical GIF image. The text-to-gif conversion is carried out by a CGI script on SAIKAM server. Downloading GIF images is a time-consuming task. Because some users may not need to know definitions of all words in the document, the proxy can also be configured to display only list of known Japanese words at the bottom of the original document by unchecking Also lookup for word definitions checkbox in figure 3. The Japanese words are Figure 4: List of known words appearing at the bottom of page (circled) then hyperlinked to SAIKAM word database. By clicking on the word, a user can easily access that word definition page. (figure 4) 5 SAIKAM Client SAIKAM client tool was developed to expand the coverage of SAIKAM users to those without Internet connection, or available but at high cost. At the same time, it is also aimed to make it more comfortable for users to edit word definitions by improving the system response time. The initial version of SAIKAM client runs on Microsoft Windows operating system. Care has been taken to make sure Thai, Japanese, and English texts can be handled properly in many localized editions of the operating system. Basic features of the client, as an electronic dictionary, allow users to Search for Japanese word definition in Thai (figure 5). Search for a kanji character by specifying the number of strokes, kanji reading, or level of difficulty. Kanji can be sorted by frequency of use or difficulty. List of words containing a particular kanji can also be obtained. Search for a list of words matching a number of criteria specified by :- word difficulty, frequency of use, and level of usage in schools. 3

4 Automatic Word Lookup Service and Client Tool for SAIKAM Online Dictionary Figure 5: SAIKAM Client Search Screen The dictionary content on client side is not totally static. From time to time, the client may connect to SAIKAM main server to Retrieve recent updates to dictionary database. Reserve words for local editing. After finishing editing word definitions, upload changes to main SAIKAM server 5.1 Database Synchronization Synchronization of dictionary content between SAIKAM main server and user client can be done by the fact that all changes made to SAIKAM main server are logged with timestamp, and the dictionary data residing on client side is dated. After connecting to SAIKAM main server, the client may query for all updates made after the date of local dictionary data. (figure 6). Then the same changes are applied to the client local data. This mechanism allows incremental update to dictionary data on client side. 5.2 Word Editing Editing of word definition is also allowed on SAIKAM client. However, in order to avoid conflicts among several users, a word reservation system is introduced on the server side. Before editing any words, the user must request for exclusive write access to those words from the reservation system. SAIKAM client does not allow modification of any words whose write access are not yet granted. Update Log Date Server Side Changes Client Side Date: Local Data Figure 6: Database Synchronization: Changes on main server made after are replayed on client side This reservation system is implemented by using the concept of word basket [3] already available on the WWW browser interface of SAIKAM. A word must be added to the user's word basket first before it can be edited. The word cannot belong to more than one word basket at the same time. This is to ensure no two or more users are working on the same word. In summary, the procedure for editing word definition is 1. A user subscribe himself/herself as a SAIKAM developer. The system assign a login name and password. 2. While browsing Japanese words on SAIKAM client, the user may decide to add some words to his word basket for editing. 3. After collecting a number words in his word basket, 4

5 NII Journal No.1 ( ) the user connect to SAIKAM server using the assigned login name and password.the client then requests for exclusive write access to words inside the user s word basket. SAIKAM server grants access to words not belonging to any other word baskets yet. In addition, these words are marked for Client-Side Editing (see below). Modifications to these words are not allowed on the WWW browser interface. 4. List of words whose write access is granted is returned to SAIKAM client. Client then disconnects from the server. The user can edit word definition on his local machine. 5. The user connect back to SAIKAM server again and upload all modifications. Changes are logged on SAIKAM server with timestamp. Words may be released from the user s word basket. Note that word basket defines only ownership of words. With the introduction of word editing feature in SAIKAM client, words can now be edited either by using WWW browser interface, or on SAIKAM client itself. In order to avoid conflicts between changes made on the WWW interface, and SAIKAM client, an additional attribute is attached to words in the word basket to distinguish WWW-side edited words from client-side edited words. jigsaw toward the completion of SAIKAM s set of tools and components. After this, it will be a matter of time for the participants to work on the dictionary content toward the eventual goal of a public domain Japanese Thai dictionary. References [1] V.Ampornaramveth; SAIKAM: An Online Dictionary Development Project, Proc. of the 4th Intl. Workshop on Academic Information Networks and Systems, February 1998, NACSIS Seminar House, Karuizawa, Japan. [2] V.Ampornaramveth, Trilingual WWW Interface to SAIKAM Dictionary Project, Proc. of the 5th Intl. Workshop on Academic Information Networks and Systems, December 1998, AIT, Thailand. [3] V.Ampornaramveth; Akiko Aizawa; Tasanee Methapisit, Implementation of an Internet-Based Dictionary Development Environment: SAIKAM, Research Bulletin of NACSIS, No. 12, March [4] ChaSen, Japanese Morphological Analysis Sysem, 6 Conclusion At the beginning of SAIKAM project, Internet penetration was accelerating. People were spending more and more time on the Internet in seeking for any kind of information. It was not easy to foresee the necessity of an offline client tool for SAIKAM dictionary. As we learned from users feedbacks and experience, however, dictionary is more often used in such environments without the luxury ofinternet access. An offline client, which functions as an electronic dictionary, should be more suitable for a larger group of users. Since SAIKAM client contains a collection of dictionary database and statistical data on Japanese words, it is, in fact, an integrated tools for learning of Japanese language. In addition to electronic dictionary feature, it can also serve as an exploration tool allowing students to extract interesting information from statistical data. From the dictionary developers perspective, it is also an collaborative dictionary editing tool. The dictionary lookup service for web page is SAIKAM s attempt to attract more users to its service on the WWW by providing the dictionary lookup service when the user needs it. Together, both of these two new services are the missing 5

6