Development of an Enhanced Web-based Automatic Customer Service System

Transcription

1 Development of an Enhanced Web-based Automatic Customer Service System Ji-Wei Wu, Chih-Chang Chang Wei and Judy C.R. Tseng Department of Computer Science and Information Engineering Chung Hua University Reporter: Judy C.R. Tseng

2 Outline Introduction Relevant Works Structure of The EACS Information Retrieval Algorithms of EACS Implementation and Evaluation Conclusions 2

3 Introduction [1/3] [/] ICESA 2007 International Conference on Enterprise Systems and Application Most existing web-based customer services heavily rely on manpower. It not only increases the service cost, but also delays the time for responding the service requests. In recent years, automatic customer service system has been discussed and presented. 3

4 Introduction [2/3] [/] ICESA 2007 International Conference on Enterprise Systems and Application Some problems still exist The keywords have to be set by a certain personnel. The knowledge and experience of the personnel will absolutely influence on the quality of the keyword set as well as the quality of the answers. The knowledge and experience of the personnel is not able to be efficiently kept in the system. The quality and the accuracy of the answers still have potential to be improved. 4

5 Introduction [3/3] [/] ICESA 2007 International Conference on Enterprise Systems and Application To cope with these problems, an Enhanced Automatic Customer Service system EACS is developed d in this paper. A new keyword weighting g method, called Importance Factor IMF, is also proposed. IMF is applied to develop an automatic keyword extraction system to achieve fully automation. IMF is also applied to an indexing method for FAQ to improve the accuracy of the automatic customer system. 5

7 Relevant Works [1/2] [/] ICESA 2007 International Conference on Enterprise Systems and Application Researchers have proposed several development of intelligent customer service systems in recent years. Hoch 1994 Cohen 1996 Li and Tseng 2001 Tseng and Hwang 2007: ACSS Development of an Automatic Customer Service System on the Internet, Electronic Commerce Research and Applications. Vol. 6, No. 1, pp SSCI 7

8 Relevant Works [2/2] [/] ICESA 2007 International Conference on Enterprise Systems and Application Although the previously proposed automatic customer service systems can automatically handle customer requests. the keywords have to be set manually the accuracy of the answers still have potential to be improved 8

10 Structure of The Enhanced Automatic Customer Service System EACS [1/7] 10

11 Structure of The Enhanced Automatic Customer Service System EACS [2/7] The Chinese Knowledge and Information Processing Database is a database of Chinese terms, containing general terms, proper p nouns and idioms etc. 11

12 Structure of The Enhanced Automatic Customer Service System EACS [3/7] The FAQ Database is a database of frequently asked questions. 12

13 Structure of The Enhanced Automatic Customer Service System EACS [4/7] The Automatic Keyword Extraction Module, uses the CKIP database to extracts the domainspecific keywords contained in FAQ to the Keyword Database. 13

14 Structure of The Enhanced Automatic Customer Service System EACS [5/7] For each FAQ, a Characteristic Vector CV is used dto represent tthe characteristics ti of fthe question, which is extracted by the Characteristic Extraction Module and maintained in the Characteristic Database. 14

15 Structure of The Enhanced Automatic Customer Service System EACS [6/7] Once a question is submitted by the customer, the Question Answering Module will search the FAQ Database to find the best-fit answer and sent it to the customer for reference. 15

16 Structure of The Enhanced Automatic Customer Service System EACS [7/7] The satisfaction analysis module will collect the user-feedbacks and record them in the Userfeedback Database. 16

17 Outline Abstract Introduction Relevant Works Structure t of The EACS Information Retrieval Algorithms of EACS Implementation and Evaluation Conclusions 17

18 The Importance Factor IMF [1/2] IMF is a novel modified d TF IDF scheme TF IDF term frequency inverse document frequency A well-known scheme for representing term weight. It is a statistical measure for evaluating the importance of a term to the corpus it contained. From experience, long keywords tend to be more important than short keywords Eg. Database v.s. Data However, the keyword length is not considered d in the original TFxIDF formula 18

19 The Importance Factor IMF [2/2] IMF Importance Factor L n j TFi, j IMF, = , = log / i j IDFj IDFj N C L i, max TF i,max i= 1 i, j IMF i,j : The weight of term j in the i-th question Q i in the FAQ database. L j : The length of the j-th term. L i,max : The maximum length of terms in Q i. TF i,j : The number of occurrences for term j in Q i. TF i,max : The maximum number of occurrences for the terms in Q i. N : The number of FAQ. C i,j : Equal to 1 if Q i contains term j; Equal to 0, otherwise. 19

20 The Automatic Keyword Extraction Algorithm [1/4] Based on the IMF formula Retrieve all the terms contained in each questions Q i in FAQ database by using the CKIP database. Calculate IMF ij i,j for each term T i contained in Q i. Sort IMF i,j descendantly and add the terms with the top P percent of the IMF s to the keyword database. 20

21 The Automatic Keyword Extraction Algorithm [2/4] Example: Assume that there are four FAQ s in the database. Serial number Q 1 Q 2 Q 3 Q 4 Content of the questions What is a business recovery plan, and what is its purpose? When considering the purchase of ready-made software, what is the purpose of the RFI request for information, and what is the purpose of the RFP request for proposal? What should the response to the RFP include? What are the goals of information security measures? What are the risks involved in purchasing ready-made software? 21

22 The Automatic Keyword Extraction Algorithm [3/4] Example: The statistical data relevant to IMF ID Keyword Number of Length of Max TF Max Length df i occurrences keyword T1 what T2 is T3 business T4 recovery T5 play

23 The Automatic Keyword Extraction Algorithm [4/4] Example: The IMF of the first five terms IMF 4 2 log 4 1, 1 = = = IMF 2 2 log 4 1, 2 = = = IMF 8 1 1, 3 = log4 = = IMF log 4 1, 4 = + = = IMF 4 1 log 4 1, 5 = = =

24 Information Retrieval Algorithms of EACS [1/5] The questions in FAQ are represented as some Characteristic Vectors CV. The CVs are sets of keyword-weight pairs. The CV of the question Q i and customer s request Q can be represented as Q = { K,W, K,W,..., K,W,... K } CV, i 1 i, 1 2 i, 2 j i,j n,wi,n K j : j-th keyword. W ij : The IMF value of K j in question Q i. When a user issues a question Q, the question is also represented as CVQ. 24

25 Information Retrieval Algorithms of EACS [2/5] By using inner product to compare the similarity between CVQ and CVQ i n D = i Wk Wik k= 1 the most similar question Q k can be found the answer A k of Q k is then retrieved and sent to the customer. 25

26 Information Retrieval Algorithms of EACS [3/5] Example: Suppose the CVs of the questions in previous example are represented as CVQ = {K, ,K CVQ = {K 6 2, ,K 8, }, ,K 10, } CVQ 3 = {K3, ,K4, ,K8, ,K9 CVQ = {K, K,K, } 4 = 5 0 7, } 26

27 Information Retrieval Algorithms of EACS [4/5] Example: When a customer issues a question to the system Question What are the risks involved in purchasing ready-made d software? the question is also represented as a CV CV Q = { K, } 5 27

28 Information Retrieval Algorithms of ICESA 2007 International Conference on Enterprise Systems and Application g EACS [5/5] Example: For the previous example, the similarity values by y y employing the inner product method D... D = = = + + = D..... D = + = = = D = + = 28

30 Implementation and Evaluation [1/9] The interface for maintaining the FAQ database 30

31 Implementation and Evaluation [2/9] Several experiments have been conducted to evaluate the performance of EACS. Two FAQ databases with different languages were adopted in the experiments The Yahoo!-Kimo database contained 578 FAQ s in Chinese. The MiTAC International Corp database contained 646 FAQ s in English. Three weighting methods, IMF, TFxIDF, and the weight method used in ACSS are compared. 31

32 Implementation and Evaluation [3/9] Several experiments have been conducted to evaluate the performance of EACS. Three types of requests are used to evaluate the performance of different approaches. Case1: All of the customer requests are similar to the question part of some FAQ s. Case2: All of the customer requests are quite different to the question part of any FAQ; however, the answers of the requests exist in the FAQ database. Case3: Customer requests are randomly selected from Case1 and Case2. 32

33 Implementation and Evaluation [4/9] Experiments for finding best-fit answers from Yahoo!KIMO FAQ database with Case % Accuracy of the answe ers 80.00% 00% 70.00% 60.00% 50.00% 00% 40.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 33

34 Implementation and Evaluation [5/9] Experiments for finding best-fit answers from Yahoo!KIMO FAQ database with Case % Accuracy of the answe ers 60.00% 50.00% 00% 40.00% 30.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 34

35 Implementation and Evaluation [6/9] Experiments for finding best-fit answers from Yahoo!KIMO FAQ database with Case % Accuracy of the answer rs 65.00% 55.00% 45.00% 35.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 35

36 Implementation and Evaluation [7/9] Experiments for finding best-fit answers from Mitac FAQ database with Case % Accuracy of the answer rs 90.00% 80.00% 00% 70.00% 60.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 36

37 Implementation and Evaluation [8/9] Experiments for finding best-fit answers from Mitac FAQ database with Case % Accuracy of the answe ers 80.00% 75.00% 70.00% 00% 65.00% 60.00% 55.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 37

38 Implementation and Evaluation [9/9] Experiments for finding best-fit answers from Mitac FAQ database with Case % Ac ccuracy of the answers 85.00% 80.00% 75.00% 70.00% 65.00% 60.00% 50% 60% 70% 80% 90% 100% ACSS IMF TFIDF Keywords used 38

40 Conclusions ICESA 2007 International Conference on Enterprise Systems and Application EACS can automatically reply customer requests by selecting the mostly feasible answers from the FAQ database. It not only reduces the service cost, but also increases the competition advantages of enterprises. Future Works We will incorporate query expansion techniques. The relevance feedbacks from the users will also be considered to adjust the system performance. 40

41 Thanks for your attention! 41