Language Modeling and Word Prediction

Transcription

1 Language Modeling and Word Prediction Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa A.Yarmohammadi

2 Papers for Today Tonio Wandmacher; Jean-Yves Antoine. Methods to Integrate a Language Model with Semantic Information for a Word Prediction Component. EMNLP/CoNLL, 2007, pp Trnka, Keith, John McCaw, Debra Yarrington, Kathleen F. McCoy, and Christopher Pennington. User Interaction with Word Prediction: The Effects of Prediction Quality. ACM Transactions on Accessible Computing (TACCESS), 1(3), 2009, pp Tam, C., Reid, D., Naumann, S., O'Keefe, B., Effects of word prediction and location of word prediction list on text entry with children with spina bifida and hydrocephalus. Augmentative and Alternative Communication. 18 (September), Higginbotham, D. J., Bisantz, A. M., Sunm, M., Adams, K., Yik, F., The effect of context priming and task type on augmentative communication performance. Augmentative and Alternative Communication 25 (1),

3 Main topics of papers Wandmacher et al. (2007) Integrate semantic information with a standard language model for word prediction system; proposing and evaluating two main methods and different forms of combining them Trnka et al. (2009) Compare three different text entry methods for AAC text, letter-byletter, basic and advanced word prediction systems, and study their impact on input rate, communication rate, and prediction utilization Tam et al. (2002) A study to evaluate the effect of word prediction on written productivity for children with spina bifida and hydrocephalus Higginbotham et al. (2009) The effects of context priming and task type on communication performance; a system-level viewpoint instead of just device-level viewpoint

4 Wandmacher et al. (2007) Explore the predictive power of Latent Semantic Analysis (LSA) LSA is good at relating contexts to specific content words LSA is a bag-of-words model, so it is very poor at predicting the actual positions of the words in the sentence N-gram language models (LM) are used in the most current word prediction systems, however LM cannot exploit any deeper linguistic structure such as longdistance syntactic relationships, semantic or thematic constraints. Integrate LSA-based information with a standard language model semantic cache partial reranking different forms of interpolation

5 Latent Semantic Analysis In the LSA model a word w i is represented as a vector, derived from a term document (term term) co-occurrence matrix of a training corpus. The history h (= w 1,, w m ) can be represented by the sum of the vectors of w 1,, w m An utterance or a text to be entered is semantically cohesive => probability value p(w i h) can be calculated as the cosine similarity for the word vector w i with the vector h of the current context (cos(w i,h))

6 An example of the LSA predictor Context: Mon père était professeur en mathématiques et je pense que ( My dad has been a professor in mathematics and I think that ) Most probable predicted words: All predicted words are semantically related to the context It neglects the syntactic structure of the current phrase

7 Semantic cache model Words that have already occurred in a text are more likely to occur another time Words probability is raised by a decaying factor, depending on the position of the element in the cache The decay function in an exponentially decaying cache of length l, for a word w i at position p in the cache:

8 Semantic cache model Each element having occurred in the context (W occ ) is added to cache as well as its m nearest LSA neighbors, if their cosine similarity is greater than The size of the cache is adapted accordingly (for, and l) The cache function: w i occ : the word that has recently occurred w i : a nearest neighbor to w i occ f cos (w i occ, w i ): cosine similarity between and

9 Partial reranking Regard only the best n candidates from the basic language model for the semantic model Reranking probability of w i : : a weighting constant D(w i ): the average similarity of the m nearest neighbors (density) I(Best n (h),w i ) = 1 iff w i is in the best n candidates, 0 otherwise

10 Interpolation Interpolation is used to integrate information from LSA and LM. Linear interpolation: Geometric interpolation: Confidence-weighted interpolation:

11 Experimental design Calculate n-gram model on a 44 million word corpus from the French daily Le Monde ( ) Compute a 4-gram LM over a 141,000 word vocabulary using the SRI toolkit Calculate the LSA space on a 100 million word corpus from Le Monde ( ) Generate a term term co-occurrence matrix for a 80,000 word vocabulary using the Infomap toolkit

12 Keystroke savings for a 5-word list 4-gram + decaying cache LSA using linear interpolation LSA using geometric interpolation LSA using linear interpolation and confidence weighting LSA using geometric interpolation and confidence weighting partial reranking decaying semantic cache

13 Perplexity 4-gram + decaying cache LSA using linear interpolation LSA using geometric interpolation LSA using linear interpolation and confidence weighting LSA using geometric interpolation and confidence weighting partial reranking decaying semantic cache

14 Conclusion Significant gains for keystroke saving rate for all methods incorporating LSA information, compared to the baseline Significant perplexity reduction for all methods except LSA using linear interpolation and confidence weighting Most successful method in terms of ksr was confidenceweighted geometric interpolation (CWGI; +1.05%) Most successful methods in terms of perplexity reduction were standard and confidence-weighted geometric interpolation (GI and CWGI; -9.3%)

15 Trnka et al. (2009) Many studies found that word prediction could increase the rate of text production (Newell et al., 1992;Anson et al.,2006;wobbrock & Myers, 2006;Wandmacher et al.,2007) A reduction in the number of keystrokes reduced the amount of effort required to produce more text than they would otherwise Spelling and grammar is also improved with word prediction Some studies have found that word prediction was not beneficial to communication rate (Horstmann & Levine, 1990;Venkatagiri, 1993,1994; Koester & Levine,1996,1994) There is significant cognitive and perceptual load when using word prediction This extra time outweighs the reduction in time from keystroke saving

16 Studies reporting decreased communication rate Horsmann and Levine (1990) Model row-column scanning in three conditions: no word prediction simulation of the PACA-2 scanning/prediction interface simulation of the PAL scanning/prediction interface Both PACA-2 and PAL were slower than a scanning system with no prediction Venkatagiri (1993) Copy task doing by one participant with no motor impairment Scan the prediction list of length 15 after every keystroke The communication rate with word prediction was not significantly different from letter-by-letter entry, despite nearly 50% keystroke savings

17 Studies reporting decreased communication rate Venkatagiri (1994) Extend the previous study to evaluate the effects of the length of prediction list on communication rate Copy task of three text samples done by 12 participants Prediction lists of length 5, 10, and 15, scan the list after every keystroke The communication rate was not significantly different, despite a significant increase in keystroke savings between the three window sizes The additional overhead of scanning larger lists outweighed the additional benefit of higher keystroke savings

18 Studies reporting increased communication rate Newell et al. (1992) Study the effects of the PAL system on users over a long term work with over 50 clients at a variety of age for 6 months Increase in communication rate, decrease in the number of keystrokes, improvement of the quality of the written text Wandmacher et al. (2007) The Sibylle system evaluated on 20 participants Most of the participants found a significant increase in communication rate Feeling more comfortable in longer sessions Fewer typing, spelling, and grammatical error when using the system

19 Studies reporting increased and decreased communication rate Koester and Levine (1994, 1996) Study two different word prediction usage strategies: scan the predictions before every keystroke not to scan the predictions for the first two characters Copy task done by 6 participants with spinal cord injuries (SCI), 8 participants with no motor impairments A drastic decrease in communication rate for participants with SCI, especially for lower ksr savings A moderate increase in communication rate for participants without motor impairments in the case of 30-40% ksr savings A drastic increase in communication rate for participants without motor impairments in the case of 50% ksr savings The results of SCI group is difficult to interpret, because of their expertise at letter-by-letter entry and also signifciant added overhead of using mouth stick

20 Approach Compare three different text entry methods for AAC text: letter-by-letter a basic word prediction system an advanced word prediction system The followings are investigated: the effect of the entry method on input rate the effect of the entry method on communication rate the prediction utilization of the algorithms survey results from participants hypotheses: A more advanced word prediction algorithm will allow for a faster communication (in spite of slower input rate) Users will more fully utilize a more advanced system

21 Experimental design Copy text samples selected from the Switchboard corpus, which is a collection of telephone conversations Study 33 adults with little or no prior experience with word prediction and no motor difficulties Simulate the effects of a motor impairment by requiring each key press to take at least 1.5 seconds Minimize cognitive overhead of the copy task and the prediction system Each participant completed three sessions lasting less than one hour at different days The participants were not pressured to use the word prediction system, but were encouraged to do quickly and accurately

22 User Interface

23 Input rate (seconds per keystroke) Highly significant differences between the method s input rate

24 Communication rate (output rate) Significant differences between the method s output rate

25 Keystroke savings

26 Prediction utilization The actual keystroke savings divided by the potential keystroke savings

27 Survey results Less tiring: advanced word prediction Easier to use: advanced word prediction Faster: advanced word prediction More useful: advanced word prediction Less distracting: advanced word prediction

28 Threats to validity The timed delay approximation of AAC users may not be a correct estimation participants can use the free time to aim for their next selection Non-AAC users are studied. Conducting large-scale studies of AAC users is difficult control for the wide physical and cognitive variation wide variety of AAC devices they use feel more comfortable with another technique like scanning instead of direct selection biased AAC user to favor or dislike word prediction, or user interface

29 Input rate

30 Input rate

31 Prediction utilization

32 Communication rate

33 Actual keystroke savings

34 Model of speedup Develop a mathematical model to characterize the relationship between keystroke savings, input rate, and communication rate The speedup equation directly shows the trade-off between keystroke savings and input rate

35 Conclusions Use of word prediction can lead to higher communication rates and less fatigue than either poor word prediction and no word prediction The increased keystroke savings compensates for the added cognitive load due to word prediction Participants more fully utilize a better word prediction system Vastly fewer typos with advanced prediction

36 Tam et al. (2002) Physical, visual, and cognitive difficulties of children with spina bifida and hydrocephalus should be considered in the application of word prediction technology A study to evaluate the effect of word prediction on written productivity for children with spina bifida and hydrocephalus the rate and accuracy of text entry the effect of location of the word prediction list (upper right, following the cursor, lower middle border) on written productivity Hypotheses: use of word prediction will significantly improve the written productivity individuals will prefer specific location of word prediction list written productivity will be significantly higher when the list is located at user preference place

37 Method Four children with spina bifida and hydrocephalus between 10 and 12,verbal IQ of 80 or above, a minimum Grade 3 reading level, the ability to type using a standard keyboard A repeated-measures, single-subject, alternating-treatments research design (ATD) was used a baseline phase: type without using word prediction for 5 min an alternating-treatment phase: type with and without word prediction using 2-letters-then-search strategy, with the prediction list in three locations a follow-up phase: word prediction was withdrawn A paragraph randomly selected from a story is printed and placed between the monitor and the keyboard KeyREP word prediction software is used with custom dictionary

38 Participant Characteristics

39 Data analysis Kump s (1992) directions for scoring typing tests is used for calculating the accuracy and rate of text entry Accuracy words copied incorrectly = 1 error each time this occurs incorrect spacing between letters within a word = 1 error Rate: 5 keystrokes were counted as one word, 5 min typing, rate = total number of correct keystrokes/25

40 Rate of text entry Accuracy of text entry Ra

41 Rate of text entry Accuracy of text entry Ra

42 Rate of text entry Accuracy of text entry Ra

43 Rate of text entry Accuracy of text entry Ra

44 Results Contrary to the hypothesis, a significantly faster rate when not using word prediction The rates are increased throughout the alternatingtreatments phase The rate gains with word prediction were less in the latter half than in the first half of the alternating-treatments phase A significantly higher accuracy with word prediction in the LM location Insignificant differences in accuracies between typing with no word prediction and typing in UR and FC conditions

45 Discussion Increasing rates of text entry may be related to short-term use of word prediction system regular typing practice during the study period the application of different typing conditions on the same day reduced visual-cognitive functioning of the participants shorter average length of the words in the text than the words in standard English text two-letters-then-search means at least 3 keystrokes for a word KeyREP prediction list is generated by frequency alone additional visual demands for copy task

46 Discussion Additional visual demands for copy task may reduce accuracy Word prediction in combination with auditory feedback may be effective in reduction of spelling errors Contrary to a common belief among software developers, the FC location was not a favorite location for any of the participants Limited generalizability of this study is its primary limitation

47 Higginbotham et al. (2009) Study AAC device use, task performance, and user perceptions across 3 tasks, in conditions where AAC device was (or was not) primed with task specific vocabularies A system-level viewpoint to examine the ultimate impact of device characteristics such as word prediction Relevant performance measures include: device or user performance measures (e.g., communication rate, keystroke savings) task outcomes measures (e.g., success of task completion, task completion time) subjective assessments regarding the interaction (e.g., work load, user satisfaction)

48 Method 24 pairs of adults with no disabilities and no prior AAC device experience One participant in each pair is AAC user and the other one is partner Participant completed 3 different tasks in terms of the communication symmetry and interaction requirements: a tangram task a map task a narrative task Prior to the experiment, the AAC device was trained on taskspecific vocabulary Context priming potentially improved performance (measured by keystroke savings) for all the tasks

49 Experimental design 12 pairs were randomly assigned for context-on condition, and 12 pairs for context-off condition The narrative task was limited to 20 min, map and tangram tasks were limited to 30 min After each task was completed, each participant completed a workload assessment for that task After all tasks were completed, the QUIS was used to measure the user satisfaction All participants completed three cognitive tests: a narrative comprehension test, a card rotation test, and a form boar test

50 Dependent measures Cognitive test performance measures obtained from the 3 post-test, narrative comprehension, card rotation, and form board test Task process measures total number of words produced words per minute keystroke savings Task performance measures narrative comprehension test score map task completion score narrative comprehension test errors map task error task completion time for all tasks User satisfaction measures QUIS

51 Results Task performance measures User satisfaction measures Task process measures Little impact of context priming intervention in this study

52 Task process measures

53 Task performance measures

54 Discussion The marginal differences in keystroke savings did not translate to higher level of communication or performance measures Direct links between commonly used measures in AAC research (such as keystroke savings, communication rate) and task performance measures (used in hunam factors research) have not typically been investigated so far Task-specific interface designs may be important for ensuring optimal communication performance