Eva A. van Weel Honours part bachelor thesis Credits: 6 EC. Bachelor Opleiding Kunstmatige Intelligentie

Transcription

1 This site fits YOU mom! A user evaluation of a model underlying an adaptive website to support mothers during decision-making about whether or not to have their daughter vaccinated against HPV Eva A. van Weel Honours part bachelor thesis Credits: 6 EC Bachelor Opleiding Kunstmatige Intelligentie University of Amsterdam Faculty of Science Science Park XH Amsterdam Dr. F.M. Nack Informatics Institute Faculty of Science University of Amsterdam Science Park XH Amsterdam Supervisors Dr. A. Heuvelink Perceptual and Cognitive Systems TNO Kampweg DE Soesterberg July 29th, 2014

2 Contents 1 Introduction 1 2 Focus groups Participants Methodology Requirements and claims of the proposed model Adaptive interactions of the virtual assistants Clarification of information Feedback based on answers to background questions Information about progress Feedback based on answers to knowledge questions Adaptive navigation support in main menu Presentation of adaptive information Results 9 5 Discussion 13 6 Conclusion and future work 15 Appendices 16 A Questionnaire expert evaluation (in Dutch) 17 i

3 1 Introduction In (van Weel, 2014), a model is proposed underlying an interactive and personalised website to help mothers make an informed decision about whether or not to have their daughter vaccinated against the human papillomavirus, i.e. the virus that can cause cervical cancer. On this website, interactive and personalised information is presented to the mothers, as well as personalised interaction with two virtual assistants. These virtual assistants guide the mother during her visit on the website and provide her with information about the personal relevance of the presented information. The model underlying this website was implemented and evaluated by experts in order to verify whether the actions performed by the website based on the model were correct from an expert point of view. Furthermore, the experts evaluated whether the actions followed logically from the input of the user, i.e. its behaviour on the website and the answers provided by the user. Besides evaluating such an implementation with experts, an evaluation should also be conducted with the target group in order to verify whether the website fulfils the user s expectations and needs. This user evaluation, which was conducted using various focus groups, is the topic of the current paper. On the website three types of personalisation, i.e. adaptation, can be distinguished: the presentation of adaptive information, adaptive navigation support, and adaptive interaction with the virtual assistants. In order to fully understand the implementation of these types of adaptivity and the context in which the focus groups are conducted, it is recommended to read (van Weel, 2014) first. References will be made to some of the proposed requirements of the model underlying the interactive and personalised website as described in the aforementioned paper, that is those requirements that are evaluated in the focus groups. Based on the results of the focus groups, the following research question will be answered: Is the model constructed adequate in providing adaptivity on the website?. As this adaptivity involves three types, i.e. adaptive information, adaptive navigation support, and adaptive virtual assistant interaction, three subquestions are defined: 1. Is the model constructed adequate in providing adaptive information on the website? 2. Is the model constructed adequate in providing adaptive navigation support on the website? 3. Is the model constructed adequate in providing adaptive interaction with the virtual assistants? Given that the model is constructed based on literature research and verified by domain experts from TNO, the hypothesis for this research question is the following: The constructed model is adequate in providing adaptivity on the website. The hypotheses for the subquestions are the following: 1. The constructed model is adequate in providing adaptive information on the website 2. The constructed model is adequate in providing adaptive navigation support on the website 3. The constructed model is adequate in providing adaptive interaction with the virtual assistants In this paper, the evaluation of the model by means of several focus groups will be elaborated on, after which the results will be presented and discussed in more detail. Finally, conclusions will be provided and recommendations for future work will be offered. 1

4 2 Focus groups In order to answer the research question and its subquestions, 3 focus groups were conducted. According to Powell and Single (1996, p. 499), a focus group can be described as follows: a focus group is a group of individuals selected and assembled by researchers to discuss and comment on, from personal experience, the topic that is the subject of the research. The subject of the research in this case consists of the adaptivity of the website. The focus groups were conducted in collaboration with Mirjam Pot, the PhD student on the project, and Hanna Klop, a master student in social psychology. Figure 1 shows the set up of the focus groups. During the focus groups, the roles were divided clearly in order to structure the course of the focus groups. Hanna took the role of panel chairman in which she guided the conversations during the focus groups and ensured that all participants had an equal share in the discussion. Furthermore, extra clarifications were requested when a statement of a participant was not clear. As a panel chairman, Hanna was seated on the head side of the table which gave a good overview of the participants during the discussion. Figure 1: The set up of the focus groups Mirjam was assigned the role of domain expert in which she answered possible substantive questions about the information presented on the website. My role involved the answering of the technical questions during the focus groups, such as questions about the functioning of the website. I was therefore introduced as the expert of the website. The following subsection will elaborate on the participants who collaborated in the focus groups. 2

5 2.1 Participants For the focus groups, mothers were approached who participated in any of the first two experiments performed for the same project. All of these mothers had indicated that they could be approached for further research concerning this project. Because of the interest and drive to contribute to the project, the group of participants is biased. Due to this bias, the group probably does not entirely represent the actual target group. However, care was taken to assemble the focus groups in such a way that as many as possible different views on the website could be gathered even though not the entire target group was represented during the focus groups. The assembling of the focus groups will be elaborated on below. The approached group of mothers consisted of 367 mothers of which 118 mothers responded to the appeal to participate in the focus groups. From these 118 mothers, 34 mothers indicated that they would like to participate in the focus groups, whereas the other mothers indicated that they did not want to participate. Based on the decision whether or not to have their daughter vaccinated, the amount of ambivalence experienced during decision-making, the dates that they could participate and the distance from TNO Soesterberg to their place of residency (with regard to travelling cost repayment) the mothers were divided between the various focus groups. After fitting as many as possible mothers in the various focus groups based on these 4 variables, twenty mothers were eventually invited to come to TNO Soesterberg for the focus groups. Three groups were assembled; one group of mothers who decided to have their daughters vaccinated (group 1), one group who decided not to have their daughters vaccinated (group 2), and a mixed group of mothers who decided to have their daughter vaccinated and mothers who decided not to have their daughter vaccinated (group 3). Furthermore, group 1 and group 3 consisted of mothers who were certain about their choice whether or not to vaccinate and mothers who still felt ambivalent about their choice when the decision was made. All mothers in group 2 were certain about their decision. After cancellations fifteen mothers participated in the focus groups in the end, all of which made the decision whether or not to have their daughter vaccinated against HPV at the beginning of Group 1 consisted of five mothers, group 2 of four mothers and group 3 of six mothers. The age of the mothers varied between 38 and 50. For the participation in the focus groups all mothers received a gift voucher of 10 euros and travelling costs repayment. 2.2 Methodology The goal of the focus groups is to obtain that information from the participants that would not be obtained when they would fill in a questionnaire. A discussion between two or more participants with different views on the same subject, for example, can provide interesting input that can be used to optimise the website. It is namely those differences in viewpoints that can be used to adapt the website to the different users that will eventually use the website. In order to obtain these different viewpoints, an environment needs to be provided in which all participants can speak their opinions about the subject of discussion freely. Providing such an environment is one of the challenges when conducting focus groups. Each focus groups took 2 hours and consisted of four different phases. First a general introduction in which the goal of the focus group was explained to the participants and they were able to get acquainted with each other and the researchers. After this introduction, the second phase consisted of the mothers individually visiting the website and interacting with it. On each web page an empty field was available for the mothers to write down any comments they might have on that page. The third phase consisted of individually filling in a questionnaire, see Appendix A. This questionnaire contained both closed-end questions, i.e. multiple choice and 5-point Likert scales, as well as open-end questions. From this questionnaire only those questions will be 3

6 discussed that involve several of the requirements as proposed in (van Weel, 2014), that were implemented for these focus groups. The next section will elaborate which of these requirements were implemented for and evaluated during the focus groups. The answers to the remaining questions will be used to improve the informative content of the website and are not relevant for this paper. The final phase of the focus group consisted of a group discussion in which all the questions of the aforementioned questionnaire were discussed and different opinions of the participants were gathered about the website. 3 Requirements and claims of the proposed model The three types of adaptivity that are implemented on the website include the presentation of adaptive information, adaptive navigation support, and adaptive interaction of the virtual assistants. The first two types of adaptivity originate from the adaptation model (AM) and the third type from the embodied conversational agent model (ECAM), which are both sub models of the model underlying the website. The AM describes the possible adaptations that the website can perform based on the available information about the user and the domain. These adaptations should ensure that the website fits the user s information needs and level of knowledge. The ECAM coordinates the interactions of the virtual assistants based on the available information about the user and the adaptations performed by the AM1. For the AM and the ECAM various requirements were defined with corresponding claims. These claims describe the rational behind the requirements, i.e. the expected positive and possible negative effects that occur when the requirement holds. For the existing requirements, new claims are defined that are evaluated during these focus groups. The claims that were already defined in the proposed model should also be evaluated with the target groups, however, given the limited time span of two hours for the focus only those claims were evaluated that are most relevant in this stage of the development of the website. During these focus groups, it was primarily the adaptive interactions of the virtual assistants that were evaluated. Below, each subsection will describe the requirements from the proposed model and the corresponding claims of the three types of adaptivity that were evaluated during the focus groups. Furthermore, it will for each requirement be described what adaptations were implemented, as the implementation that was evaluated during the focus groups is different on some points than the implementation described in (van Weel, 2014). 3.1 Adaptive interactions of the virtual assistants The first type of adaptation that will be discussed with its corresponding requirements and claims is the adaptive interactions of the virtual assistants. Based on the available information about the mother, e.g. her answers to the knowledge and background related questions and the information already visited on the website, and the domain, the actions of the virtual assistants are adapted to the mother. Four different interactions will be discussed: clarification of the presented information, advice given about what information to read next, information about the progress on the website and finally feedback based on the mother s answers to the knowledge related questions. 1For detailed information about the sub models see (van Weel, 2014) 4

7 3.1.1 Clarification of information One property of the virtual assistants is that they can clarify the information that is presented to the mothers. The most important or relevant information on a web page could be encircled or underlined, i.e. highlighted. Within the implementation that is evaluated by the mothers during the focus groups, one fragment of text is underlined on the web page about the effectiveness of practising safe sex in order to prevent an infection with HPV. Only mothers who indicated that they found practising safe sex a more effective method of protection against an infection with HPV than the vaccination received the information on this web page and the underlining adaptation. In this implementation the underlining is thus adaptive in the sense that not all mothers receive the underlined fragment, however, an implementation in which this adaptation only occurs for a specific subset of mothers would be more adaptive. In that case, the adaptation would depend on more information than just the fact that the mother finds practising safe sex a more effective method of protection than the vaccination. To implement this, more research will have to be conducted to determine based on which information about the mother this adaptation should be made and which information should then be highlighted. It could even be researched if the method of highlighting has different effects on mothers, such as encircling or underlining the information. Given the defined requirements in the proposed model, the following requirement from the AM corresponds to this property of the virtual assistant: AM-Req 1.1.3: the AM shall provide canned text adaptations As the underlining is shown on the web page and is not necessarily an action of the virtual assistant, the adaptation is part of the AM instead of the ECAM. By means of this adaptation from the AM, the virtual assistant is able to clarify the information presented on the web page. Besides the highlighting of text fragments, canned text adaptations also include the alteration and insertion of text fragment. The former will be discussed in Section 3.3 and the latter will not be discussed as it is not implemented on the website that is evaluated during the focus groups. For this requirement the following claims were evaluated during the focus groups: Claims AM-Req 1.1.3: + It is pleasant for the user that the relevant information is clarified - The user could experience irritations or inconvenience because the clarification of the information is not wanted or necessary During the focus groups the following question was asked to verify the above mentioned claims: "How did you experience that the virtual assistant helped you by the clarification of the information, e.g. by encircling or highlighting information?, where the answers ranged from very unpleasant to very pleasant on a 5 point Likert scale, see question 10 in Appendix A Feedback based on answers to background questions Another property of the virtual assistants is to advice the user what information to view next based on the answers provided by the user to the background related questions. This includes for example questions about the user s level of eduction and the amount of information gathered beforehand about HPV and cervical cancer. The suggestions or advice provided by the virtual assistants are formulated as being feedback to 5

8 the user based on the answers provided to these questions. The following requirement from the ECAM corresponds with this property: ECAM-Req 1.2.2: the ECA shall provide the user with feedback about the user s answers to the background related questions In this implementation, mothers who indicated that they had gathered few information about HPV and cervical cancer before visiting the website received the suggestion to view the basic information component on the website first. This type of adaptive virtual assistant interaction can also be developed further within a next implementation, e.g. more different types of advices could be provided and more information can be combined to provide advice that fits the user s current needs. The following claims were evaluated during the focus groups for this requirement: Claims ECAM-Req 1.2.2: + The user experiences the advice given as pleasant as they are based on the answers provided by the user - The user could experience irritations because she does not want to listen to the advice provided by the virtual assistant During the focus groups the following question was asked to verify the above mentioned claims: "How did you experience the advices provided by the virtual assistants on the website, e.g. suggestions what information to view next?, where the answers ranged from very unpleasant to very pleasant on a 5 point Likert scale, see question 11 in Appendix A Information about progress Furthermore, the virtual assistants have the ability to inform the user about the progress that is made on the website, e.g. which information is visited/unvisited and how much of the total amount of information is already viewed. The following requirement was defined to describe this property: ECAM-Req 1.2.3: the ECA shall provide the user with feedback about the progress made on the website Within the implementation the virtual assistant notifies the user that all the available information on the website is viewed, when the user has clicked all the links in the main menu. In the next implementation, the user could be notified about its progress more often, e.g. when half of the information is viewed or specific components are finished. The following claims correspond with the above mentioned requirement: Claims ECAM-Req 1.2.3: + The user knows how much of the information is already visited + The user knows how much of the information is left to visit - The user could experience irritations because she does not want to listen to the feedback provided about the progress During the focus groups the following question was asked in order to verify the user experience of the claims mentioned: To what degree did you experience that the virtual assistants helped you with your progress on the website?, where the answers ranged from not at all to very much on a 5 point Likert scale, see question 12 in Appendix A. 6

9 3.1.4 Feedback based on answers to knowledge questions The final property of the virtual assistants that was evaluated during the focus groups is the adaptive feedback that the virtual assistants provide based on the answers given by the user to the knowledge related questions, such as Is HPV a virus?. The following requirement was defined in the proposed model to describe this property of the virtual assistant: ECAM-Req 1.2.1: the ECA shall provide the user with feedback about the user s answers to the knowledge related questions Within the implementation this adaptivity can be found in the different methods of protection component on the website. The mother is asked how effective she thinks the different methods of protection are to prevent an infection with HPV. Based on her answers to these questions the virtual assistant provides the mother with feedback. The following claims were evaluated during the focus groups: Claims ECAM-Req 1.2.1: + The user experiences the feedback given as pleasant as they are based on the answers provided by the user - The user could experience irritations because she does not want to listen to the feedback provided by the virtual assistant During the focus groups the following question was posed in order to verify the above mentioned claims: To what degree did you experience that the feedback provided by virtual assistant to your answers was adapted to you?, where the answers ranged from not at all adapted to me to very much adapted to me on a 5 point Likert scale, see question 13 in Appendix A 3.2 Adaptive navigation support in main menu The navigation support provided by the web pages is also adaptive to the user. One way to provide adaptive navigation support is called adaptive link annotation, which comprises the changing of the appearance of the links on the web pages, e.g. by changing the colour of the link or enlarging the font size when the user hoovers over the link. Within the implementation that is evaluated during the focus groups, adaptive link annotation occurs in the main menu. In this menu the links change change colour depending on whether the link has been visited or not. For this adaptivity of the links on the web pages, the following requirement was defined in the proposed model: AM-Req 1.2.3: the AM shall provide adaptive link annotation The claims that correspond with this requirement are the following: Claims AM-Req 1.2.3: + The user can easily see which information has been visited before, as the appearance of the links has changed - It is unclear for the user which appearance of the links corresponds to visited information and which to unvisited information During the focus groups the following question was posed in order to verify the above mentioned claims: I knew which link I had to click on in order to navigate to the information that I was searching for, where the answers ranged from not at all to very much on a 5 point Likert scale, see question 18 in Appendix A 7

10 3.3 Presentation of adaptive information Finally the information presented on the web pages is adapted to the user by taking into account the answers the user provides to the knowledge related questions within the information components of the website. This adaptivity again involves canned text adaptations, but contrary to the fragment highlighting as introduced in Section 3.1.1, this time the canned text adaptations performed are fragment alterations. Based on the answers provided by the user an altered text fragment is presented to the user, e.g. a fragment with a lower of higher degree of knowledge about the subject presented on that page. Given the predefined relations between the possible answers and the corresponding text fragments in the domain, the website can decide which fragment to show when a certain answer is provided. The requirement that was defined in the proposed model for this type of adaptivity is the following: AM-Req 1.1.3: the AM shall provide canned text adaptations The claims that were evaluated during the focus groups that correspond with the requirement presented above are the following: Claims AM-Req 1.1.3: + The adapted information presented to the user is the information the user is searching for, as the fragment alterations are based on the answers provided by the user - The user could experience irritations when the information presented is not the information that was searched for During the focus groups the following question was posed in order to verify the above mentioned claims: The information that was presented to me was the information that I was searching for, where the answers ranged from not at all what I was searching for to exactly what I was searching for on a 5 point Likert scale, see question 23 in Appendix A 8

11 4 Results The results from the 5-points Likert scale questions, as introduced in the previous section, are visualised by means of histograms and box plots. By doing so, the distribution of the scores provided by the participants and the median of these scores can be observed. The results of each question will be described below: Question 10: clarification of information Figure 2 shows that the participants provided scores for this question ranging from 1 to 5, where 8 out of 12 of the participants give a score of 3 or 4. The median is found at 3. Figure 2: Results question 10 Figure 3: Results question 11 Question 11: feedback based on answers to background questions In Figure 3 a similar distribution is found, where again the median is found at 3. However, the distribution is more equally spread in this case, as the interquartile range ranges from 2 to 4 instead of from 2.5 to 4. Question 12: information about progress Figure 4 shows the distribution of the scores provided by the participants for the question about progress information provided by the virtual assistant. As can be seen, most of the scores (11 out of 13) are lower or equal than 3 with the median at 2. The score of 5 of one of the participants is considered to be an outlier given that the score is distant from the majority of the scores provided by the other participants. 9

12 Figure 4: Results question 12 Figure 5: Results question 13 Question 13: feedback based on answers to knowledge questions In Figure 5 the resulting distribution of the scores provided by the participants is shown. The box plot indicates that the interquartile range ranges from 3 to 4 with the median located at 3. Noticeable is the fact that no participants scored the question a 2, whereas 3 participants provided a score of 1. The majority of the participants, however, scored the question either a 3 or a 4 (9 out of 14). Question 18: adaptive navigation support Figure 6 shows the results of the question about the self-evidence of the navigation support in the main menu, i.e. the adaptive link annotation. From all 15 participants 8 scored this question with a 5 and none of the participants provided a score lower than 3. Given that more than half of the participants provided a score of 5, the median is located at 5 as well. Figure 6: Results question 18 Figure 7: Results question 23 10

13 Question 23: presentation of adaptive information In Figure 7 it is shown that half of the participants who answered this question provided a score of 4 on the Likert scale. The interquartile range ranges from 3 to 4 with the median at 4. None of the participants provided a score of 1. Welcoxon-Mann-Whitney tests on different subsets of participants Besides analysing the results of each question for the entire group, the results have also been analysed on the differences in answers provided between subsets of participants. For these tests the nonparametric Wilcoxon-Mann-Whitney test was used to investigate possible significant differences between the medians of the subsets of participants (Mann and Whitney, 1947). As the virtual assistant is an important medium on the website to provide adaptivity and personalisation during the website visit, the first two subsets of participants that were made are: mothers who indicated that they would turn on the virtual assistant during the visit on the website (group 1, 8 mothers) and mother who indicated that they would rather turn off the virtual assistant (group 2, 6 mothers). By analysing the answers of these two subsets, it can be investigated why some mothers want to turn off the virtual assistant and others do not. Table 1 shows the medians of these two groups and the level op significance p for each question. Green indicates a significant difference between medians of the groups, that is p < The answers of one mother have been left out of the analysis, as she indicated both reasons to turn the virtual assistant on as well as reasons why to turn it off. Furthermore, for each question it occurred that one or more of the participants did not provide an answer, however, the amount of answers for group 1 is always between 6 and 8 and for group 2 between 5 and 6. The mothers who wanted to turn on the virtual assistant found the advice given by the virtual assistant significantly more pleasant (p < 0.02) and experienced the help from the virtual assistants of being more of added value (p < 0.04) than the mothers who wanted to turn off the virtual assistant. After finding these significant results, the answers from the other Likert scale questions concerning the virtual assistants, that is questions 6, 7, 8 and 9, have been analysed with the same test as well. A significant effect was found for question 6, where the mothers who wanted to turn on the virtual assistant found the contact with the virtual assistant more pleasant than the mothers who wanted to turn of the virtual assistant (median group 1 = 3, median group 2 = 5 and p 0.018). Furthermore, two effects of p has been found for questions 7 and 9, which could indicate a possible interesting point for further research, as the results seem to point in the direction that mothers who want to turn on the virtual assistant feel more understood (median = 5 versus median = 3.5) and find the support of the virtual assistant more of added value (median = 4 versus median = 2) than mothers who want to turn off the virtual assistant. Question median group 1 median group 2 p median group 3 median group 4 p Table 1: Results Wilcoxon-Mann-Withney test for subsets 1, 2, 3 and 4 11

14 Moreover, it was investigated if the decision whether or not to have the daughter vaccinated correlated with the answers provided by the mothers. Therefore, the group of participants was also split in a group of mothers who decided to have their daughter vaccinated (group 3, 11 mothers) and a group who decided to not have their daughter vaccinated (group 4, 4 mothers). From these 4 mothers, three indicated that they wanted to turn off the virtual assistant. The results of the Wilcoxon-Mann-Whitney test on these subsets of participants are shown in Table 1 as well, where blue indicates a possible trend that could be researched further (0.05 < p < 0.15). For these subsets it also occurred that one or more of the participants did not provide an answer for a question, however, the amount of answers for group 3 is always between 9 and 11 and for group 4 between 3 and 4. The mothers who decide to have their daughter vaccinated found the advice given by the virtual assistant significantly more pleasant (p < 0.05), it more clear what component to click on in the main menu to find the information that was searched for (p < 0.05) and the presented information more what they were looking for (p < 0.04) than the mothers who decided not to have their daughter vaccinated. Furthermore, the results from question 13 indicate that the information from the virtual assistant about the progress of the user might also be a an interesting factor on which these two groups of mothers could differ (p < 0.15). This could be researched further in a next experiment. For these subsets of participants the other questions concerning the virtual assistant have been analysed as well, which resulted in a near significant effect for question 9 (p 0.062). This result points in the direction that mothers who decided to have their daughter vaccinated experience the support of the virtual assistant as being more of added value than mothers who decided not to have their daughter vaccinated (median group 3 = 4 versus median group 4 = 2). Results group discussion During the group discussion it appeared that mothers who wanted to turn off the virtual assistant had various reasons to do so: the virtual assistant was childish, didactic or not of added value. Furthermore, one participant indicated that she wants to search the information on her own instead of being guided by a virtual assistant. The participants who indicated that they would turn on the virtual assistant also provided various explanations, e.g. it facilitates the website visit, the feeling of missing information when turning off the assistant or the feeling that it is nice that somebody thinks along. The navigation support in the main menu was experienced as very clear and self-evident. The texts on the buttons were good guidance towards the information that was searched for. Especially the mothers who decided not to have their daughter vaccinated indicated that the presented information was very pro-vaccination, while they actually hoped to be presented with objective information. Moreover, information about the informative content of the website has been gathered during the focus groups, e.g. some information was too pro-vaccination, not correct or information was missing. These results are not discussed in detail in this paper, but were also obtained during these focus groups. 12

15 5 Discussion In this section possible explanations for the presented results will be elaborated on. Taken into account should be the fact that the focus groups were conducted with fifteen participants and that the results indicate directions for further research. To ensure the significance of the results, the experiment should be repeated with a larger group of participants, especially with more mothers who decided not to have their daughter vaccinated (n = 4 in these focus groups). Question 10: clarification of information The equally spread distribution can be explained by the fact that the clarification of information by means of highlighting by the virtual assistant, i.e. underlining in this case, only occurred once on the website, because of which not all mothers might have encountered this adaptivity. Another explanation might be that the clarification of the information has been interpreted by the mothers as the explanations provided by the virtual assistant. This occurred more often and at different web pages, which may have caused the distribution to be spread out more. Looking at the analysis of the answers of the different subsets of participants (group 1, 2, 3 and 4), no interesting results are found. The medians are equal or near equal (3, 3.5 and 3 respectively) and no significant effects has been found. These results indicate that this type of adaptivity has either not been implemented optimally yet or that there is no difference for this type of adaptivity between the subsets of users. Given that the adaptation was only implemented at one page of the website, it is recommended to implement this type of adaptivity on more web pages and then evaluate the question again with a larger group of participants. Question 11: feedback based on answers to background questions For this question the same observation holds as for the previous question. The actual advice to view certain information in a specific component on the website only occurred once and the distribution is spread. On the other hand, when looking at the results of the subsets of participants both sets provide significant results. Given that the mothers who indicated that they wanted to turn off the virtual assistant explained that they found the virtual assistant not of added value or wanted to search for the information themselves, this result is comprehensible. With this feeling towards the virtual assistant, advices given will most likely not be experienced as pleasant. This same explanation holds for the significant effect found for groups 3 and 4, as 3 out of 4 mothers who wanted to turn off the virtual assistant also belonged to the subset of mothers who decided not to have their daughter vaccinated. Question 12: information about progress The mothers were supposed to be notified about their progress once they finished all the components in the main menu, however, none of the mother finished all the components within the time set for this part of the focus groups (30 minutes). Therefore, none of the mothers encountered this adaptation provided by the virtual assistant which explains the median of 2. A did not encounter this adaptivity answer option would have been good, as the scores provided would then include only scores that involve the questioned phenomenon. From the answers it is now not possible to verify why 2 participants scored this question with a 4 and 5, as it is certain that none of the mothers finished all the components and thus encountered the adaptation. The enormous difference between the results of groups 1/2 and groups 3/4 (p < 0.04 and p < 0.99) can only be explained by the fact that the actual adaptation was not encountered. Therefore, it should be ensured in a next implementation that the mothers encounter the adaptation in order to properly evaluate it. 13

16 Question 13: feedback based on answers to knowledge questions The results from this question are quite noticeable, as most of the participants provided a score of 3 or higher (11 out of 14), none provided a score of 2 and 3 participants provided a score of 1. Two of these participants indicated that they wanted to turn off the virtual assistant. Unfortunately, it is not clear why the third participant provided this low score from the answers provided or the remarks in the group discussion. Looking at the results from the Wilcoxon-Mann-Whitney test, the p of indicates a possible difference for this type of adaptation between the mothers who decided to have their daughter vaccinated and those who did not. This could be investigated further in a next implementation. Question 18: adaptive navigation support From the group discussion it appeared that most of the mothers found the navigation support in the main menu self-evident, which corresponds with the results from the Likert scales. This self-evidence might be a result of familiarity on other website, where the appearance of the links also change when information is visited. The results from the analysis of groups 3 and 4 show a significant effect for this question. From the group discussion it can not be explained why mothers who decided not to have their daughter vaccinated found the navigation support in the main menu less self-evident than mothers who decided to have their daughter vaccinated. This should be investigated further to ensure that the effect still holds for a larger group of participants. If that is the case then the navigation support in the main menu could be different for mothers who indicate the intention not to have their daughter vaccinated in the final implementation of the website. In order to realise this, it should also be investigated how this adaptive navigation support should be different. Question 23: presentation of adaptive information The results show that most mothers felt that the information presented was the information that they were looking for. This feeling can also be strengthened by the fact that the navigation support in the main menu was experienced as good, as it apparently led them to the information they were searching for. The lower scores (2 or 3 on the Likert scale) can be explained by the fact that some of the mothers indicated that the information felt very didactic or pro-vaccination. Mothers who were searching for objective information and experience this feeling probably indicated that the information presented was not entirely what they were searching for. As the mothers who indicated this, include the mothers who decided not to have their daughter vaccinated, the significant significant effect found for groups 3 and 4 is explained. Given that the objective information that was searched for was not always found by these mothers, the information presented did not correspond entirely with the information that was searched for. Explanations for the results of the remaining virtual assistant related questions The significant effect found concerning the contact with the virtual assistant (question 6) corresponds with the observation that some mothers wanted to turn on and others wanted to turn off the virtual assistant. The mothers who wanted to turn off the virtual assistant found the contact with the virtual assistant less pleasant and vice versa. The p found concerning the support being more of added value and the feeling of being more understood by the virtual assistant (question 7 and 9) also agree with wanting to turn on the virtual assistant. The nearly significant effect of approximately of question 9 for groups 1 and 2 can be explained by the fact that 3 out of 4 of the mothers who decided not to have their daughter vaccinated also wanted to turn off the virtual assistant, because they found the virtual assistant among other things not of added value. 14

17 6 Conclusion and future work Based on the results of the focus groups the sub questions and research question can be answered. The proposed model is adequate in providing adaptive navigation support and adaptive presentation of information. The presented information was that information that was searched for by most of the mothers. However, the presented information was also too pro-vaccination for some of the mothers, while objective information was actually expected. Even though this is not an adaption problem, but a content problem, this should definitely be adjusted and evaluated again with mothers before the final implementation is finished, as the goal of the final website is to provide objective information. More specifically, this next evaluation should be conducted with a larger group of participants, which contains more mothers who decided not to have their daughter vaccinated, after the results of these focus groups have been used to improve the implementation. The results concerning the adaptive interactions of the virtual assistants indicate that the proposed model is not (yet) adequate. Given that some of the adaptations have not been encountered by the mothers and others encountered just once, it is recommended to implement more of this types of adaptions and evaluate them again with the target group. The results from the statistical tests indicate that differences in the experience of the adaptivity of the website exists between mothers who did/did not have their daughter vaccinated and mothers who wanted to turn the virtual assistant on/off. Again, these results should be verified with a larger group of participants, but the results indicate that different adaptations could be made based on these subsets of mothers. How these adaptations should differ exactly should be investigated further, but let s not forget that this was only the first development cycle of the interactive and personalised website to support mothers during decision-making about the HPV vaccination. Once the adaptive interactions of the virtual assistants are optimised based on the results of these focus groups, the model will most likely be adequate in providing the three types of adaptivity on the website. More research will be performed up to the moment that a website is developed that fits YOU mom!. 15

18 Appendices 16

19 A Questionnaire expert evaluation (in Dutch) Evaluatie van de website Nu u de website heeft doorlopen, zijn we benieuwd naar uw mening over de website. Met deze vragenlijst proberen wij meer te weten te komen over hoe u over de website denkt. De antwoorden kunt u ook gebruiken tijdens de groepsdiscussie. U helpt ons het meest als u ons niet alleen vertelt wát u vindt, maar ook waaróm u dat vindt. Wij willen u vragen om de vragenlijst kort en bondig in te vullen. Als de beschikbare ruimte te weinig is kunt u op de achterpagina doorschrijven (met vermelding van het vraagnummer). Naam:. Leeftijd:...jaar Het nummer van mijn laptop is: LTL. 1. Heeft u uw dochter dit jaar in laten enten? o ja o nee 2. Hoe zeker was u over de keuze om uw dochter wel of niet te laten inenten? o Ik twijfelde nog steeds toen ik de keuze maakte o Ik was zeker over mijn keuze 3. Hoeveel informatie heeft u verzameld om uw keuze te maken? o weinig o gemiddeld o veel Virtuele assistent 4. Wat vond u van de moeder en de dochter die zich voorstelden? 5. Hoe vond u het contact met de virtuele assistent? U kunt hierbij denken aan de manier waarop de assistent u aansprak, hoe de assistent bewoog enzovoorts. 6. Ik vond het contact met de virtuele assistent onplezierig o o o o o plezierig 7. Ik had het idee dat de virtuele assistent mij niet begreep o o o o o mij wel begreep 8. Ik vond de hoeveelheid contact met de virtuele assistent te weinig o o o o o te veel 17

20 9. Hoe ervoer u de ondersteuning van de virtuele assistent tijdens uw bezoek aan de website? Ik vond de ondersteuning niet van toegevoegde o o o o o wel van toegevoegde waarde waarde 10. Hoe vond u dat de virtuele assistent u hielp bij het verduidelijken van informatie (bijvoorbeeld omcirkelen en onderstrepen)? erg onaangenaam o o o o o erg aangenaam 11. Hoe vond u het dat de virtuele assistent u adviseerde op de website (zoals het geven van suggesties wat u kon bekijken)? erg onaangenaam o o o o o erg aangenaam 12. In hoeverre had u het idee dat de virtuele assistent u hielp bij de voortgang op de website (zoals aangeven wat u allemaal al bekeken had) helemaal niet o o o o o helemaal wel 13. In hoeverre had u het idee dat de feedback van de virtuele assistent op de gegeven antwoorden aan u aangepast was? helemaal niet op mij o o o o o helemaal op mij aangepast aangepast 14. Als u kon kiezen, zou u de virtuele assistent uit- of aanzetten? o uit, vanwege o aan, vanwege Heeft u nog algemene tips over het contact met de virtuele assistent? Denk hierbij aan zaken die u mistte en graag toegevoegd zou zien, of contact dat u juist overbodig vond. Website 16. Hoe ervoer u de vragen vooraf om de informatie op u aan te passen? Denk hierbij aan de vragen over opleiding en verzamelde informatie. 17. Wat vond u van het keuzemenu op de website? (Hiermee bedoelen we het menu met de blauwe cirkels waarin u de onderwerpen aan kunt klikken) 18. Ik wist op welke onderdelen ik moest klikken om de informatie te vinden die ik zocht. helemaal niet o o o o o helemaal wel 18

21 19. Dit wist ik omdat. 20. Wat vond u van de werking van de website? Denk hierbij de wachttijd voordat u naar de volgende pagina kon, de navigatie, of u verder kon komen etc. 21. Ik vond de website moeilijk in gebruik o o o o o makkelijk in gebruik 22. Wat vond u van de informatie die aangeboden werd op de website? Denk hierbij aan informatie die u goed vond, plaatjes ter verduidelijking, specifieke informatie die u mistte etc. 23. De informatie die ik zag was helemaal niet waar ik o o o o o helemaal wel naar ik naar op zoek was naar op zoek was 24. In hoeverre vond u de informatie sturend? helemaal niet sturend o o o o o erg sturend 25. Heeft u nog algemene tips voor de inhoud van de website? Denk hierbij aan zaken die u mistte (bij de hulp van de virtuele assistent en de onderwerpen) en graag toegevoegd zou zien, of inhoud die u juist overbodig vond. Dankuwel! Lever de vragenlijst aan het einde van de focusgroep bij de discussieleider. 19

22 References Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50 60, Richard A Powell and Helen M Single. Focus groups. International journal for quality in health care, 8(5): , E.A. van Weel. This site fits you mom! The design and implementation of a model underlying an adaptive website tosupport mothers during decision-making about whether or not to have theirdaughter vaccinated against hpv, June Graduation project BSc Kunstmatige Intelligentie. 20