Evaluation of Adaptive Web Systems

Size: px

Start display at page:

Download "Evaluation of Adaptive Web Systems"

Frederick Cobb
8 years ago
Views:

1 INFSCI 2480: Adaptive Web Systems Evaluation of Adaptive Web Systems Peter Brusilovsky School of Information Sciences University of Pittsburgh, USA With slides of Michael Yudelson, CMU Three Types of Evaluation Formal Look at prediction of performance, complexity and learnability Task Analysis, GOMS Analytical Without users Heuristic Evaluation, Cognitive Walkthrough Empirical

Three Types of Evaluation Formal Look at prediction of performance, complexity and

2 Empirical Evaluation of Adaptive Systems Empirical evaluation assessing by observing in experiments, rather than appraising theoretical validity Adaptive systems created for practical use, hence empirical methods should be used for their evaluation. Empirical Evaluation Best for objective assessment of design Best for broadest range of usability and performance problems Requirements to do empirical evaluation 1. Careful planning 2. Careful execution 3. Users must represent actual user population 4. Must have expense account pay users

Empirical Evaluation Best for objective assessment of design Best for broadest range of usability and performance problems

3 Empirical Methods Focus groups Interviews Questionnaires Observations Passive Instrumented Think-aloud Contextual enquiry Expert reviews Cooperative evaluation Participative evaluation Usability testing Controlled experiments Evaluation in a Broader Context Given the abundance of existing techniques and their implementations, the evaluation of the already suggested approaches is as important as coming up with novel ones It is important to know whether a particular adaptive technique works, what is the context of its most potent application, and what is the scale of the adaptation effect it produces Lack of evaluation data and/or inability to generalize from it, is one of the major barriers on the way of adaptive methods to become mainstream technology Evaluation is an essential part of research and practical application of the adaptive technologies

as coming up with novel ones It is important to know whether a particular adaptive technique works, what is the context of its most potent application, and what is the scale of the adaptation effect

4 Quick Example Evaluation example System A non-adaptive [web] system System B adaptive version of system A Assess (evaluate) user satisfaction with working in system A or B (we assume users like B better) System interface (A/B) is controlled factor, independent variable User satisfaction measured, observable, dependant variable Agenda Variables Factors controlled (interface, time, etc) Quantitative vs. qualitative Subjective vs. objective User-related vs. system-related Study Designs Useful Tips Examples

independent variable User satisfaction measured, observable, dependant variable Agenda Variables Factors controlled

5 Quantitative vs. Qualitative Variables Quantitative measured on numeric scale (ordinal, interval, rational, etc) Speed, number of problems solved, number of pages read Qualitative (categorical) no natural sense of ordering (nominal scale) Navigational pattern (sequential, cyclic, etc) Learning style Subjective vs. Objective Variables Subjective involve personal opinion explicitly/implicitly expressed User satisfaction, perceived responsiveness of the system Objective involve unambiguous manifestation of phenomena Time spent, no. errors found, prediction quality Objectively measured variable may express subjective theory navigational pattern

Qualitative (categorical) no natural sense of ordering (nominal scale) Navigational pattern (sequential, cyclic, etc) Learning style Subjective vs.

6 User and System Variables User-related describe user characteristics Demographics (gender, age, SES, etc.) Observed performance (success rate, number of pages viewed, etc.) Typically collected using questionnaires System-related describe system s operation Average response time, no. concurrent queries, perceived response delay (etc.) Typically collected using system logs Quick Test Give example of Quantitative variable, qualitative variable Subjective variable, objective variable User-related variable, system-related variable

) Typically collected using questionnaires System-related describe system s operation Average response time, no.

7 Design Single condition vs. multiple conditions Between-subject, within-subject, and crossed Time-constrained vs. longitudinal Holistic vs. layered Single vs. Multiple Conditions Condition setup of the environment/system Multiple conditions Non-adaptive vs. adaptive system comparison (2) Non-adaptive vs. adaptation 1 and adaptation 2 (3) Single condition Correlation of system use with success rate Compare time spent reading to understanding

8 Between- or within-subject Between-subject Each subject is only exposed to one condition Comparison is drawn between condition groups Control condition system without any studied enhancements (adaptation) control group Within-subject Subjects exposed to several (all) conditions E.g. both adaptive and non-adaptive system are available at the same time, user preference is studied Crossed design Subjects are exposed to all withinconditions Ordering is varied to compensate for practice effect. E.g. adaptive vs. non adaptive Group 1: Non-Adaptive, Non-Adaptive (control) Group 2: Non-Adaptive, Adaptive Group 3: Adaptive, Non-Adaptive Group 4: Adaptive, Adaptive

oup Within-subject Subjects exposed to several (all) conditions E.g.

9 Time-constrained vs. Longitudinal Time-constrained limited time for interacting with system Best for limiting uncontrolled factors Subject might have a bad day and not perform up to his true ability. Longitudinal time is not limited and/or regulated Less probe to the bad day effect Prone to uncontrolled nuisance factors Nuisance factor Intensive reading in addition to using an AHS Holistic vs. Layered Holistic (black-box) System is studied as a whole w/o attention to internal implementation Layered Adaptive systems are often multi-component, components are quite independent Study of the whole with special attention to each component (or layer) E.g. UMS and adaptation layers in the AS

10 Performance Evaluation Benchmark test Consistency / reproducibility of results Capacity planning test How far the system will scale under certain conditions Soak test Longitudinal (with control condition), for performance degradation, memory leaks over time Peak-rest test Load ramp-up and decrement Additional References Brusilovsky, P., Karagiannidis, C., & Sampson, D. (2004). Layered evaluation of adaptive learning systems. International Journal of Continuing Engineering Education and Lifelong Learning, 14(4/5), Carmichael, D. J., Kay, J., & Kummerfeld, B. (2005). Consistent modelling of users, devices and sensors in a ubiquitous computing environment. User Modeling and User-Adapted Interaction, 15(3-4), Chin, D. N. (2001). Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and User-Adapted Interaction, 11(1-2), Kobsa, A., & Fink, J. (2006). An LDAP-based user modeling server and its evaluation. User Modeling and User-Adapted Interaction, 16(2), Yudelson, M., & Brusilovsky, P. (2008). Adaptive Link Annotation in Distributed Hypermedia Systems: The Evaluation of a Service-Based Approach. In: W. Nejdl, J. Kay, P. Pu & E. Herder (Eds.), 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH 2008) (pp ), Hannover, Germany, July 29 - August 1, 2008: Springer, Berlin/Heidelberg Yudelson, M., Brusilovsky, P., & Zadorozhny, V. (2007). A User Modeling Server for Contemporary Adaptive Hypermedia: An Evaluation of the Push Approach to Evidence Propagation. In: C. Conati, K. McCoy & G. Paliouras (Eds.), 11th International Conference on User Modeling (UM 2007) (pp ), Corfu, Greece, June 25-29, 2007: Springer, Berlin/Heidelberg

Layered evaluation of adaptive learning systems. International Journal of Continuing Engineering Education and Lifelong Learning, 14(4/5), 402-421. Carmichael, D. J., Kay, J., & Kummerfeld, B. (2005).

Conference Navigator 2.0: Community-Based Recommendation for Academic Conferences

Conference Navigator 2.0: Community-Based Recommendation for Academic Conferences Chirayu Wongchokprasitti chw20@pitt.edu Peter Brusilovsky peterb@pitt.edu Denis Para dap89@pitt.edu ABSTRACT As the sheer