WEB PAGE AESTHETICS AND PERFORMANCE: A SURVEY AND AN EXPERIMENTAL STUDY

WEB PAGE AESTHETICS AND PERFORMANCE: A SURVEY AND AN EXPERIMENTAL STUDY Kristi E. Schmidt, Michael Bauerly, Yili Liu, and Srivatsan Sridharan Department of Industrial and Operations Engineering The University of Michigan 125 Beal Avenue Ann Arbor MI 489-2117 Corresponding author s e-mail: krischmi@umich.edu Abstract: A dual-process research and evaluation methodology was used to identify the underlying clusters of design variables affecting aesthetic judgment of a Web page, and to examine user preference, ease of interaction, and interaction speed for Web pages with various font and graphic sizes. To identify the clusters of variables, 57 design variables were identified by conducting a content analysis on relevant literature and by conducting structured interviews. A balanced incomplete block survey of the 57 variables was administered. Cluster analysis of the results revealed underlying clusters, two of which were selected to conduct a 2 experiment that explored Web pages with two levels of graphic size and ten levels of font size. User preference and ease of interaction increase as font size increases and graphic size decreases. There was no difference in interaction speed among Web pages with varying font or graphic sizes. 1. INTRODUCTION The World Wide Web (WWW) has grown in user population and breadth to a great extent since its creation over three decades ago. It is now viewed as a backbone for several sectors including advertising and marketing, business and ecommerce, entertainment, healthcare, communication, education, religion, and government (Nua, 23). There were nearly 414 million global home Internet users who each spent an average of over twelve hours online during the month of July 23, up 1.46% from the previous month (Nielsen//NetRatings, 23). The breadth, volume, and accessibility of the Internet has made it popular for individuals and organizations to create and maintain Web pages that go beyond communication to collaboration and even Internet commerce. The surge in Internet presence was not, however, paired with widespread design savvy or consideration for usability. The increased complexity of the Internet and the sheer volume of Internet users make the World Wide Web a very complex and often competitive environment. If users are unable to find what they need from a given Web page due to the lack of information or the complexity of navigation, they will become frustrated and move on to another site. On average, users spend one minute viewing each Web page (Nielsen//NetRatings, 23). This relatively short amount of time demands the Web page to communicate critical information rapidly and demands high information processing capability of the user. Variables affecting user judgments of a Web page have often been intuitively defined and communicated by Web page designers through instructional Web page design manuals. Oliver (23) defines four principles of web interface design and development: 1) usability how intuitively or easily the media item is navigated and processed; 2) visualization creation of visually interesting and aesthetically pleasing media items while avoiding potentially distracting or unnecessary features; 3) functionality features of the media item and how useful they are for supporting a given task; and 4) accessibility tools that help users access the site in alternative formats and provide increased functionality. Burstein (23) groups Web page design variables into fifteen design elements: links, color issues, images, image maps, animated images, spacing, tables, frames, style sheets, cookies, JavaScript, Java, plug-ins, screen size, and file distribution. These online Web page design manuals are often based upon designer intuition and qualitative evaluation of existing Web pages. Research about Web page usability, preference, and performance has also been published in peer-reviewed journals, yet this research often represents qualitative survey results and literature reviews of case studies, rather than empirical quantitative research. Turner (22) identified seven categories affecting Web page usability: navigation, page design, content, accessibility, media use, interactivity, and consistency. Cox and Dale (22) developed a conceptual model to assess how a Web page can meet user expectations based upon six quality factors in Web page design and use: 1) clarity of purpose; 2) design; 3) accessibility and speed; 4) content; 5) customer service; and 6) customer relationships. Design was 478

further broken down into five key issues: 1) links; 2) consistency, menus and site maps; 3) pages, text and clicks; 4) communication and feedback; and 5) search and fill-in forms. In addition to overall conceptual Web page design considerations being defined in a scientific and quantitative means, relative importance of the design elements that are defined must be determined and quantified in order to prioritize efforts in Web page design. Checklists such as the Heuristic Evaluation by Proxy (HEP test) of Web page usability (Turner, 22) aim to quantitatively and qualitatively evaluate several observations and criteria, yet checklists such as this are often based on surveys, compilation of existing literature, or case studies rather than empirical research work. Clearly, current literature lacks definition of factors affecting user judgments of a Web page based upon rigorous measurement tools and lacks further definition of the quantitative relationship among these factors. This study quantitatively obtains and determines relationships among Web page design variables. The questions addressed in this study were: 1) what are the ranked importance of design variables and the underlying variable clusters affecting the user s judgment of Web page aesthetics; and 2) what are the tradeoffs between Web page font size and Web page graphic size? A dual-process engineering aesthetics research and evaluation methodology (Liu, 21, 23) was used to scientifically and quantitatively investigate these issues. This dual-process methodology utilizes two parallel but closely related types of research methods that are aimed at achieving a comprehensive, rigorous, and quantitative understanding of aesthetic response, in this case with respect to Web page design. The two types of research methods that define the dual-process research and evaluation methodology and that are carried out simultaneously are multidimensional construct analysis and psychophysical analysis. Multidimensional construct analysis is a global top-down analysis that quantitatively answers questions involving the conceptual and mathematical structure of the aesthetic constructs involved in aesthetic judgment, the definition and measurement of the major psychological and physical dimensions involved, the identification of the relative importance and relationship of these dimensions, and the development of a multidimensional evaluation scale to measure the aesthetic construct with accurate validity and reliability. In this study, multidimensional construct analysis is used to identify variables of importance with respect to Web page design using content analysis and structured interviews, ranks those identified variables according to user preference by surveying several participants, and then uses multivariate statistical data reduction to cluster and factor the ranked variables according to relationships involving user perception taken from the survey s profile data. Psychophysical analysis is a local bottom-up analysis that establishes a quantitative view of how user preference changes as a function of specific aesthetic variables identified in the multidimensional construct analysis. Specifically, user ability to perceive and judge values, changes and variations in design parameters, and corresponding preferences of the levels of values of aesthetic variables are of interest. In this study, a psychophysical experiment was conducted to quantitatively investigate user preference, ease of interaction, and performance tradeoffs between two variables from two separate clusters of variables identified in the multidimensional construct analysis and to identify and establish a quantitative relationship among these variables. 2.1 Method 2. MULTIDIMENSIONAL CONSTRUCT ANALYSIS Texts were selected that addressed aesthetics, usability, and/or design guidelines for Web page design (Nielsen, 23; De Graff, 23; Gibbs and Szentivanyi, 23; Hom, 23; Ericsson, 23; Perlman, 23; Marion, 23; Burstein, 23; and Instone, 23). A content analysis was performed on these texts to obtain relevant variables affecting Web page aesthetics, Web page usability, and Web page design guidelines by extracting all meaningful words. Meaningful words are those words that aren t just transition words and proposition words (i.e. not a, and, then, with, or to). A parallel structured interview process was conducted upon twenty undergraduate engineering students. Participants ranged in age from 18-22 years, each had at least five years of Internet experience, and each logged onto the Internet at least once daily. Each participant was specifically instructed to list variables that they thought were important in affecting Web page usability while browsing Web pages at their leisure. The participants were encouraged to list as many items as they could think of and to not limit their thought and creativity. The results of the content analysis and the structured interview were combined to obtain a list of variables including frequency of appearance. Fifty-seven variables were ultimately extracted based upon a tradeoff between the total number of variables collected, the frequency of each variable collected, and the limitations of the survey design, including potential central fatigue of the participants due to the length of the survey. 479

A survey was conducted to rank the fifty-seven variables that resulted from the content analysis and structured interview. The survey used a Balanced Incomplete Block (BIB) design (Dunn-Rankin, 1983), in which a large set of ranking items are broken up into smaller groups. This survey design reduces the cognitive load on the participant by only having to rank 8 variables at a time rather than ranking the full list of fifty-seven variables. There were a total of fifty-seven small groups of 8 variables each that every participant (n=2) ranked, thereby comparing each variable to the rest of the variables twice. 48

CLUSTER RANK VARIABLE Page Progression/ Targeting Strategy Image and Text Balance Navigation Information Value Relevance/Speed Trust (Security) Platform Independence Marketing Appeal/Diversion Accessibility and Multimedia 4 Frames 38 Non-Frames 42 Opening of New Brow ser Window 34 Visual Design Cues 6 Visual Groups 53 Coordinated Audio and Video 36 Pictures Instead of Description 32 Simple Images 22 Graphic File Size 26 Font Size 48 Graphics for Graphics and Text for Text 28 Position in the Screen 24 Clear Exits 14 Wait-Time Feedback 29 Printable Contents 25 Navigation Support 3 Back Button 19 Grouping and Subheadings 27 Simple Headlines and/or Titles 23 Innovative 18 Provide Search 2 Length of an Article 17 Interactive 21 Minimized Scrolling 15 Simple Uniform Resource Identifiers (URIs) 39 Accurate Plain-Language Error Messages 2 Server Response Times 3 Time to Load 4 Dow nload Time 5 Speed 12 Timely Information 16 Updated Regularly 1 Information Layout 9 Location of Information 8 Credible and Original Information Privacy 7 Security 13 Brow ser Independent 11 System Independent 57 Advertisements 52 Banners 55 Sudden Pop-Up Window s 46 Animations 33 Graphics 35 Background Images 43 Entertainment 37 Drop Dow n Menus 31 Free Service 56 Songs 54 Movies 41 Games 51 Icons 45 Logo 5 3-D Images 49 Multiple Colors 44 Standard Colors for Links 47 Accessible for Users w ith Disabilities DENDROGRAM 481

2.2 Results Figure 1. Ten Clusters Consisting of 57 Ranked Variables and Corresponding Dendrogram Overall rank for the fifty-seven variables for each participant as well as an overall rank considering all participants responses is obtained from the BIB survey profile data. A multivariate statistical data reduction using K-means nonhierarchical and hierarchical cluster analyses as well as a traditional factor analysis and a factor analysis with Varimax rotation were also carried out on the profile data from the BIB survey. The cluster analyses identify clusters, or groups, of similar variables according to the underlying user perception of the variables. The factor analyses describe the relationship among the observed variables in terms of a few underlying, but unobservable, constructs called factors. Figure 1 illustrates the fifty-seven variables that were identified, ranked, and clustered. BIB ranking yielded the five most important variables affecting Web page design (most important first): information layout, server response time, time to load, download time, and speed. The five least preferred variables affecting Web page design (least preferred first) were: advertisements, songs, sudden pop-up windows, movies, and coordinated audio and video. Hierarchical cluster analysis performed using the BIB ranking results produced the dendrogram that illustrates the underlying relationships among the variables by grouping similarly quantified entities at various stages of relationship formation. Ten clusters were identified based upon these underlying relationships: page progression/targeting strategy, image and text balance, navigation, information value, relevance/speed, trust (security), platform independence, marketing, appeal/diversion, and accessibility and multimedia. Factor analysis yielded similar results as the hierarchical cluster analysis. The similarity of results between the factor analysis and hierarchical cluster analysis illustrates consistency among the various analyses and proves the method is valid. 3.1 Method 3. PSYCHOPHYSICAL EXPERIMENT Two clusters of variables identified in the multidimensional construct analysis were selected for the psychophysical experiment: 1) appeal/diversion; and 2) image and text balance. One variable from each of the two clusters, respectively, was selected for this illustrative psychophysical experiment: 1) graphics; and 2) font size. This phase of the experiment quantitatively examined how user preference and ease as well as interaction speed is modified as a function of Web page graphic size and Web page font size. Twenty participants aged 22.3 to 29.3 years (mean 24.7 years, standard deviation 1.9 years) participated in the experiment. All participants had normal or corrected-normal vision and normal color vision. Each participant accessed the Internet on average at least two hours a day, and each participant has had at least four years of prior Internet experience. The participants were compensated $. for approximately one hour of their time. Each participant participated in twenty experimental trials that involved viewing twenty Web pages that were designed based upon The New York Times on the Web (23). The participant was instructed to read the article text displayed on the Web page and then to click a link at the bottom of the page when they were finished. At the conclusion of every trial, participants answered a four-choice multiple choice comprehension question about the article and then to rate their preference for the Web page and ease of their interaction with the Web page on a scale from (low) to (high). The total time the participant spent viewing each Web page was also recorded. Among the twenty Web pages each participant viewed, the content of the article, the graphic size, and the font size varied. A total of twenty different news articles were selected for the experimental stimuli and then reduced in length to between 18 and 19 words. There were two graphics that corresponded to each article, a small graphic that ranged from 124-3 pixels high by 175-2 pixels wide, and a large graphic that ranged from 224-382 pixels high by 552 pixels wide. The two graphic sizes corresponded to the small picture/large picture format of The New York Times on the Web (23). There were ten font sizes possible for each article: 7.5, 8.5, 9, 9.5,,.5, 11, 12, 13, or 14 point. Twenty articles, two graphic sizes, and ten font sizes combined to create four hundred unique Web pages. No article content/graphic size/font size condition was replicated within-subject or between-subjects. Each participant viewed each article once, each graphic size ten times, and each font size two times. Each Web page was presented to the participant on a traditional 17 inch CRT visual display terminal with 6 Hertz refresh rate and 128x24 pixel resolution. Participants used a mouse with a scroll wheel as an input device. 482

Time (seconds) 6 5 4 3 2 Large Graphic Size Small Interaction time Preference Ease of interaction Figure 2. Average Interaction Time, Preference, and Ease of Interaction for Each Graphic Size Condition 9 8 7 6 5 4 3 2 1 Rating (=low to =high) Time (seconds) 6 5 4 3 2 7.5 8.5 9 9.5.5 11 12 13 14 Font Size (point) Interaction time Preference Ease of interaction Figure 3. Average Interaction Time, Preference, and Ease of Interaction for Each Font Size Condition 9 8 7 6 5 4 3 2 1 Rating (=low to =high) 3.2 Results A 2 x repeated measures analysis of variance (ANOVA) was performed with the 4 data points from the experiment (2 participants, 2 graphic sizes, font sizes). The statistical analysis was a within subject design. The within subject measures included Web page graphic size (small or large) and Web page font size (7.5, 8.5, 9, 9.5,,.5, 11, 12, 13, or 14 point). Figure 2 displays the average interaction time, the average preference rating, and the average ease of interaction rating by each graphic size condition. The main effect of graphic size on interaction time was not significant (p=.1771). The main effect of graphic size on user preference was significant (p=.27). The main effect of graphic size on ease of interaction was also significant (p=.179). Figure 3 displays the average interaction time, the average preference rating, and the average ease of interaction rating by each font size condition. The main effect of font size on interaction time was not significant (p=.4913). The main effect of font size on user preference was significant (p<.1). The main effect of font size on ease of interaction was also significant (p<.1). 4. DISCUSSION AND CONCLUSIONS This study applied a dual-process engineering aesthetics research and evaluation methodology to Web page design evaluation. The multidimensional construct analysis (top-down) approach yielded 57 ranked variables affecting user judgments of a Web page as well as identified ten clusters grouping the 57 variables that reflect the underlying mental structure of the user preferences. The ten clusters summarizing user judgment of a Web page are: page progression/targeting strategy, image and text balance, navigation, information value, relevance/speed, trust (security), platform independence, marketing, appeal/diversion, and accessibility and multimedia. The 57 variables (Figure 1) and ten clusters identified using the dual-process engineering aesthetics research and evaluation methodology are relatively consistent with previous studies that were based upon intuition, case studies, and literature review alone. The ranked variables as well as the ten descriptive clusters provide valuable insights to Web page designers regarding the underlying motivations and perceptions of Web page users in a quantitative and analytical manner as opposed to an intuitive generation of a prioritized list or a literature review based upon case studies or qualitative surveys. The psychophysical analysis (bottom-up) approach illustrated the ability to quantify relationships between variables that summarize user judgment of a Web page. Results showed that user preference and ease of interaction increase as font size increases and graphic size decreases, however, there was no difference in speed of interaction among Web pages with varying font size or graphic size. This finding provides useful insight to Web page designers that performance may not be the best indicator of Web page preference or ease of use. This psychophysical analysis also provides an 483

introduction to further investigation of relationships among the variables underlying clusters identified in the multidimensional construct analysis. Future study may investigate further issues such as age, task of Web page browsing (general purpose browsing or a directed fact-finding search), content of Web pages (user interest or choice of the researcher) (Marchionini and Shneiderman, 1988), and different methods of achieving high level of aesthetics without significantly degrading loading speed. 5. REFERENCES 1. Burstein, C.D. (23). Viewable with any browser: Campaign. http://www.anybrowser.org/campaign/ 2. Cox, J. and Dale, B.G. (22). Key quality factors in Web site design and use: An examination. International Journal of Quality and Reliability Management, 19(7): 862-888. 3. De Graaff, H. (23). HCI index. http://degraaff.org/hci/ 4. Dunn-Rankin, P. (1983). Scaling Methods. Lawrence Erlbaum Associates, Hillsdale, New Jersey. 5. Ericsson, M. (23). HCI resources: Guidelines, styleguides, standards. http://www.ida.liu.se/~miker/hci/guidelines/ 6. Gibbs, S. and Szentivanyi, G. (23). Index to multimedia information sources. http://viswiz.gmd.de/multimediainfo/ 7. Hom, J. (23). The usability methods toolbox. http://www.best.com/~jthom/usability/usable.htm 8. Instone, K. (23). Usable web. http://www.usableweb.com/ 9. Liu, Y. (21). Engineering aesthetics and aesthetic ergonomics: A dual-process methodology and its applications. Proceedings of the International Conference on Affective Human Factors Design, pp. 248-255.. Liu, Y. (23). Engineering aesthetics and aesthetic ergonomics: A dual-process methodology and its applications. Ergonomics, (in press). 11. Marchionini, G. and Shneiderman, B. (1988). Finding facts vs. browsing knowledge in hypertext systems. IEEE Computer, 21(3): 7-79. 12. Marion, C. (23). Software design smorgasbord. http://www.chesco.com/~cmarion/ 13. The New York Times on the Web. (23). http://nytimes.com/ 14. Nielsen, J. (23). Jakob Nielsen on Usability and Web Design. http://www.useit.com/ 15. Nielsen//NetRatings: The global standard for digital media measurement and analysis. (23). http://www.nielsennetratings.com/news.jsp?section=dat_gi 16. Nua. (23). Online Internet surveys, demographics, statistics and market research. http://www.nua.ie/surveys/index.cgi 17. Oliver, K. (23). Web Interface Design. http://www.edtech.vt.edu/edtech/id/interface/index.html 18. Perlman, G. (23). HCI Sites. http://www.hcibib.org/hci-sites/ 19. Turner, S. (22). The HEP test for grading Web site usability. Computers in Libraries, 22(): 37-39. 484