MS Internet Explorer Vs Opera Comparative Usability Test Farrukh Sahar (800418 P-510) mrsahars@yahoo.com Usability from an industrial perspective DV-1301 Abstract This report presents comparative usability test of Internet Explorer versus Opera based on some important feature of both the web browsers. Usability attributes were based on ISO 9241 and 9241-11[1] standards. Different features of these two explorers were identified during the usability test. The usability test was based on involving a group of 3-students that were observed during their interaction with these web explorers. A number of new issues have been raised during the test with many important aspects. These aspects will be helpful for interaction designers, CEO s and software developers in making there software product more attractive from the user perspective then their competitors does. We then propose that the web explorer should include advanced and sophisticated functions, but their interface should hide the complexity of the contents, providing flexible and easy interaction appropriate to attract the users. 1. Introduction This report discusses the role of comparative usability testing procedure and its advantages in a competitive marketplace. That how we can perform the comparative usability test on competitor s products that are already on the market, in order to understand their strengths and weaknesses, and to help our product meet user s needs better than the competitor does[2]. The main goal of usability evaluation is to identify the main part of the obstacles, problems, and breakdowns for the user when interacting with a web explorer, being the usability the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments (ISO 9241-11)[3]. 2. What is usability testing The goal of a usability test is to improve the usability of a product, through a test that is planned on the basis of specific goals and concerns. [4]. 3. Goals of testing Usability testers constantly point out the work of others in software systems to make it more usable for the others. As professional usability testers we must be very careful to ensure the usefulness, accuracy and usability of our products. It will be very effective if we follow the ISO standards for usability like standards for Use in context, interface, interaction, and documentation and so on. Adopting the ISO standards covers all the issues regarding user interaction. 4. Usability assessment methods In this comparative usability test scenario we are measuring the performance by defining the tasks based on time factor with observation then Questionnaires and short interview for evaluation. Questionnaires are useful methods for studying how users use software systems. Measuring the performance of the software systems in addition gives an idea what features users particularly like and dislike. This method is best suited when you have limited time frame [5]. 5. Comparative usability testing out comes 1. It will determine the purpose of the product 2. The reasons why we are doing comparative usability tests. 3. Evaluation results will be useful in making recommendation for the developers, product planner and CEO. 6. Type of the test The type of the test is comparative usability test and the basic purpose of choosing comparative test is to measure the performance of competitor s product in various aspects [6] In our case the comparative usability test is used to measure the two web browsers performance based on the different ISO Standards attributes that deals with interface and interaction such as efficiency, satisfaction and leanability attributes from ISO 9241 standards [7].
6.1. The goal of the test In this usability test two different web browsers Microsoft Internet Explorer (IE) and Opera are evaluated based on some performance attributes of ISO 9241 and ISO 9241-11 standards [8]. 6.2. Purpose of the test Measure performances based on time for most important features of each web browsers. Find out and make a comparison of the user productivity while they are dealing with each of the web browsers. Determine the user preference in choosing the web browsers to get access to the internet applications. Determine which web browser best feasible for the user who has knowledge about the internet. 7. User profiles Three users were selected for our test. they are well experienced in computer field and belong from Computer Sciences Program. they all have an experience and knowledge about the Microsoft Windows XP 2008 which is important aspect because that was the operating System that we had used during the test.they all are also familiar with the functions of both web browsers such as: Where is the address bar in the web browser? How to navigate different web page within web browser? Where the metaphors are located in the web page? How to stop the page loading process of browser? All users had experience with the internet explorer and opera. All users have no specific preference in favour of any All users were participated in this test with their willing. [9]. 8. Test environment and equipment configuration We have gone through this test with our 3 group fellows belongs from Computer Science department in the computer lab then a time is allocated to each group members to perform a test. 8.1. Computer Configuration AMD Athlon(tm) 64 Processor 3200+1.99 GHz, 960 MB of RAM Microsoft Windows XP Professional, Version 2002 Service Pack 2 9. Conducting the test 9.1. Participant Complement Each participant provides a comfortable and relax environment before attempting a test. 9.2. Orientation phase Each participant got a short orientation and introduction that what is a purpose of this test. Ensure them that purpose of this test is the evaluation of the web browsers not the evaluation of the participants. Because some times doubts decrease the performance of the user. 10. The test monitor role The tester read out aloud so that participants can understand what tasks they have to perform. observing the users Check the time allocation for each part of the tasks. Conduct a short interview after completing the tasks.. 11. Performance test In the performance test there is a parts of the tasks that the participants were said to work out. This test based on the attributes from ISO 9241 and ISO 9241-11 [10] standards. 11.1. The time aspects To measure the task timing, when the tester finish the reading instruction aloud about the tasks the start time was recorded and then said go to the user. When the user indicates that he has completed the part of the task then the stop time was recorded. Users were provided a 4 minutes time for specific part of the tasks. If the user exceeded the provided time for each part of the tasks the Tester said him to stop working on current part of the task and move on next part. To check the downloaded timings the user ask to wait until the page downloads but if the page could not be downloaded then the user asked to press the STOP button. It is important in test to perform a task even in some cases where the page was not downloaded but the main concern is that the user should perform the part of the task in limited time allotted for that part.
While checking the time span scenario the browser faces usability problems according to ISO 9241 standards such as when browser faces troubles in connecting to sites. 12. Task scenario [11] The tester read out aloud so that participants can understand what tasks they have to perform. Both of the browsers are able to perform the same features that are added in the tasks. Tasks were switched between IE and opera. Means that it is possible that tester ask to perform the first task on IE then second task on Opera. It was done to avoid the preference aspect towards specific The tasks were developed according to the specific and most frequent features used in a web browser such as: Visiting web pages Opening a web file that is saved on hard drive Saving favourite web pages in a favourite folder Downloading contents such as files, pictures to the hard disk. Cleaning the temporary internet files Searching process on the internet using the 12.1. Task for Internet explorer and Opera Part 1: 1. Go on the URL http://www.yahoo.com 2. Click on the link Autos 3. Click on the link Vehicle Spotlight, wait until it completely downloads. 4. Save the picture on your hard drive which is on the current page. 5. Now back on your first http://www.yahoo.com page 6. Save this yahoo page in favourite folder. Part 2: 1. click the home button go to the browser home page. 2. Take out a print of the page that you are viewing. 3. Open and HTML file which is on your desktop using your web 4. Go on computers.com web site using favourite list. 5. Click on the image of a computer. Add this page to your favourite folder 6. back to the computers.com web site. 7. Go to the printer s page and add this to favourite folder Part 3: 1. Organize the favourite folder by grouping the list as yahoo and computers 2. Put the printer s site in the computers group. 3. Find lap tops information on the internet. 4. Remove entry in computers group. Part 4: 1. Clean out the temporary internet files. 2. Make http://www.yahoo.com your default web page. 13. Evaluation instrument 13.1. Questionnaires After testing both of these web browsers users were asked to complete a product satisfaction questionnaire based on 10 questions. The user can rank each of the browsers from 1-6 scale, 1 means they strongly agree with the stated statement and 6 meant they strongly disagree with the stated statement. For each browser, users rated according to given scale from 1-6 of the following questions: [12] 1. The features those are required to complete the tasks were available 2. I was able to complete the tasks effectively 3. i was able to complete the task speedy 4. I was able to complete the tasks efficiently 5. I really feel comfortable to work with that 6. I was satisfied with the working of the browser 7. The browser was easy to use 8. Working with the browser is pleasant to use 9. The browser was quick in response 10. I was able to easy recover from errors. 13.2. Short Interviews After completing the questionnaire the users asked the following questions 1. Which browser the want to choose for accessing the internet? 2. What are the main reasons and their preference when they are rating the questionnaire? 3. Is there any specific difference that they found while using the browser? 14. Task timings The time result is based on the time taken by each part of the task and the overall time taken by each
This shows the time factor regarding the time taken by each browser but not showing that user completed a tasks correctly or in correctly. Time for each part of the task were assumed to be = 4 minutes = 240 seconds. 15. Task time results based on allotted time Time for 4 parts of task =960 IE Opera USER 1 688 seconds 853 seconds USER 2 573 seconds 689 seconds USER 3 592 seconds 744 seconds 1000 800 600 400 200 0 IE Opera Tasks completion time taken in IE and Opera 15.1. Result user 1 user 2 user 3 According to of efficiency attribute from ISO 9241 standards the IE has better performance then Opera. 15.2. Task success rate The task success rate refers to what is the percentage of successfully completed parts of the tasks with in the time allotted for each part of the task that is 4 minutes. 16. The parts of the task will not count in the successfully completed tasks in following conditions 1. If the user gave up in completion of the task. 2. If the user exceeds the 4 minutes time allotted for each part of the task. 3. If the user incorrectly completed the part of the task. 17. Satisfaction questionnaire Result After getting the result from questionnaire: IE got 1.22 according to satisfaction scale. Opera got 2.17 according to satisfaction scale. In a short interview tester had raised some of the issues regarding interface and functionality point of view. 18. Results for CEO The comparative usability test results for the CEO based on IE and Opera is best suitable in respect of his responsibilities. This comparative usability test gave an overview of company s product status and its competitor s product status. It will help CEO in making company s future strategy to satisfy its users in better way more then their competitor does. 18.1. Test results All the users completed each part of their tasks in allotted timings better in IE then Opera. The satisfaction level of user in concern of both the web browsers was quite satisfactory according to the tested features results. 18.1.1. Draw backs in IE and opera. The major draw back in the opera is that it is not allowing the use of multiple windows at the same time. The major draw back in the IE is that if error occurs then whole window will close down with all pages resides in that particular window and it s not possible to access again those pages which were opened in that window. 8.1.2. Advantages. The major advantage in opera is that it allows the access to the pages from its history those were opened in the tabs in a single window and closed downed when error occurred. 19. Recommendations for interaction designer based on Satisfaction questionnaire Result When Designing interaction models, workflows, and user interfaces concentrate on the context in which the application is going to work on. Concentrate on maintaining the speed as well as accuracy.
20. Tested Task Results for interaction designer. 20.1. On appearance When we first look at the Opera window Its is hard to find in Opera that where the STOP button is because in opera the RELOAD button become STOP button only when the page starts to load. In IE both STOP and RELOAD buttons are visible to avoid confusion. 20.1.1. Recommendations. Make visual design such as metaphor, buttons, and icons visible to the user so that user can perform actions without confusing. Make necessary functions visible to the user and hide unnecessary features. 20.2. Searching the internet When search the internet the IE shows the results in a more readable form and its quality is better then Opera. 20.2.1. Recommendations. Always think about the ways to make it easier and more significant for the people who use them such as in presentation of results must be in a readable form of text so that a user can easily read out the searched results. always think in a user perspective not as an expert perspective. 20.3. Bookmarks IE favourite folders are more manageable then opera Bookmarks when we perform grouping in managing favourite web pages. 20.3.1. Recommendations. While managing the sub features of a specific function try to make it simple and try to avoid complexity and provide a pleasant working environment to the user. Like in the case defined above some times user feel complexity in managing groups. It is good idea to work according to ISO standards in a specific context of use. 21. Tabs option. 21.1. IE aspects We can open multiple tabs in IE browser and multiple IE browsers at the same time. 21.1.1. Draw backs. if one page faces an error in any of the opened tabs then whole browser window will close downed with all of the opened tabs inside it. 21.1.2. Advantages. if error occurs then the page that faced an error in any of the tabs inside particular IE browser will only be closed downed not all of the other opened IE browsers pages. 22.1. Opera aspects Opera supported multiple tabs but not multiple opera windows at the same time. 21.1.1. Draw backs. if one opera window closed then all of the tabs reside in the window will be closed downed. 21.1.2. Advantages. if we start the opera next time again then we can continue from the last session where we had encountered the error. 21.1.3. Recommendations. there should be any recovery feature such as in opera case if any error occur it will make all the pages in the tabs again visible when the user next time open a web There should be error tolerance attribute from ISO 9241 standard. 22. Future work In this comparative test we had conducted a test on web browsers based on some of attributes from the ISO 9241 standards. In Our future work we will conduct a test on the web browser on the attributes from ISO 9241-11 and ISO/IEC 11581 and we will try to test as much features as we can so that we are able to provide a list of usability problems for the developer with the recommendations that how these problems can be solved. 23. Terminology After getting the positive response from the entire three participants it shows that the participants followed the instructions according to the terminology that was used.
24. Conclusion All the users completed each part of their tasks in allotted timings better in IE then Opera. The test didn t not suppose to bias for any web The satisfaction level of both the web browsers was quite satisfactory in the tested features. The participants in the test prefer to use the IE against the Opera but they agree that both of these web browsers are good in their functionalities. The major draw back in the opera is that it is not allowing the use of multiple tabs at the same time. The major draw back in the IE is that if error occurs then whole window will close down with all pages resides in that particular window and it s not possible to access those pages which were opened in the tab. The major advantage in opera is that it allows the access to the pages from its history those were opened in the tabs in a single opera window and closed downed when error occurred. The overall presentation of the icons on both the web browsers was quite reasonable and user feels comfortable while dealing with it. I think the most important thing is that the IE is built-in with the Windows and mostly peoples are well aware of the functionalities of IE then Opera. It might affect a little on the judgment but not on overall task scenario. The tests were formed and conducted after taking consider all the aspects. 25. Remaining questions regarding this assignment Q 1: Write about how the test is related to theories in the area of usability? Testing in Software Engineering Testing a process that is performed to support quality assurance, and provide confidence in the quality of software, and an emphasis on software quality requires improved testing methodologies that can be used by practitioners to test their software. [13] According to Nielsen, the most fundamental usability method, which is in some sense irreplaceable, is usability testing. This is so since it provides direct information regarding people s use of computers, and of their problems with the interfaces that are tested. [14] Q 2: Write about how the activity of testing is related to the previous parts of the Course? The usability testing is the most vital feature of usability. When we discuss the usability testing it combines all the aspects of usability ranges from GUI design patterns to ISO standards. In previous occasions in the course we discussed usability issues from the aspects of efficiency, learnability to satisfaction and so on. These all issues are important aspects in usability testing. we have to make consider these all issues when we are dealing with usability testing. Usability testing goal is to improve usability of a product. Q3: Write down your reflections concerning what you have learnt from choosing, performing, analyzing and presenting the results of the test. After performing the test i have reached on the following results: When Designing interaction models, workflows, and user interfaces concentrate make focus on context in which the application is going to work on. Try to maintaining the speed as well as accuracy. Make visual design such as metaphor, buttons, and icons visible to the user so that user can perform actions without confusing. Make necessary functions visible to the user and hide unnecessary functions. Always think about the ways to make it easier and more significant for the people who use them such as in presentation of results must be in a readable form. When making an application always think in a user perspective not as an expert perspective. While managing the sub features of a specific function try to make it simple and try to avoid complexity and provide to user a pleasant working environment. 26. References [1]http://www.usabilitynet.org/tools/r_international.htm,vi sited,date 10th may 08 [2] (Jakob Nielsen, Usability Engineering. San Diego: Academic Press, 1993) [3]http://www.usabilitynet.org/tools/r_international.htm#9 241-1x, visited date 10th may 08 [4] (Jakob Nielsen, Usability Engineering. San Diego: Academic Press, 1993) [5]http://www.usabilitynet.org/tools/methods.htm, visited date 10th may 08
[6] http://www.usabilitynet.org/tools/methods.htm visited date 10th may 08 [7] http://www.usabilitynet.org/tools/r_international.htm visited date 10th may 08 [8]http://www.usabilitynet.org/tools/r_international.htm, visited date 10th may 08 [9] By Mike Kuniavsky, Observing the User Experience: A Practitioner's Guide to User Research, Morgan Kaufmann Publishers 2003 [10]http://www.usabilitynet.org/tools/r_international.htm, visited date 10th may 08 [11] by Mike Kuniavsky, Observing the User Experience: A Practitioner's Guide to User Research, Morgan Kaufmann Publishers 2003 [12]http://www.usability.gov/refine/learnusa.html, visited at 10 may, 08 [13](M.J. Harrold, Testing: A Roadmap [14] (Jakob Nielsen, Usability Engineering. San Diego: Academic Press, 1993)