Experimental Comparison of Existing Application Sharing with a Revised Version with Respect to Criteria of Usability

Transcription

1 Experimental Comparison of Existing Application Sharing with a Revised Version with Respect to Criteria of Usability Tanja Neumann Fraunhofer-Institute for Telecommunications Heinrich-Hertz-Institut (HHI), Einsteinufer 37, Berlin, Germany tneumann@hhi.fhg.de Abstract Obviously, usability is an important criterion for the acceptance and use of applications in the field of information and communication technology. Usability is particularly important for general-purpose software products such as Microsoft s NetMeeting. This paper presents a critical review of the application sharing functionality of NetMeeting. Application sharing allows two or more persons to use identical electronic Word, Excel or Photoshop documents and to cooperate on that basis. In an expert evaluation and first preliminary user test various usability deficits of the program were found. Subsequently the established version of application sharing was modified to eliminate usability deficits observed. On that basis an experimental user test was started to find out whether the modified version proves to be superior to the established one with regard to the effectiveness, the efficiency, and the satisfaction of use. Interim results obtained by now suggest at least partial advantages of the new version. Final results will be available at the HFT conference in December Key words: Application sharing, Usability, User test, Satisfaction Variables, MS NetMeeting 1. Background and motivation of the study The emphasis of the present paper is on the application sharing functionality offered by MicrosoftNetMeeting in addition to other functions such as video conferencing. Application Sharing (AS) gives two or several users the possibility to cooperate on the basis of a common application like Word, Excel, PowerPoint, Paint, and Photoshop. AS runs in principle on one computer and can be started only by its owner. Nevertheless it can be also used, as illustrated below, from other computers. To explain AS we differentiate between the INitiator (IN) and the Participating Person (PP). AS is possible in three variants: 1. Joint Viewing: IN can use his applications without any restrictions. However PP may only observe the activities of IN, e.g. may watch the movements of the mouse cursor or the opening of windows. Joint Viewing is often claimed to be helpful in e-learning, e.g. if a tutor wants to explain technical mechanisms or to illustrate which steps are necessary to insert page numbers in a word document.

2 2. Joint Editing: Both PP and IN may use all functions of office applications. Thus PP is allowed e.g. to change or store a MS Word text on the initiators computer. Joint Editing might be helpful in case both IN and PP are in charge of editing a report. Here both people can work on a text, changing alternatively and discuss it online. 3. Visual Awareness (Whiteboard): Here word, Excel or Photoshop documents are available to IN as well as PP. However PP only may use the documents in form of screenshots, can look at them and use a remote pointer to mark various spots of interest. 2. Usability deficits of existing application sharing and suggestions for improvement By an expert evaluation and a preliminary user test various usability deficits were observed while using application sharing. These deficits mainly were explained by ambiguous or unclear control references. Two cases of a total of eight deficits will be described in the following as well as the improvement suggestions for these: Case 1: As prerequisite of using any version of application sharing IN must establish a connection to another participant. Additionally IN has to enter the address of the participant (Fig. 1). Many users do not know which kind of address is admitted. Actually there are four different possibilities: (1) , (2) phone number, (3) computer address and (4) IP-address. In principal, this information is supplied by the system and can be retrieved by using the question mark ( Help icon) in the title bar of the window shown in Fig. 1.. Preliminary trials showed that the information about the four adress options usually was not found by the subjects. Figure 1. Place a call window of the established version To eliminate that problem a question mark was established close to the input field labelled Address (Fig. 2). Clicking on that question mark the information on the various addresses is presented by the system (Fig. 2). Figure 2. Place a call window of the new version Case 2: As shown in Case 1, the IP adress is admitted for establishing connections to AS participants. Many users do not know where to find it. Actually, the IP address is available within the program. One can find it when one selects the question mark Help (Fig. 3) in the main menu and then activates Info. Usually, the Info window contains descriptions of copyright details and no additional user relevant information. Therefore people expect the information about the IP address at other locations of the program, particularly they hope to find it when activating the "Tools" menu ( Extras in the German version ). Therefore, to avoid the above usability deficit the IP address was additionally included as an option in the "Tools" ("Extras") menu.

3 Figure 3. Main menu of NetMeeting 3. Experimental approach For the experimental comparison the established and modified versions were implemented as digital mock ups. To test both versions, five tasks had to be performed either using the established or the modified version. The tasks required to Task 1: find the four kinds of addresses for establishing a communication channel Task 2: search the IP address Task 3: activate joint viewing Task 4: activate joint editing Task 5: activate visual awareness (Whiteboard). Subjects used either the established or the new version of application sharing. Before the trial subjects were made familiar with the capabilities of the different versions. The usability of both versions was measured according to the established usability variables effectiveness, efficiency and satisfaction. The performance of a task was called effective, if the task was solved within ten minutes. The efficiency was defined by the time and the number of steps needed by the subjects. To assess satisfaction a questionnaire was designed, which allows subjects to rate both AS versions according to the following variables: Orientation and certainty as to which steps to take Ambiguity of terms used Boredom Enjoyment Non acceptability of the time taken to perform the task Overall satisfaction

4 According to the above trial concept 20 of totally 40 subjects were tested 1. The subjects selected had basic skills to handle PC and InterNet applications. They were aged between 24 and 65 years. Both sexes and different levels of education were represented. 4. Preliminary results In contrast to our expectations the established version proved to be nearly as effective as the modified one (see Fig. 4). Only with regard to the task establishing a communication channel the expected superiority was observed. This result might be explained as follows: Both versions were realised in the form of mock ups. To do that it was decided to make available only those options respectively functions that definitely were needed to perform application sharing. All other functions available within NetMeeting were inactive and could be recognised as such by the subjects. Therefore, when performing the experimental tasks the subjects had to choose between fewer alternatives than offered by the original NetMeeting version. Thus fewer distractors existed that could produce conflicts and errors when performing the various tasks. visual awareness joint editing tasks joint viewing IP-address communication channel number of tasks, succesfully accomplished Figure 4. Effectiveness measured by the number of tasks accomplished for the established version and the new version. This fact also might explain the results with respect to the second usability variable efficiency, measured by the mean number of steps needed. Here again the advantages in favour of the modified version were smaller than expected (Fig. 5), however higher than with respect to the variable effectiveness. This is evident, when one looks at the results for the tasks activating visual awareness, searching the IP address and establishing a communication channel. Similar results are obtained when efficiency is measured by the mean time needed to accomplish the given tasks (Fig. 6). 1 As only half of altogether 40 subjects are tested until now results with regard to the statistical significance are left out for the moment.

5 visual awareness joint editing tasks joint viewing IP-address communication channel number of steps (average) used for task accomplishment Figure 5. Efficiency, measured by the mean average number of steps needed for task accomplishment of the established version and the new version Similar advantages are obtained, when the efficiency is measured by the time needed for performing the tasks (Fig. 6). visual awareness joint editing tasks joint viewing IP-address communication channel time in seconds (average) used for task accomplishement Figure 6. Efficiency, measured by the mean average time needed for task accomplishment of the established version and the new version

6 unfavourable evaluation favourable evaluation overall satisfaction non acceptance of time enjoyment boredom ambiguity of terms orientation on each step number of evaluations Figure 7. Number of ratings in favour of or to the disadvantage of the established version and the new version Looking at the various satisfaction variables (Fig. 7) slight advantages are observed for nearly five of six variables. Of course, these data and also the data with respect to effectiveness and efficiency are based on a trial with only 20 subjects. This picture might change when the trial with altogether 40 subjects will be finished. 5. Resumee In an expert evaluation and a preliminary user test of MS NetMeeting's application sharing functionality, various usability deficits were found. Subsequently the existing software version was modified to eliminate usability deficits observed. Next, the modified version was compared with the established one with regard to the effectiveness, the efficiency, and the satisfaction of use. Interim results obtained by now suggest at least partial advantages of the new version. Final results will be available at the HFT conference in December In summary, the work presented is a contribution to assure usability, a variable regarded as one of various factors determining acceptance and use of internet, intranet services and software products. Accordingly deficits of usability are taken seriously. Usually, three approaches are used to avoid such deficits: One approach is to fight the problem at the root, e.g. by providing plug ins. Another approach is to develop generic solutions, e.g. interfaces of similar appearance that help people to use new applications without high effort. A third approach seems still to be neglected: Every new application causes its own specific usability problems, e.g. problems that result from ambiguous and unclear wording. Of course, also application specific usability deficits have to be taken seriously. This was done in the present study with respect to application sharing, a facility that, though not yet widely used, might become very useful, when existing bottle necks related to transmission bandwidth within access networks are removed.

7 References DIN EN ISO (1999). Ergonomic requirements for office work with visual display terminals (VDTs); Guidance on usability. Luckhardt, H. D.(Januar 2003). Labels als Mittel der Informationsarchitektur, DIN EN ISO (1999). Ergonomic requirements for office work with visual display terminals (VDTs); Guidance on usability. Luckhardt, H. D.(Januar 2003). Labels als Mittel der Informationsarchitektur, IBM Lotus Sametime 3 (September 2002): Microsoft Corporation (2003). Rosenfeld, L. and Morville, P. (1998): Information Architecture for the World Wide Web Designing Large-Scale Web Sites. Sebastopol 1998: O Reilly & Associates.