Application Sharing Services. Design and Performance

Transcription

1 Application Sharing Services Design and Performance

2

3 Application Sharing Services: Design and Performance PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Delft, op gezag van de Rector Magnificus prof. dr. Ir. J.T. Fokkema, voorzitter van het College voor Promoties, in het openbaar te verdedigen op dinsdag 25 april 2006 om 13:00 uur door Jenny Joyce BEUMER doctorandus in de cognitieve ergonomie geboren te Enschede.

4 Dit proefschrift is goedgekeurd door de promotor: Prof. dr. J.H.T.H. Andriessen Samenstelling promotiecommissie: Rector Magnificus, voorzitter Prof. dr. J.H.T.H. Andriessen, Prof. dr. F.W.G. van den Anker, Prof. dr. H.G. Sol, Prof. dr. W. Veen, Prof. dr. ir. G.J. de Vreede, Dr. M. Wiethoff, Technische Universiteit Delft, promotor Universiteit Hamburg Rijksuniversiteit Groningen Technische Universiteit Delft University of Nebraska at Omaha Technische Universiteit Delft

5 There are more things in heaven and earth, Horatio, Than are dreamt of in your philosophy. William Shakespeare, Hamlett, Act 1. Scene V Voor mijn vader, in nagedachtenis aan mijn moeder.

6 Colophon Published and distributed by: Jenny Joyce Beumer Van der Kamlaan KP DELFT The Netherlands Phone: +31 (0) English editor: Nick Rowling Printing and binding: Druk.Tan Heck, Delft Cover picture: Ans Beumer Jenny Joyce Beumer Application Sharing Services: Design and Performance Doctoral Dissertation, Delft University of Technology ISBN-10: ISBN-13: Keywords: new technologies, design, performance, application sharing, remote support, distance rehabilitation, remote control Copyright 2006 by Jenny Joyce Beumer All rights reserved. No parts of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, or otherwise, without the written permission of the author.

7 ACKNOWLEDGEMENTS My thanks to everybody who helped me with preparing my dissertation. It would not have been possible for me to finish it without all of the help I received. First, I will thank my husband Jelle. You gave me all the emotional support I needed, and took the care for our children Anneke, Saskia and Herbert. Moreover, you gave me a lot of content related help: you did the data analysis and found some excellent literature references. I want to thank my promoter and co-promoter, Erik and Marion: for content related help during the research and the fine-tuning of my dissertation. Erik, thank you for the long discussions that gave direction to the content of the dissertation. I was always looking forward to these appointments. Furthermore, I was really impressed by the many times that you read my dissertation to fine-tune the document. I thank you for all your efforts. I would also like to thank my previous supervisors. Ab de Haan, you helped me with the first part of my research: the foundation, the first experiments, writing the first articles and you motivated me to carry out this research. Harry Bouwman, you helped me a lot with structuring my thoughts and helped me to learn how to write scientific publications. I want to thank my father: thank you for all your corrections concerning language, in English and in Dutch. You read my complete dissertation several times in different phases. You really did a great job. Thanks to my student assistants Marijke, Erik, and Eline and Elise for typing out audio and video tapes, analyzing data and searching on the Internet and in libraries. Many thanks are due for the two organizations that participated in the casestudies. I would like to thank Stichting Het Loo Erf, the national rehabilitation center for visually impaired persons, and the sponsoring fund Vrienden van het Loo Erf, Rob Meijer, director of the center, Herman Philipsen, project leader, Bert Ebbe, technical support, Annemiek Wildenberg and all the employees and rehabilitants of the center. Without your help it would not have been possible to conduct this research. I also wish to thank the sponsoring organization Center for their additional support. I would like to thank the Dutch Inland Revenue Services (Dutch IRS), Harm Jan van Burg, commissioner, for arranging the financial support of the Dutch IRS, Maarten Visser, project leader, for all your assistance during the research: contacting the necessary stakeholders, coordinating meetings and for your availability every time it was required. My thanks also go to all stakeholders of the Dutch IRS for their input, and thanks to all employees and taxpayers volunteers who participated in the experiment. Finally, I would like to thank the funds and the volunteers who helped me in the last phase of my research, the funds `Landelijke Stichting voor Blinden en Slechtzienden, Stichting Blindenhulp, Stichting tot verbetering

8 van het lot der blinden en slechtzienden, and the Gelderse Blindenvereniging for their financial support. For the finishing touch I would like to thank Miranda Aldham Brearry for the English corrections, Nick Rowling for his English translations, Tamrat, who had just finished his Ph.D. at TPM, Helen and Lieneke, secretaries, and Ype, my brother in law, you made a lot of corrections, thank you. Finally, thanks to my sister Riëtte, and all my friends and relatives who supported me in practical ways at home and emotionally. You all made this dissertation possible. Thank you all very much. Joyce Beumer Delft, April 2006

9 Table of Contents 1 Introduction Setting the scene Research question Sub question A1: Sub question A2: Sub question B: Composition of the dissertation Chapter 2: Orientation on ASAS services Chapter 3: Method: ASAS services design and performance Chapter 4: Scenario s for ASAS based services Chapter 5: Experiment based evaluation Chapter 6: Scenario based evaluation Chapter 7: Discussion and conclusions Orientation on ASAS services Orientation on ASAS technology ASAS technology in Computer Supported Collaborative Work and GroupWare Systems ASAS technology in education and instructional systems ASAS technology in E-CRM and call centers Choosing ASAS technology Factors influencing performance in future ASAS services ASAS services and the expected performance effects Factors influencing performance Hypotheses Discussion and conclusions Method: ASAS service design and performance Towards a new design approach Designing ASAS services: EXSE - Early experiment and Scenario Based Evaluation Performance testing Testing the hypothesis methodology Organisational performance related Individual performance Discussion and conclusions Scenario s for future ASAS based services Method Procedure The interviews Casestudy one: Support for Dutch Visually Impaired Computer Workers... 62

10 4.2.1 Summary of preferred ASAS technology features Services and Additional Scenarios for VICWs Support with ASAS Services Casestudy two: Support for Dutch tax payers The Dutch Inland Revenue Services Preferred ASAS technology features Services and scenarios for taxpayer support Discussion and conclusions Experiment based evaluation Choosing ASAS technology for the 2 casestudies Experiment one: rehabilitation at a distance Results Conclusions Casestudy 2: Supporting taxpayers at a distance Method Analysis Results Summary and conclusions Scenario based evaluation Method Hypotheses Subjects Interviews Questionnaire Workshops Casestudy 1: Support for Dutch Visually Impaired Computer Workers Introduction Results concerning hypotheses Conclusions Casestudy 2: the Dutch Inland Revenue services Introduction Results concerning hypotheses Conclusions Discussion and conclusions : Discussion and conclusions Introduction Audio-Supported Application Sharing services Performance effects Performance effects as measured in experiments Expert judgements over performance effects Design method EXSE Comments about the design method Reflection on the experiment...129

11 7.4.3 Conclusion Answering the main question Generalisability Follow-up research Practical implications In conclusion References List of abbreviations Appendix 1: Available Application Sharing Software Appendix 2: Design related interview protocols and questionnaires Interview protocol Case-study 1: Support for Dutch Visually Impaired Computer Workers Interviewprotocol revalidant 5 april Interviewprotocol medewerker Loo Erf 12 april Interview protocol Case-study 2: Support for Dutch tax payers Evaluation questionnaire Casestudy 2: Support for Dutch tax payers..156 Vragenlijst Project Multimedia Support voor belastingplichtigen Nut en mogelijkheden van ASAS-Technologie bij het geven van Hulp Bij Elektronische Aangifte (HUBEA) Verwachte invloed van ASAS-technologie op kwaliteit van hulp bij aangifte Verwachte invloed van ASAS-Technologie op efficiency Waargenomen NUT van ASAS-Technologie Overige vragen Appendix 3 Design requirements for ASAS services Design requirements casestudy Technology related requirements Task related requirements User related requirements Official related requirements Organisation related requirements Design requirements casestudy Technology related requirements Task related requirements User related requirements Organisation related requirements Introduction related requirements Samenvatting Werkwijze...170

12 Prestatie effecten van een ASAS-dienst Ontwerp van ASAS-diensten Resultaten Individuele taak prestatie Voor- en nadelen voor de organisatie De ontwerpmethode Generaliseerbaarheid Vervolgonderzoek Tenslotte Summary Method Performance effects of ASAS services Design of ASAS services Results Individual task performance Advantages and disadvantages for the organization Design method EXSE Generalisability Follow-up research In conclusion About the author...187

13

14

15 Chapter 1 Introduction 1 Introduction Sunday, o clock: a visually impaired computer user phones a friend: I have been struggling all weekend. Last month I decided that I wanted to use mail at home, but my Braille line and speech module didn t want to show and read the mail messages. I phoned the supplier of my computer aids. He told me I had to buy another version of my Braille software, so I did. I had a lot of problems: first, it took me a long time to install the software: I had to make a lot of choices and I really did not know what to do. Then, after I installed this version, I could no longer use the Dutch language on my speech synthesizer. It was talking an illiterate type of English. Finally, after installation, I wanted to start the Internet Explorer, but I couldn t establish any connection due to a modem problem. It is not installed, and I couldn t find it in the list of modems. It took me my whole weekend! It s driving me crazy, can you please help me? 1.1 Setting the scene I encountered many computer problems during my rehabilitation period at the Dutch national rehabilitation center for visually impaired people Stichting Het Loo Erf, and observed that other visually impaired persons had similar problems. It is quite easy to imagine how difficult it is for a visually impaired person to work with a computer. First they have to learn to work with the aids, and only then they can start to learn a specific application, such as Microsoft Word. When using an application, Visually Impaired Computer Workers (VICWs) will be constantly hindered by their handicap: they always 'see' only a small part of the screen at a certain moment. In Braille, they read a half to a full line at a time; using a speech synthesizer, they hear all the information in succession, and with a magnification package they see again only a small part of the screen at a time. This means that VICWs have little or no overview of the screen, so that certain actions are less selfevident for them than for a well-sighted computer user (Beumer & Haan, 2000; Beumer, Jameson, & Waterham, 2000). As a result of the rapid growth in new computer applications and computer adaptations for VICWs there are long waiting lists for places at special computer training rehabilitation centers, and there is thus a need to find ways to decrease these waiting lists. A new technology concept called Audio Supported Application Sharing (ASAS) can be used in services to solve these computer problems at a distance. An ASAS service enables a trainer at a rehabilitation center to help a VICW by telephone and to look over their shoulder, because a copy of the digital information presented to the VICWs, e.g. a certain application or a web page, is sent to the trainer, who can see the problem directly. The trainer can point and click at an informationitem presented on the screen of the VICW (application sharing). The impact of this service, for both VICWs and society, could be great. VICWs will no longer need to leave their own environment to learn new skills, but can learn the skills they need at their place of work, or at their home, where they can apply them immediately. The colleagues of the VICW will see him or her slowly relearning how to function in the workplace. This makes rehabilitation easier. Later, VICWs can acquire customized rehabilitation care, contacting the rehabilitation center only if they need help. Having 15

16 Application Sharing Services: Design and Performance acquired the knowledge they need to work with the computer aids and applications they can contact the rehabilitation center on an incidental basis if they have problems with an application or a computer adaptation tool. Society profits from the rapid rehabilitation of the VICW, their brief absence from the labor market, and from a cheaper rehabilitation process. The use of ASAS services is not restricted to helping VICWs, but can also be expanded to encompass all computer users who have problems using their computer. ASAS services can be used to support helpdesks when clients phone in with computer related problems. They can be used to support distance education. They can be used for homecare advice for the sick, handicapped and elderly, and to support those who cannot go out to shop or visit friends and family. ASAS services can be used to support such groups digitally and to help them to become less isolated and dependent on others. Although the ASAS technology comes as standard with such frequently used systems as Windows 2000 (Microsoft NetMeeting ) it is infrequently used. This is a pity, because it could have a significant impact. Implementing the ASAS service concept to support VICWs at a distance would, for instance, give them the possibility to learn the skills they need at their place of work, where they can apply them immediately. Implementing ASAS services when providing helpdesk services, like the helpdesk of the Dutch Inland Revenue Service, would allow taxpayers to show their problems with their digital tax return form to a tax official. The tax official would then be able to see immediately what the taxpayers problem is and be able to provide a solution. Three reasons can be given to explain why ASAS services are not, as yet, universally implemented to support people requiring computer aided helpdesk/rehabilitation services. First, ASAS services are too new and too complex to know in advance how they will be envisioned. One reason for this might be that the organisational communication processes will change when ASAS services are increasingly commonly used in the future. Secondly, little is known about the way an organisation or individual users may benefit from an ASAS service. Three, even if it is known how the ASAS services will be envisioned, and how it may influence an organisations or users efficiency, effectiveness and satisfaction, it is difficult to persuade the various stakeholders involved in an organisation to adopt the new service. This is because they have to change their current communication processes and many people do not like changes. The development of a new service requires an extensive participatory design phase (De Sanctis & Poole, 1994). The reason for this is that the precise features of the service will depend on the future context in which it will be used. This context is not always a continuation of the current situation under consideration. Using a new service can sometimes change how an organisation will realize organisational goals e.g. its banking via the internet, and therefore changes the role of the organisation (Orlikowsky, 1992). Benefits, cannot always be predicted before a scenario has been developed (Van den Anker, 2003). In this research a participatory design approach is proposed to tackle this problem. Using this approach, possible areas of application will be charted and evaluated directly by looking at the current way of working. During this orientation phase, attention will be paid to the area of application, which will possibly come into existence after changing the communication processes of the organisation. Specific scenarios will be designed for different applications, in which the new service will be envisioned along with how the ASAS 16

17 Chapter 1 Introduction service concept will be incorporated in the organisation. This result will be obtained by using stakeholder interviews, the internal reports of the organisation in question and, if needed, a technology analysis in which the characteristics of the available technology are compared and matched to the organisational needs of the organisation in question. The scenarios will then be evaluated and further developed during interviews using an iterative process. During this process ideas extracted from previous interviews will be extended, or discarded, again using internal reports and a technology analysis if necessary. Throughout this process the interviewee will be free to give input to the process of designing the scenario(s). The second problem will be tackled by performing an experiment and prototype evaluation of the service-concept. The envisioned ASAS service concept described above will be used as input for this experiment. The experiment results will be used to show the organisation how it can benefit from the new services. This will be achieved in two different ways: at the level of individual behavior, and at the level of organisational performance. Measuring efficiency, effectiveness and satisfaction at the individual level requires task performance to be assessed. In this case, the tasks will be related to the user problems determined in the scenario, and the new service will be evaluated to see if its use decreases time needed to carry out specific tasks (efficiency), decreases incorrect answers (effectiveness) and increases the happiness of the persons who have to perform the tasks. These measurements will be obtained during an experiment based evaluation. Measuring at the organisational level requires that the productivity of the organisation is evaluated. In this situation efficiency, effectiveness and satisfaction can be defined as less money, time and manpower being needed to perform the services an organisation provides (efficiency), better and increased service can be provided using the new service (effectiveness) and the stakeholders involved in the organisation are more pleased with the system (satisfaction). Such measurements will be made during a scenario-based evaluation. The third problem of implementing the new services, deals with persuading the various stakeholders of an organisation to implement and use the new services, will also be solved implicitly using the participatory design approach, described above. The involvement of people from operational to management levels in such a process creates a broad base. One may expect that the stakeholders will become enthusiastic about performance at the individual level when they interact with the questioner during the participatory design process as resistance towards the system will be reduced due to a better design (Markus, 1983; Muller and Kuhn, 1993). One may also expect that managers will be enthusiastic about performance at the organisational level. In this dissertation the performance effects for two new ASAS services are described. To do so, it was necessary to design the services. Two casestudies are presented: one was carried out at the Dutch National Rehabilitation Center for visually impaired persons and the other at the Dutch Inland Revenue Services. It is beyond the scope of this chapter to describe the performance measurements and the design approach in detail. In this chapter the object of study will be described, give the motivation for the research and describe the research question. The research questions will be positioned in the next section along with an outline of this dissertation. 17

18 Application Sharing Services: Design and Performance 1.2 Research question The main research question deals with measuring performance effects of two future ASAS services. Action research will be used for the design of these services. This will be combined with experiments to measure the differences in performance regarding the current way of support and the ASAS service support. That is, the future service in which the current support is extended with application sharing. The main research question is formulated as follows: What are the optimal ASAS service features in relation to its context and how do these features influence individual and organisational performance? This research question needs to be divided into sub questions due to its complexity. This will allow step by step answering of the research question until the final, complete answer is obtained. The first two sub questions are context related. The first context related sub question consists of two parts. In the first part the object of research is defined: the ASAS service and its features. In the second part the main aspects of the ASAS service are considered with help of two casestudies Sub question A1: What are the existing ASAS service concepts, in which contexts can these services be used? The criteria that ASAS services should meet will be analyzed to answer this sub question. Then, an analysis will be presented of the state of the art concerning ASAS services and to what degree these services meet the criteria. ASAS service(s) will be described which match the context under consideration together with the method followed to determine an optimal ASAS service. Two different service contexts, or more specifically organisations, will be selected for this research given the above-mentioned ASAS service context match. Second, a description will be presented of how the service context can be envisioned. Aspects related to the context will be specified. Taking these aspects into account, a description will be given of how the service will probably change communication processes due to the introduction of the ASAS service. This will be achieved using a participatory design approach, this information will then be used to formulate a future scenario. Having determined the object of this research, the performance effects of ASAS services can be determined. These will be obtained in two ways: first, with help of an experiment and secondly, using stakeholder opinions, but first, a foundation has to be provided as the assumption that ASAS services will lead to changes in the efficiency, effectiveness, satisfaction and interaction intensity. Some aspects influence these processes and have to be determined. This assumption has to be supported using the available literature. 18

19 Chapter 1 Introduction Sub question A2: Do ASAS services allow consulting with higher effectiveness, efficiency and satisfaction and do they change interaction intensity compared to present forms of service provision? To answer this sub question, the available literature will be reviewed to give an explanation for the expected effects of new technologies such as ASAS technology on efficiency, effectiveness, satisfaction with and interaction intensity, and how aspects related to the context under consideration influence this. A theoretical model will be built to act as the foundation for the research presented in this dissertation. Secondly, the sub question deals with analyzing ASAS service performance effects on efficiency, effectiveness, satisfaction with and interaction intensity. Parts of the future application will be tested in an experiment-based evaluation in which the features of the ASAS service and its related context will be manipulated. Individual effects on effectiveness, efficiency, satisfaction and interaction intensity will be determined. The analysis of ASAS service performance will be further refined with stakeholder opinions about the future service using interviews, workshops and questionnaires. The scenarios and experiment fragments will be presented in the workshop with the aim of refining the contextual factors. Stakeholders will be asked to discuss the possibilities for generalizing the ASAS and applicability of ASAS services. The determination of performance effects connects all the parts of the research presented in this dissertation. The effects of participative development of the new service will promote a rich qualitative description of the service developed, given that so many different stakeholders, from workers to management, all with their own knowledge and different perspectives on the organisation will have participated in this process. The experiments will be used to test the performance and to visualize the ASAS service, this will deliver quantitative data about the expected efficiency, effectiveness, satisfaction with and interaction intensity when using ASAS. This data can then be used to help visualize new services. Sub question B also deals with the effects of all this information on the perceptions of the stakeholders of the organisation. The last question is process related. It deals with the question of how this research will be performed, and is related to choosing a design approach Sub question B: What is the appropriate design approach for this research? To answer this question it is necessary to provide background information on participatory design approaches and to clarify why, and how this approach was used for in this study. An explanation will be given of how the new services were designed, pre-tested and evaluated. The approach that will be followed to design the new services in which the ASAS service will be used in future will be discussed. The answer to sub question B will provide a foundation for the assumption made in the introduction regarding the need for a participatory design approach. Furthermore, the approach used in this dissertation will be compared with existing approaches found in the literature. It will be argued why and how these approaches are not appropriate for the context of developing future ASAS services. 19

20 Application Sharing Services: Design and Performance In the next section of this chapter a description is given of where the different research findings discussed above can be found in this dissertation to show how the different research parts are linked together to answer the central research question. 1.3 Composition of the dissertation Chapter 1:Introduction Chapter 2: Orientation on ASAS services Chapter 3: Method: ASAS services design and performance Chapter 4: Scenarios for ASAS based services Dutch National Rehabilitation Center for visually impaired persons Chapter 5: Experiment based evaluation Chapter 6: Scenario based evaluation Dutch Inland Revenue Services Chapter 7: Discussion and conclusions Figure 1-1 Composition of the dissertation Chapter 2: Orientation on ASAS services First, the precise meaning of ASAS services will be explained. Second, an overview of the domains in which ASAS services can be applied is given in this chapter. An explanation is given of the domains and their characteristics in which an ASAS service will be helpful. Confirmation is provided regarding why an ASAS technology was chosen to be the topic of research and the design requirements are given for such a service. These requirements are used to choose the ASAS technology for the experiments discussed in chapter 5. Further on, evidence from the literature is presented concerning the expected effect of the introduction of ASAS services on efficiency, effectiveness, satisfaction with, and interaction intensity, and the role of certain conditions in this context. This provides a partial answer to sub question A1. 20

21 Chapter 1 Introduction Chapter 3: Method: ASAS services design and performance This chapter discusses participatory design approaches. It is argued that some elements are missing in the existing design approaches. This asks for a new approach used in this dissertation, called Early experiment and Scenario based Evaluation (EXSE). An explanation is given of how EXSE is based upon an earlier developed approach and of the stepwise approach of EXSE. EXSE uses a combination of research approaches. This combination reflects the two parts of the basic research question. The what part asks for the development of a new service which uses qualitative data and the how part asks for an experiment (Yin, 1994). A partial answer on sub question B is provided in this chapter Chapter 4: Scenario s for ASAS based services In this chapter the design process for the two future ASAS services is described, and the chapter concludes with a discussion of the possible generalizations of ASAS service use towards other services in other domains. An answer to sub question A1 is provided in this chapter Chapter 5: Experiment based evaluation In this chapter experimental tests of Prototypes of the two ASAS services are described. It analyses the effects of ASAS services on efficiency, effectiveness and satisfaction, and interaction intensity. A partial answer to sub question A2 is provided in this chapter Chapter 6: Scenario based evaluation In this chapter the analysis of ASAS service performance is further refined with stakeholder opinions about the future service by means of interviews, workshops and questionnaires. In the workshop the scenarios and experiment fragments are presented with the aim to refine contextual factors. Stakeholders will discuss the possibilities of generalization and applicability of ASAS services. Second, a critical note will be placed to the development of stakeholder opinions. It will be discussed how their opinions changed through the participatory design approach Chapter 7: Discussion and conclusions In this final chapter first the main findings of this research are summarized, thereby providing an answer to the main research question: What are the optimal ASAS service features in relation to its context and how do these features influences individual and organisational performance? Finally, the possibilities of ASAS services will be discussed in general and a reflection will be presented on the research methods that are used in this study. 21

22 Application Sharing Services: Design and Performance 22

23 Chapter 2 Orientation on ASAS services 2 Orientation on ASAS services In this chapter an overview will be presented of Audio Supported Application Sharing (ASAS) services, their technological development over the years, the technologies and their use based on scientific research. l start with an orientation on ASAS service technology. An overview will be given of the domains in which ASAS services can be applied and how these influence the technological design guidelines. This will partially provide an answer to sub question A1: Which ASAS service concepts exist and in which context can this service be used? Then, evidence drawn from the literature will be presented concerning the expected effect of ASAS services. This information will be used to a model which factors will affect the efficiency, effectiveness, satisfaction and interaction intensity of future ASAS service use, and the role of certain factors in this context. This will partially provide an answer to sub question A2: Does an ASAS service allow consulting with higher effectiveness, efficiency, satisfaction and does using one change interaction intensity in comparison to the current way of consulting? 2.1 Orientation on ASAS technology Digital information is becoming increasingly integrated in our daily life. We can communicate with each other via and exchange documents, photographs, video fragments and use webcams during a telephone call ( nice for the kids ). Digital information is used in education to communicate with other students and teachers and it is used for commercial purposes and for public services: delivering information direct to the citizens at home without visiting an office at any time a citizen wishes to obtain such information. The technology used for these systems is more and more communication mediated. That is, these systems are used for communication between two or more persons and during this communication by text, video and audio can also be sent to the other participant(s). The focus of this research is a special sort, or way, of computer mediated communication: communication facilitated by ASAS technology. The services that use ASAS technology are called ASAS services. ASAS stands for Audio Supported Application Sharing. A lot can be said about this technology, and this will be discussed further in this chapter, but main characteristics of this technology are a shared audio connection and the possibility of sharing applications at distance. The idea of sharing applications with others working on connected computers is not new. In 1990 the first studies appeared about how to build infrastructures for shared multimedia applications (T. Crowley, et al., 1990). These studies, however, do not talk about 'application sharing', but about 'data-sharing'. Data sharing enables insight into large amounts of information, where users are dispersed and hard- and software varies. Data sharing (as the name already implies) is primarily concerned with data, and not about applications using such data. In practice, this means that data sharing is primarily used for providing different users, often at different locations, with the same set of data. These users 23

24 Application Sharing Services: Design and Performance should be able to use different ways of connecting to the data (modem, Internet, mobile, et cetera), and access to data should also be independent of operating systems (Windows, Linux, MacOS et cetera). The focus of this study is application sharing in combination with audio support. Application sharing software originates from data sharing software. Application sharing can be formulated as follows ( Wolf, 1995): 'The basic concept of application sharing is multiplexing of output streams from applications to terminals and multiplexing (filtering) of user input. The term terminal refers to any highresolution graphics display and its driver or display server software. Sharing is supported on the application level or window level: either single windows are shared or all windows of an application.' This development can be observed in different domains, all with their own goals for using this technology. One domain uses application sharing to assist computer supported collaborative work. Computer-based systems are developed especially designed to support groups of people working together, called 'GroupWare' (H. Ishi, 1997, p 435). This domain focuses mainly on the interaction aspects of the system being built. Learning effects, technical specifications and the relationship between different participants (customer or agent), receive minor attention in such kinds of software. A second domain deals with application sharing to develop and use instructional systems. Developing such systems ask for a clear understanding of the learning process for the developer to understand what the interaction of the official and client with the instructional system should look like. Application sharing can be used to facilitate this form of learning (J.J. Beumer et al., 2000). A third domain deals with application sharing to enable customer relationship management electronically (e-crm). Electronic customer relationship management uses a set of technologies and flexible implemented practices that can be constantly changed to service customers better (R.A. Gable, 1993). It uses some of the work of other domains: i.e. uses application sharing to improve telephones and helpdesk improvement. A fourth domain deals with the use of application sharing to improve telephones. It is not primarily concerned with the interaction between customers and agents, but focuses mainly on the technical part of the system. One of the developments in this field is call centers. A call center is a telephone contact point for a company or organisation that can be reached by a number of customers using one dialed telephone number. A fifth domain deals with application sharing as a means to enable better computer-support. Computer support is commonly realized via helpdesks. Helpdesks are a practical example of how application sharing can be used. Helpdesks have introduced the concept of application sharing in a lot of organisations. Customers can phone the helpdesk with questions and problems. The members of the helpdesk can solve these problems without leaving their own office. Help desks use web enabled call centers most of the time. This technology is combined with the technology used in GroupWare and Instructional Systems. In the next section the above mentioned domains will be discussed in more detail, and the points of attention for the ASAS technology will be drawn from these domains. These points 24

25 Chapter 2 Orientation on ASAS services of attention will then be used to choose a specific ASAS technology for an experiment, this will be discussed in chapter ASAS technology in Computer Supported Collaborative Work and GroupWare Systems Developing and testing systems and interfaces for systems that allow people to share information with others who are working on different computers is not new. Stefik et al. (1987) developed and tested one of the first multi-user interfaces in which people in the same room could share information presented at their own computers with others. This led to the current situation in which systems have been built and evaluated to allow geographically scattered users to work together. This includes seeing and hearing each other via a video and audio link, and sharing and manipulating information presented on the computer of the other persons (Ishi, 1997). In this section some examples are presented and some important interface aspects are described, that influence the interaction between users. These form the points of attention for an ASAS service. Points of attention The importance of What You See Is What I See, called WYSIWIS (Stefik et al., 1987) was first described when the first GroupWare systems were built and tested. WYSIWIS ensures that information presented on the computer screen of one participant is also visible to the other participants screens. If one of the participants alters the visible information, the other participants can also see this change. More of flexibility in WYSIWIS however, is sometimes needed according to Stefik et.al (1987). For example, if a lot of people are working together, and all the cursors of all the participants are visible at one time, this can be a source of confusion. If only the telepointer used when a person wants to point out something to all the other participants is shown, and not all the individual cursors, it increases the overview, although it departs from the WYSIWIS principle. Tang and Minneman (1990) developed and tested a video interface for a collaborative drawing tool for persons working in the same room. Using several experiments, they found a number of aspects that they used to produce some design guidelines, these are given below. Points of attention Hand gesturing is important to explain something to another person, other people. Users understand the drawings that are created better when time relationships are well timed. That is, when someone wants to show something to others, the technology enables this and does not disturb this due to delays in presenting screen changes to other users. The GroupWare study of Hayne, Rendergast and Greenberg (1992) confirm the above mentioned findings. They also discovered that the speed with which highlighted information 25

26 Application Sharing Services: Design and Performance can be transmitted influences overall task performance. In the same study Hayne, Rendergast and Greenberg (1992) gave concrete indications for the size of the cursor for several users working together. For two users, this is 64 x 64 pixels. If more than one cursor is used the cursors should be clearly different. Some discussion can be found about voice connection in GroupWare systems. Johansen (1992) describes how and when computer-augmented teamwork is used now and might be used in future. He describes how the development of GroupWare will lead to, for instance, the integration of telephone connections. Johnson distinguished four situations: collaborative work at the same location, geographically scattered locations, working simultaneously, or at different times. When people are working at geographically scattered locations simultaneously the case for our ASAS service Johanson claims that conference calls will increase. Johnson also claims that conference calls using PC graphics and image will become commonplace. He furthermore believes that video conferencing will show continual gradual growth, with some use of computer aids, with an emphasis on portable units and eventually desktop video ASAS technology in education and instructional systems The literature about instructional systems focuses mainly on how graphically scattered groups learn when communicating via electronic devices. The way officials and clients communicate influence the learning process and this, in turn, is influenced by the use of different devices like , chat and conferencing. Most of the time, little or no attention is paid to how object information can be used to support the learning process. Object information is information about a physical object, for instance a text document. So, most of the time literature about instructional systems neglects the importance of sharing applications. In this research the stress is placed on a different focus than most literature about instructional systems. First, it concerns how two persons communicate - the official (the agent ) and the client (the customer, also called client ) - and it is mainly not concerned with larger groups. Second, a hierarchy exists between official and client because the official gives the instruction and the client has something to learn. Third, object information in terms of sharing applications has a central position in this research. Some studies taken from the field of education, however, can be used for this purpose. Ellis (1992), for instance, distinguishes different cognitive theories that are relevant in the design of multimedia instruction. One of these theories is the contextualistic theory. This theory examines the interactions between the client, the environment and the system being used for instruction. Contextualistic theory is relevant for multimedia instruction, because it pays explicit attention to the clients interaction with the instruction system. Support via ASAS technology can be considered a specific example of this type of multimedia support. The following implications for multimedia instruction can be drawn from this theory. Use imagery to help internalizing the information. Pay explicit attention to the logic and thinking connections in the information provided. Use an event as the focus of the process, i.e. as an anchoring device. 26

27 Chapter 2 Orientation on ASAS services Ratcliffe et al. (1999) have developed, tested and evaluated an extended system that is used to support students in an academic environment. This system combines video and audio communication, including text-based communication, video conferencing, and application sharing to facilitate help desk questions, questions to tutors and questions about the Internet. The answers are given in real time and are not restricted to a specific geographical location. Ratcliffe et al. observed students and advisors in real time, gave them questionnaires, held interviews and analyzed information logged by the system. Radcliffe et al. developed a number of points of attention using their results. Only the points of attention not previously discussed are presented below. Points of attention One of the first design requirements described by Ratcliffe et al. is that the actual use of the system is dependent on the user friendliness of the system. That is, ease with which an action can be performed. This is not only dependent on the system, it is also user specific. So, users must be able to customize the system to their needs. During the development stage, Ratcliffe et al. found some technical criteria that proved to be crucially important: o How easy is it to plug the system into the existing network? o How compatible is the technology with standard sound and video cards? o How much bandwidth is available? ASAS technology in E-CRM and call centers Due to the shift from a production-oriented economy to one based on services (Cussack, 1998), Customer Relationship Management (CRM) currently receives more attention than it had in the past. One method of implementing CRM electronically is to use call centers, as they enable 'efficient and more personalized' services (Peter Keen: First a word about the positive effects of call centers on e-crm. This has been made possible by web technology, which can be used to create new options for accessing call centers (Melsoet, 2000). This allows customers to telephone the call center and to seek contact via other means of communication, such as fax, , or by contacting a company via its web pages. The value of integrating web technology in call centers has been noted by a lot of call centers. In a survey by the PELORUS Group in which 600 call center managers were questioned, over 90% of managers indicated that the Internet would be integrated into their call centers by the end of 2000 (Raritan, 2000). Using web technology, clients and agents are able to share customer and product information. Agents can help customers to fill out forms with the necessary customer information to buy an article via the Web, for instance. Or if a customer phones with a query about an order he or she has placed previously, the agent can show the order on screen, showing the customer that no further information about the order needs to be provided. 27

28 Application Sharing Services: Design and Performance In the next part of this section the technologies used at a call center will be divided into three technical layers: network layer, the equipment layer, and the report and registration layer. This will aid the description of the technologies involved when a call center is used as an ASAS service (figure 2-1), and it will enable the derivation of new points of attention for ASAS technology. Network layer Figure 2-1 Call center as an ASAS technology The first layer is the network layer, which provides physical access to call centers. The traditional telephone network transmits single phone calls to the call center. Earlier in this chapter I described how networks are used in the case of application-sharing. A call center makes it possible to combine telephone support with call-back facilities and video conferencing. Call-back facilities allow a customer to click on a button (the call-back button) when on the Internet; this indicates to the call center that the customer has a question and wishes to be called back. The call center can then phone the customer, either via the Internet (i.e. voice over IP), or on a second telephone line (ISDN, ADSL, or mobile). Video conferencing involves the transmission of a video image of the agent or customer. Two lines, mobile and fixed, or ISDN or ADSL, are also used in this case. Point of attention It is important to make an inventory of which communication methods can be used. 28

29 Chapter 2 Orientation on ASAS services Equipment layer The second layer, the equipment layer, is used after a phone call has passed the network layer. In this layer the phone calls are switched from the central call-in point to an agent. Automatic Call Distribution (ACD) switches the phone calls to a free operator known as an agent (Bayer, 1987). Computer Telephone Integration (CTI) is used to link the computer functions to the telephone functions. CTI includes applications that show details of the incoming calls on screen. ACD takes on added complexity if web technology is used. Usually, incoming messages are first divided by a server into two communication means: telephone calls, other communication types (e.g. fax, , chatting and application sharing) (Steul, 2000). This distinction is often made, as a classic PBX server can be used to handle ordinary telephone traffic. Currently, however, a server is being developed that can handle phone calls alongside fax, and chatting. One current trend is virtual phone call answering (Steul, 2000). This development started as interactive voice response (IVR): after dialing a call center, customers are not immediately switched to an agent, but are first asked to answer taped questions. The answers are used to help to match a customer to an agent who can answer his or her question. The virtual interaction component has currently been increased: information can be received when keys on the telephone are pressed, but the system also works with speech recognition. For example, when customers phone the Dutch railroad company, a voice computer asks them at which station they wish to start their journey, and at what time. The voice computer can deal with a low level of interaction, recognizing place names and times dictated by the customer. If it is unable to interpret what was said, the system can also ask the customer to repeat their answers. Points of attention It is important to notice how the incoming calls are switched to a free agent. See if interactive voice response (IVR) can be used. Look into the possibility of using speech recognition to recognize the answers given by a customer if IVR is used. One of the characteristics of call centers using CRM is that waiting times are reduced by the use of Automatic Call Distribution (ACD). The intelligence behind ACD makes it possible for customers to be informed about the current waiting time. If they are in a hurry and hear that they will have to wait too long, they can choose to use IVR instead. Report and registration layer The report and registration layer consists of software applications that withdraw information from the equipment layer, from the actual conversation with the agent, and from information previously stored in the report and registration layer. Information is gathered both when the customer calls (real time), and in relation to previous phone calls (history). Real-time reporting uses information that is collected in two ways: by 29

30 Application Sharing Services: Design and Performance automatic computer registration when the call passes the equipment layer, and by the notes made by the agent during the phone call. For example, the servers that distinguish which medium needs to be used such as or voice over IP can also perform automatic registration tasks. An agent registers information about the customer s questions. Most systems work with a kind of shorthand that is used by the agent to enter and retrieve customer data. If a customer calls more often, real-time reports can be used to draw up a customer history, showing how often a customer calls, and when and why. Points of attention Look at the possibility of gathering information in real time. o Look which kind of information is registered automatically. o Look at the possibility for the agent to enter information about a call in a database during the call. Look at the possibility to use a customer s history database. The report and registration layer has another positive effect on e-crm. Contact with the customer can be further personalized by linking real-time details with history and number identification because, for example, the agent knows what a customer commonly calls for. Sometimes it is possible for a fixed agent to handle calls from a particular customer. This can be particularly useful for dealing correctly with important customers, or when a defaulter calls. Second, the report and registration layer makes it possible for the company to make direct product deliveries to the client or deliver product information. The agent can also obtain information on stocks and orders, which, if necessary, can be passed on to the client. The ASAS technology used in chapter 5 best matches with the points of attention described above. The ASAS technology used for this match is described in Appendix 1. This comparison is based on the points of attention relevant for the domain in which it has to be applied, the general points of attention described below and the specific ASAS technology criteria needed for the casestudies in which it will be applied (see chapter 5). 2.2 Choosing ASAS technology In the previous section a description was given of how ASAS technology has been developed in different domains and the points of attention that can be drawn from this. The reasons why the ASAS technology became the topic of the research reported in this dissertation was are to be found in an experiment performed in In this experiment Beumer et al. (2000b, 2000c) undertook an experiment aimed at determining the specific points of attention that should be designed into a system intended for the provision of support to Visually Impaired Computer Workers (VICWs) dealing with computer problems. In this section I present an outline of this experiment and discuss the points of attention derived from the experiment. In the previous section points of attention were derived for specific domains. In this section some general points of attention will be presented that are important for all domains, but first the experiment of Beumer (2000) will be described and the first points of attention for the ASAS technology will be derived. 30

31 Chapter 2 Orientation on ASAS services The experiment involved 32 tasks given to 16 pairs of people. Each pair consisted of a sighted person who performed the tasks together with either a sighted person or a VICW. The persons in the experiment were homogeneous with regard to age, education, nationality and other relevant background variables. Gender was balanced. The communication strategies used to perform the tasks were analyzed. With regard to a support tool, two points of attention were established. Point of attention The official (agent) should be able to see what the client is doing on his or her screen in order to manipulate the information presented there. Motivation: Officials spent most of their time looking at how the client performed the task. This was how coordination of the task-related communication was achieved. Point of attention The audio connection is of great importance to official and client alike. Motivation: The official should be able to give instructions, and the client should be allowed to answer, both to give feedback and to ask questions when necessary. A support tool, the Audio-Supported Application Sharing technology (ASAS), was defined as a result of the experiment. It involves application sharing at a distance, accompanied by telephone support in one form or another (figure 2-2). Figure 2-2 Points of attention for ASAS technology 31

32 Application Sharing Services: Design and Performance Beumer and Van den Hooff (2000a) confirmed the importance of these points of attention. On the basis of several cognitive and communication theories, they clarify the added value of application sharing during telephone support. They argue that application sharing provides a rich context for information processing to take place, and that it therefore supports the cognitive processes whereby information is stored and retrieved. Support should be assisted, as this makes it possible for operations to be explained and exemplified for example, relevant actions can actually be shown to the user; imagery and movement can be used to help the user internalize the information; and the logic and thinking connections can also be shown. Making reference to the media richness theory, Beumer and Van den Hooff (2000) also argue the added value of application sharing with telephone support. This can offer information at varying degrees of richness, which will in turn support effective communication in a variety of tasks. The two general points of attention for ASAS technology described above application sharing and telephone support are a prerequisite. It is clear that these two points of attention need further refinements. The innovation characteristics are used as are described by Andriessen (1989) to describe the more specified ASAS technology points of attention. Andriessen describes how the processes of investigation, research, consideration and decision-making are needed before a new system can be introduced into an organisation. During this process, the organisation is influenced by three groups of variables: the characteristics of the innovation (such as costs, relative advantage and possibility to experience), the characteristics of the organisation (notably its innovativeness), and the characteristics of the organisation's environment (e.g. competition, coalitions, and support). Points of attention The costs of the technology should be taken into account. This can be divided into two components: buying the ASAS technology and integrating audio support in the system. o o Costs incurred to buy the application-sharing technology. Using freeware, software freely available via the Internet, can reduce these costs. Examples include VNC (Virtual Network Computing) and NetMeeting. Integrating telephone support in the system can be realized using a traditional telephone together with a multimedia p.c. This possibility use: ISDN or ADSL, i.e. one line for sending speech, one for sending data. traditional telephone (sending speech) combined with a cable modem, i.e. sending data. mobile telephone, sending speech, combined with a traditional telephone, sending data. traditional telephone, sending both speech and data via the Internet. Possibility to do exercises and gain experience. It is important that people can try a tool before they use it. 32

33 Chapter 2 Orientation on ASAS services Preferably free-ware will be used. Free-ware' application-sharing software can be tested and used most easily. The software can be downloaded free from the Internet. Look if enough communication means are available. For instance: o a mobile telephone needed to provide the combination of a traditional o telephone and a mobile telephone. a cable modem, an ISDN or ADSL connection is needed to provide a cable connection or ISDN or ADSL connection respectively. Look at the relative advantage. The relative advantage of the system consists of the advantages greater than the disadvantages given a specific situation. It depends on: o o o o userfriendliness of the system, that is, how easy it is to perform a desired operation. For instance, is it clear how the technology should be started and closed, is it clear how to perform an operation for another user, and how many actions are needed to achieve this goal. operating systems used, when a technology is chosen that uses another operating system than the one available, the relative advantage decreases. The common Operating systems used are Windows NT or Windows 2000, Linux and Macintosh. For instance, Timbuktu can be used on the Macintosh and NetMeeting for Windows. Platform-independent software can be used for all operating systems. Examples include Virtual Network Computing and Timbuktu. different situations require different networks. For instance, when large amounts of data must be transported, the relative advantage is increased if ISDN is used. Some application-sharing technology use ISDN, like PictureTel Live 200, others use the Internet (e.g. NetMeeting). relative advantage is also dependent on the quality of the audio connection. At this moment, voice quality is much better when using ISDN and ADSL. A mobile telephone connection has less quality, and the quality of cable is unpredictable at present. At present, voice via the Internet has the worst quality. The points of attention for ASAS technology were discussed in this section. These points of attention will be used to choose an appropriate ASAS technology for the experiments, chapter 5. This technology is the starting point for designing an ASAS service: the environment in which the ASAS technology will function. Although the technology has a central role in this environment, the service is also dependent on the different circumstances in which it will function. These circumstances are determined by different factors. Changing the value of these factors will affect performance. In the next section I will discuss how performance is expected to change and which factors values are expected to change ASAS service performance most. 2.3 Factors influencing performance in future ASAS services Before it can be argued which and how factors values are expected to change performance due to ASAS service use, it is necessary to discuss why the ASAS service is expected to change performance, see sub-section The expected performance changes, or, in 33

34 Application Sharing Services: Design and Performance other words the added value of ASAS service use during training and support will be explained with help of the literature. Then the factors that influence this performance will be discussed in sub-section This will lead to a research model and additional hypotheses ASAS services and the expected performance effects The added value of ASAS services during training and support can be explained with help of cognitive psychological theories and communication theories (Beumer and van den Hooff, 2000). Within the cognitive theories it is claimed that the more information available about the context in which a certain operation takes place, the easier it is for a user to perform the operation (Tulving, 1983). Tulving asked subjects to learn words and sentences while they were listening to music. The words and sentences could better be recognized if the music was presented. These results have been confirmed by several studies (d Ydelwalle, 1992). d Ydelwalle tested how people are facilitated by subtitles when they have to recognize photographs and sounds on television. Another research study has shown how pictures can be used to help people to recognize words faster (Schiffers, Meijer & Levelt, 1991). It can be argued that the added value of ASAS technology during support can be explained using communication theories like the work of Patty & Caccioppo (1996). They distinguish two routes of information processing: 1) The central (or cognitive) route, aimed at processing arguments, focusing on the message s content. 2) The peripheral (or affective) route, aimed at processing peripheral stimuli, focusing on the message s form. The theory predicts that both routes are of importance in information processing. Offering the possibility of processing information via both routes would lead to faster and better information processing. Paralinguistic and non verbal signals are primarily processed along the peripheral route, which is addressed by using different modes of communication more than just text. ASAS services provide the opportunity to address both the central and the peripheral route, enabling optimal communication both content and form (aimed more at relational information) are communicated. Another relevant communication theory according to Beumer and Van den Hooff that can predict the added value of ASAS technology are contextualistic theories. Using these theories it can be argued that better learning is provided due to explicit attention to the context in which learning takes place. Within these theories explicit attention is paid to the client s interaction with the instruction system. Implications for multimedia instruction from this theory are, among others (Ellis, 1992): use imagery to help internalizing the information, pay explicit attention to the logic and thinking connections in the information provided, and use an event as the focus of the process (as an anchoring device). The expected performance effects of ASAS service introduction, however, will be influenced by different factors. In the next section, evidence from the literature is presented concerning the expected effect of ASAS service use. This will be used to a model which factors will affect the efficiency, effectiveness, satisfaction and interaction intensity of future ASAS services, and the role of certain factors in this context. Hypotheses are formulated through this model. 34

35 Chapter 2 Orientation on ASAS services Factors influencing performance Figure 2-3 shows how the use of ASAS services affects performance during support. These effects are influenced by five factors (Beumer and Van den Hooff, 2000). These are: 1) characteristics of the task 2) user characteristics 3) environmental factors 4) characteristics of the design strategy 5) technological characteristics environmental factors task characteristics Use of ASAS services Performance technological characteritics introduction strategy user characteristics Figure 2-3 ASAS service factors affecting performance 1) Characteristics of the task The task determines the medium to be used. According to Daft and Lengel (1984, 1986), a richer medium will be preferable for a more complex task as this decreases the uncertainty about the task. In terms of this information richness theory, using an ASAS service during support provides a communication channel that can offer information of differing richness, which can then be used to support effective communication in a variety of tasks. Complex tasks require more memory capacity and more information-processing activities, and visualization can help to reduce the memory load of both customer and agent (Tulving, 1983; Gathercole & Hitch, 1993). Tulving states that people do not really learn words, sentences, stories or pictures with regard to memory phenomena these items (words, sentences, etc.) are remembered in certain situations, after people have been confronted with them repeatedly. These items are then activated anew, and through this process they are retrieved by a person s memory. Thus, what we actually remember is the episode associated with the presentation of a certain item. The more the actual context in which this process of remembering takes place is similar to the context in which the item was stored, the easier it is to retrieve it. 35

36 Application Sharing Services: Design and Performance 2) User characteristics We can take either an intra-personal or an interpersonal perspective. When looking at user characteristics, intra-personal factors have to do with cognitive psychology, with a focus on learning and remembering in individuals. In this context, differences between users depend on differences in individual information processing. These differences, in turn, depend on age, handicaps, computer literacy and the like. Handicaps and computer literacy are factors that may cause people to need more help when performing tasks. Beumer (2000) illustrates this in a study in which two groups of users had to perform computerized tasks: well-sighted and visually impaired persons (VIPs). VIPs needed more help and more time to complete the tasks. ASAS technology can be used to help them at a distance. S.J. Westerman et all (1995) state that cognitive capabilities are influenced by age, they confirmed empirically that people older than 50 year old need more time to process information and perform tasks. P. Verhaeghen & T.A. Salthouse (1997) confirmed these results with a meta-analysis in which 91 studies were compared. From an inter-personal perspective, we look at the processes that take place when information is exchanged, or in other words, when interaction takes place. Therefore the ability of users (both agent and customer) to fit medium to task is important (Daft and Lengel, 1984, 1986); as is the way they influence each other in these choices, and the way their respective social environments influence them in this, this affects the entire innovation process (Rogers, 1983; Andriessen, 1989). 3) Environmental factors The environment is an important factor in two ways. First as the social environment in which the adoption, introduction and incorporation of an ASAS service takes place. Negative experiences with a new technology (in this case: the ASAS service) in the past influences stakeholders opinions about the expected performance effects of the new service. Second, environmental factors are important as the physical environment in which communication tasks are performed. Noise and interruptions disturb these processes, and application sharing can compensate for such disturbances (Gathercole & Hitch, 1993). 4) Characteristics of the design strategy The design strategy must be geared to each of these factors. Current design approaches like participative design approaches take as given that the involvement of different stakeholders will lead to better system design and a more positive view of stakeholders concerning future effects on performance (see chapter 3 for more details). 5) Technological characteristics Finally, the technological characteristics are relevant in the sense that they can physically hinder or enhance the use of an ASAS service and the contribution this service makes to performance (Clement, 1993; Kiesler & Sproull, 1992). The more the actual context in which this process of remembering takes place is similar to the context in which the item was 36

37 Chapter 2 Orientation on ASAS services stored, the easier it is to retrieve it (Tulving, 1983). So, it can be expected therefore that ASAS services will increase performance as a result of this phenomena. These 5 factors give us the general idea that performance are influenced when using an ASAS service during training and support. Using this information, figure 3 can be specified further (figure 2-4). Figure 2-4 Prediction of ASAS service factor effects on performance Hypotheses Using this model as a basis some hypotheses were derived, that were then tested in the research study reported here. These are: 1. Technology related hypothesis: efficiency, effectiveness and satisfaction is higher if ASAS service is added to traditional way of consulting. 2. Task related hypothesis: the differences found in hypothesis 1 are larger for complex problems in comparison with simple problems. 3. User related hypothesis: the improvement of efficiency, effectiveness and satisfaction (Hypothesis 1) is larger for subjects who are confronted with 37

38 Application Sharing Services: Design and Performance constraints (due to e.g. computer illiteracy or handicaps), in comparison with subjects who are not confronted with these constraints. 4. Design related hypothesis: stakeholders will evaluate the service performance effects more positively at the end in comparison with the beginning of the service design process. 5. Communication related hypothesis: with ASAS services, subjects will communicate less than without ASAS. Rational In a previous experiment concerning communication of persons with visual constrains we formulated in-depth hypothesis with regard to visual constrains and how communication changes due to these limitations (Beumer et al., 2000). We discussed how these communication changes will occur due to humans limited cognitive capacity (see also Clark and Brennan, 1991; Levelt, 1981, 1989; McDonald, & McGurk, 1978; Sacks et al. 1974). This will in turn influence individual interaction intensity, and this will influence individual and organisational goals (see chapter 4 for more detail). The hypothesis in the Beumer et al., (2000) experiment covered the following hypotheses. Visually Impaired Persons (called VIPs) look less at each other than sighted people (SP's) and will use more auditory grounding to compensate for the fact that they cannot see the facial expression of the speaker (hypothesis 1). We also hypothesized that the sighted speakers have to exchange more information when engaged in a conversation (hypothesis 2). Furthermore VIP s ask more questions (hypothesis 3) and make more errors while performing tasks (hypothesis 4). With regard to turn taking we hypothesize that when VIP s participate in a conversation, more simultaneous conversation takes place (hypothesis 5). With regard to informal communication we hypothesized that speaker and listener use more informal communication when a VIP participates (hypothesis 6). We also expected that more time might be needed when a VIP participates in a (formal and informal) conversation (hypothesis 7). The hypotheses were tested in an experiment. These hypotheses can be generalized and will be transformed for the experiment, described in chapter 5, to explain communication changes in two situations: changes in technology condition (that is, telephone support, vs. observation based ASAS and intervention based ASAS) and changes in individual performance due to computer literacy and different experiences with digital tax return. The hypotheses concerning changes in the technology condition are formulated as follows. 5.1 Hypothesis: more time will be spent on tax related information when telephone consultancy is used than when ASAS services are used. The hypothesized reason is that tax agents need more time (more words and clarifications) to make their meaning about tax related issues clear to the subject, when using an ASAS service. This information is content related: related towards tax matters. 5.2 Hypothesis: more time will be spent on tax diskette related information when telephone consultancy is used than when an ASAS service is used. 38

39 Chapter 2 Orientation on ASAS services That is, tax agents need more time (more words and clarifications) to make their meaning about tax diskette related issues clear to the subject. This information is process and technical related: related towards filling in the right information at the right position. 5.3 Hypothesis: More time will be spent on issues of situation awareness when telephone consultancy is used rather than when ASAS services are used. Situation awareness is awareness of the physical location of certain information, presented on the digital tax return. 5.4 Hypothesis: More time will be needed for informal communication when telephone consultancy is used rather than an ASAS service is used. Informal communication is talk about non tax related topics and talk that is not about the digital tax return. 5.5 Hypothesis: More turn taking will take place when telephone consultancy is used instead of ASAS services. Turn taking is defined as the switch in communication from tax agent to subject and vice versa. Three hypotheses are not listed above, which were listed in the experiment discussed in Beumer et al. (2000). One hypothesis concerned the efficiency of increased media richness. This hypothesis is already answered above. The other two hypotheses concerned the relation between questions and errors made. These hypotheses were left out because it appeared to be not possible to define questions and errors unambiguously. For instance, that subjects sometimes formulated a question as a real question, and sometimes the question was implicitly interwoven in the question handling. 2.4 Discussion and conclusions In this chapter an overview of domains in which ASAS service are used, or will be used in the near future was presented. The actuality of these services was shown and how the ASAS service is increasingly being integrated into call centers. It also shows how difficult this technology is to integrate into actual work, for instance in education. The ASAS technology criteria were discussed and refined. This list is not intended to provide a finite description of ASAS technology criteria, nor is it always necessary that a certain technology meet all the criteria mentioned. This list is mentioned as a guidance, which can be used as a foundation for why a certain technology should be chosen, or not chosen. In certain cases some criteria are of crucial importance, in other case they are not. For instance in our case the cost of the technology was of great importance in the rehabilitation case, and also in the pre-test for the Dutch taxpayers case. But in interviews at the Dutch IRS stakeholders indicated that the technology cost is of minor importance, for them the most important aspects were usability and security related aspects. 39

40 Application Sharing Services: Design and Performance The criteria are clustered within domains, but this should also be reviled in practice. A lot of domains are overlapping, such as education and collaborative work. For instance, during education the collaboration with others is central, also the use of call centers is sometimes needed in education or in the area of collaborative work. A call center is important if more persons want to contact a certain person. The technological choice is also dependent on practical restrictions. Testing the ASAS technology using computer aids for VICW for instance, delivered information about how the application sharing software interacted with the aids. It appeared that some aids could not handle certain application sharing software. The VICW could not read the screen during application sharing. This kind of software is therefore not suitable in this situation, unless it is adapted to meet a lot of new points of attention. The factors influencing performance were discussed in this chapter. These factors form the base for the interview analysis and the experiments described in chapter 3. The hypotheses defined as a result of the analysis are answered step by step in the ASAS service design described in chapter 3. 40

41 Chapter 3 Method: ASAS service design and performance 3 Method: ASAS service design and performance The object of this study was presented in the previous chapter: Audio Supported Application Sharing (ASAS) services. An indication was given of the disciplines in which the service can be used, and of the factors which influence ASAS service performance. Furthermore, hypotheses concerning potential performance effects of these services were formulated. Two different methods will be discussed in this chapter, one, the design method that will be used to design these ASAS services, and two, the method used to test the hypotheses given in chapter 2, i.e. the performance testing method The design method deals with how the ASAS services will be developed. While the research method covers how, within the design process of ASAS services, the hypotheses will be tested to provide answers to the research question. The research method description and the additional answers obtained with respect to the research questions and are not necessary for the design, but they will help to shed light on the performance effects of the ASAS service design. The hypotheses have to be tested using the new design for ASAS services, this implies that the services have to be designed. To do this, I introduce a new design approach - called Early EXperiment and Scenario-based Evaluation (EXSE). In this chapter I explain how this approach was applied to the design of the new ASAS services. First some background to the participatory design approaches will be presented. I will argue that something is missing within the existing approaches and explain how EXSE can be used to solve this problem. This will allow me to give an answer to sub question B: What is the appropriate design approach for this research? 3.1 Towards a new design approach In the last few decades there has been an overwhelming growth in new service designs and design methods, in which new technology concepts play a central role. With the growing number of service designs, there has also been a change in ideas regarding how these services should be designed. In the 1980 s a rather technical approach was proposed. Services were developed in which the designer had a central role, and the designer decided what an adequate system would be to solve the problems described by the stakeholders of the problem. Such approaches to technical designs often failed to produce workable services for two reasons: one, the designer failed adequately to take into account usability (Nielsen, 1992a; Nielsen and Mack, 1994; Lindgaard, 1994). Two, the designers often did not take into account usefulness (Van den Anker, 2003; Nosek & Sherr, 1984; Ciborra, 1984). Usability is defined as follows: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use (ISO DIS ). Usefulness has a much broader view. It not only describes how a specific product will function with a specified user, but it will involve the whole context of use. Van den Anker describes usefulness in his dissertation as follows (Van den Anker, 2003, page 37): the usefulness of an ICT application is deeply interwoven with its context of use. Whether a system s implementation actually results in an improvement of organisational performance, will depend to a great extent on a wide variety of context of use factors such as the characteristics of the workers (e.g. their experience) and their tasks, 41

42 Application Sharing Services: Design and Performance the way tasks are divided among stakeholders, organisational procedures, cultural aspects and aspects of the physical working environment. In technical design approaches the designer has a central role in service development, this has led to a number of problems that can be generalized into two types (Ehn, 1998; Greenbaum and Kyng, 1991; Muller and Kuhn, 1993). 1) Approaches that are inspired by the technical possibilities of the system rather than their usability where the end users are not involved. 2) New systems that are the end product of the design process, which means that it is only possible to determine the usability of such systems after the system design is completed. If the system does not function well in practice, it is difficult to redesign the system. These observations point towards a need for end users to play a more active role during the design process, i.e. to take a user-cantered approach (Nielsen, 1992a; Nielsen and Mack, 1994; Lindgaard, 1994). In a user-cantered approach, prototypes, mock-ups and simulations are used to test and evaluate a new system during the design process. The test results are then used as the basis for further development of a system. User-centered approaches can be used to overcome the problems found in the technical design approaches. 1) End users are specifically consulted during the design process. This gives designers an idea of how the usability of the system can be improved and this will, in turn, increase the effectiveness of the designed system (Landauwer, 1995). 2) Integration of design parts, i.e. concepts, mockups and prototypes, within the design process provides designers with insight into how their final design will function in practice (Bødker and Grønbæk, 1991) and can prevent the need for a lot of expensive system redesign. This user centered method, however, also generates problems. 1) Usability does not automatically imply stakeholder involvement during the design process, stakeholders consist of all the organisation members that should be involved in the new designed system, from management to users. This can lead to resistance on behalf of users when it comes to actual use of the system, because important stakeholders have not been made aware of the advantages of the new system. Such resistance can cause adoption of a new service to fail (Van den Anker, 2003; Markus, 1983). 2) Traditional usability approaches fail with respect to integrating and combining the different interests and ideas of stakeholders regarding the current work flow activities of different stakeholders in the organisation (Van den Anker, 2003; Nosek & Sherr, 1984; Ciborra, 1984). 3) Problems and solutions are only observed in the beginning of the design process in usability driven design approaches. Therefore this approach fails to deal with the continuous environmental and organisational changes taking place within organisations (Van den Anker, 2003). Participatory design approaches were introduced to overcome these problems. Within these approaches single stakeholder involvement is replaced by active, multiple stakeholder involvement (Ehn, 1998; Greenbaum and Kyng, 1991; Muller and Kuhn, 1993) in which the 42

43 Chapter 3 Method: ASAS service design and performance development of compromises between the different stakeholder opinions play a central role (Egyedi and Boehm, 1998), and where an attempt is made to anticipate organisational changes during the design. Participatory design gives us the possibility to use and adapt service concepts, mockups and prototyping throughout a design process in which organisational changes are a key element. This was not possible in the traditional user centered design approaches, because the organisational changes were not taken into account in the design process. Testing usability and determining usefulness of a system asks for two completely different research approaches. Usability, related to questions of application use, focuses on how questions, i.e. determining the exact method required to achieve a result. For instance: How will end users work with the system? How will users find a solution for their problem,? Etcetera. Usefulness related questions focus on more exploratory what questions, for instance: What are the technological features of the system? What are the tasks which can be answered with the system? What are the user criteria? And: What are the organisational circumstances in which the system will work? How questions dealing with how some result will be achieved require an experimental approach to find answers. What questions are more exploratory and are better answered using surveys or archival analysis approaches (Yin, 1994, page 5). Applying a systematic combination of these research approaches during service design leads to a rich design, in which the relevance of the service is systematically analyzed using experiments. Moreover, a rich, exploratory design determined during a qualitative casestudy can be evaluated in an experimental setting. In other words, focusing on usability helps to produce systems that are easy to use, while focusing on usefulness helps to produce systems that match and organisational needs. This combination should provide us with concrete predictions of future performance and allow a designer clearly to envision the future situation. A combination of the approaches discussed above was used in the research reported in this dissertation. Combining inputs from the organisational context with experimental data leads to a need for changes to be made to existing participative design methods. This method will be called EXSE: Early experiment and Scenario-based Evaluation. In the next section I will first describe the foundations on which EXSE design approach is built. This is followed by a section about on how EXSE was used in this study. First I explain why EXSE was chosen as the design method of choice for the research presented in this dissertation. Although ASAS functionalities are a topic of research in a lot of research disciplines (see chapter 2), when this study was started, ASAS services were rarely used in practice. Nowadays ASAS services exist to some degree in the call centre domain, however, in other areas their use is still quite rare. There are various reasons for this. 1) It is not always directly clear in advance in which circumstances an ASAS service can be used. Many technology, tasks, users, and organisational characteristics have to be taken into account. 2) If these characteristics are investigated, it is still not clear how the service can be used. There is often no clear scenario of how the service will be used. Such a scenario is needed, because using an ASAS will change individual and organisational communication processes and organisations need to know under which circumstances the service will function. It is very difficult to develop a use 43

44 Application Sharing Services: Design and Performance scenario, it depends on the actual interaction processes of an organisation and the technological view of the organisation: how the organisation reacts to technological changes. Furthermore it is difficult to achieve a compromise between all the different stakeholder views. They will all have their own ideas and wishes. Sometimes a complete redesign is necessary because an important aspect of the technology (e.g. security matters opposed by a lawyer, or insufficient bandwidth opposed by a technician) or an organisational aspect (for instance financial matters opposed by a manager) has been forgotten. 3) Once a used scenario is made, it is still difficult to predict actual performance changes. Predicting changes is required because managers must know how the implemented service will affect their organisation. Only then managers can make a well-considered choice regarding whether or not to implement the ASAS service. 4) It is difficult to convince all members of an organisation that a new service should be used. It is important to convince all potential users, because resistance to use will make the new service less valuable for the organisation. 3.2 Designing ASAS services: EXSE - Early experiment and Scenario Based Evaluation The participatory design approach adopted in this dissertation is called EXSE, Early experiment and Scenario-Based Evaluation (see figure 3-1). EXSE is based on the design approach of Van den Anker (2003) discusses, ESE: Early Scenario based Evaluation. Van den Anker s design approach focuses on multiple stakeholder analysis which is used to predict organisational changes that may result from the adoption of new ICT. Van den Anker defines a conceptual framework for the development of new applications for multimedia communication for mobile work. He discusses how a participatory design approach, in which rich scenarios are combined with simulation, is needed to evaluate new ICT in this very early stage of development. Van den Anker s scenarios are divided into detailed descriptions of the current and future work context in which the new ICT will function. Phase 1 of EXSE consists of a literature overview, and is nearly identical with the ESE phase 1. A review of current ASAS service technology is performed with respect to its features, the ASAS service concept is explained with the help of literature and Internet sources. Points of attention regarding preferred ASAS service features are then constructed on the basis of this information. The results are used to determine a suitable ASAS technology for a selected casestudy. If more then one ASAS technology can be used, a choice of which technology to use will be made after the stakeholder interviews, which take place in phase 3. During this phase the designer will have the chance to discuss appropriate solutions with the stakeholders. In EXSE phase 1 the context in which the ASAS service will possibly be useful is described. Then, on the basis of the information found in this phase an application domain, or to be more specific an organisation, can be chosen for which the ASAS service will be useful in the future. A contextual analysis of the current situation of the organisation under investigation is made in EXSE phase 2. Again it is nearly identical to Van den Anker s ESE phase 2. This phase 44

45 Chapter 3 Method: ASAS service design and performance incorperates a contextual analysis of the current situation of the organisation under investigation to gain a deep insight in the actual work processes of the organisation. This analysis is conducted using internal reports and interviews in which different stakeholders from different levels of the organisation are interviewed. A detailed, written scenario should be defined during this process. This should include an elaborate description of design choices, points of attentions, expected outcomes, and of any expected problems that might be met during task performance. Field observations are used to refine this scenario further. A description of how the future scenario is envisioned is delivered in EXSE phase 3. Again the EXSE phase 3 is closely related to ESE phase 3. The description is made on the basis of the application concept as defined in phase (1) and the current situation description phase 2. The future work organisation is envisioned, using this information, the new technology is described in detail, along with future users, tasks, the physical working environment and expected organisational changes including communication processes and interaction mechanisms. EXSE phase 4 concerns the development of the future context of use scenarios, and again this phase is closely related to Van den Anker s ESE step 4. In this phase, some concrete situations are developed (textual scenarios) from the elaborate future context of use descriptions provided in EXSE phase 3, using a specific problem as input, a concrete storyline can be defined: i.e. a specific person, performing a specific action in a concrete situation. This scenario is then used to support a simulation of the future context of use for the ASAS service under design (see chapter 4). Note that these scenarios are possible design alternatives. They give the direction in which use of ASAS services can be envisioned, rather than a cost estimation. The future ASAS services context of use scenarios serve as input for the (development of the) experiment-based evaluation performed in the next phases, EXSE phases 5 and 6. The context described in the scenarios is as much as possible used in the experiment, for instance the tasks, users and technology characteristics. The scenarios are evaluated in the scenario-based evaluation performed in EXSE phase 7. The development of an experimental setting takes place during EXSE phase 5. In this phase the experiment that will be performed in EXSE phase 6 is designed. The start up phase of the experiment design EXSE phase 5 consists of a discussion with the key player of the project, a management team member of the organisation, in which scenario or combinations of scenarios are used as input for the experiment. The technology, tasks, users and organisational settings of the experiment are determined on the basis of the scenarios. EXSE phase 5 is completely different from ESE phase 5, Van den Anker s ESE phase 5 deals with the participatory evaluation of the scenario in a workshop attended by workers, managers, developers and other relevant stakeholders. This phase is not performed in EXSE, although there is some resemblance to Van den Anker s ESE phase 6, which covers the development of a simulation. This simulation is set up with a (paper) prototype of the new ICT and is used to envision the developed future scenarios. Tests are selected from the predicted target groups to play their role in the scenario. Resembling the experiment setting in EXSE phase 5, the selected stakeholders, problems and tasks presented cover a variety of expected future settings. Experiment based evaluation takes place in EXSE phase 6. This phase does not exist in ESE. The experiment is performed in this phase, one or more scenarios are tested empirically. 45

46 Application Sharing Services: Design and Performance Two functions are served in EXSE phase 6: One, differences in efficiency, effectiveness, satisfaction and interaction intensity between the old and the new service can be determined. Two, video fragments of the experiments serves can be used as an input for stakeholders during the scenario-based evaluation performed in phase 7. Finally, EXSE phase 7 consists of the scenario-based evaluation. This is the final evaluation of the scenarios developed in EXSE phase 4. In this evaluation the analysis of ASAS service performance is further refined using stakeholder opinions about the future service. In this phase the stakeholders will be asked to discuss the possibilities of generalizing the ASAS services and their applicability. This evaluation workshop is called the scenario-based evaluation, because the complete scenario is evaluated during this workshop. EXSE Phase 7 is somewhat different from ESE phase 7. In ESE phase 7, two parts can be distinguished: the running of the simulation and the evaluation of this simulation. In EXSE phase 7, not only the simulation is run, but also video fragments of the experiment are shown and written descriptions of the future context of use, represented in scenarios, are used to evaluate the complete scenario. In sum, the main difference between ESE and EXSE is the deletion of an evaluation workshop and an additional experiment phase in EXSE. ESE phase 5 is not performed in EXSE this evaluation phase was deleted because in EXSE the development of the future context of use is constantly evaluated during the interviews. A preliminary analysis of the current situation and the future context of use made on the basis of the previous interviews are given to each new interviewee with every new interview. The interviewee can agree with this information, add information to it, or make changes if necessary. If no changes are necessary and all the relevant stakeholders have been interviewed, the scenario can be declared complete. This scenario is then used as the base for the experiment setting. An experiment-based evaluation is added to allow the designer to observe differences in efficiency, effectiveness, satisfaction and interaction intensity between the old and the new service, it also serves as an input for stakeholders during the scenario-based evaluation performed in phase 7. The phases in EXSE are presented in figure

47 Chapter 3 Method: ASAS service design and performance Phase 1: Orientation on potential ASAS services domain Phase 2: Contextual analysis of current situation Phase 3: Envisioning future context of use Phase 4: Development of a future context of use Phase 5: Experiment development Phase 6: Experiment based evaluation Phase 7: Scenario based evaluation Figure 3-1 Performing Early experiment and Scenario based Evaluation (EXSE) 3.3 Performance testing The research method used for performance testing will be explained next. I will describe how, within the design process of ASAS services, the hypotheses discussed in chapter 2 are tested to answer the research question. These hypotheses can be split into two types: perceived organisational performance related and observed individual performance related. The perceived organisational performance related hypotheses cover stakeholders remarks which are used to judge the hypotheses on their realistic expectations. The individual performance related hypotheses cover changes in individual task performance. The hypotheses were tested in two different casestudies. Before the research methodology is described in detail, I will briefly introduce the two casestudies. In casestudy 1, ASAS services were designed to support Visually Impaired Computer Workers (VICWs) with various kinds of computer problems at a distance. Background for this service design is as follows. VICWs have obtained new opportunities to work in jobs due to the rapid growth of computer use (Edward and Levis, 1999). An increasing number of computer applications are available that enable VICWs to function in situations where in the past the barriers were impossible to overcome. Astroft (1984, p 108) stated that "Technology is one of the blind person's most powerful allies in overcoming the detrimental effects of blindness." VICWs can work with the same applications as Well Sighted Computer Workers (WSCWs) when using adaptations like speech synthesizers (Arato & Vespori, 1982; Blenchorn, 1982) and braille displays (Dune & Dune, 1999). New questions arise with the 47

48 Application Sharing Services: Design and Performance introduction of these adaptations, (Beumer, 2002). VICWs have to learn to work with new computer adaptations and applications. The rapid growth in new applications and adaptations causes long waiting lists for special computer training at rehabilitation centers and efficient methods of training need to be developed to decrease these long lists. One important solution to resolve these long waiting lists is to present on line help at a distance, to enable trainees to receive part of their training at home or to provide support for new users after they have finished their training. In casestudy 2, ASAS services were designed to support Dutch taxpayers with various kinds of computer problems. The background for this service design is as follows: the Dutch Inland Revenue has forecast an increase in the number of tax payers making their tax returns digitally and it is their policy to stimulate this. Individual automated self reporting will be stimulated by the Dutch IRS using help facilities like FAQ s (Frequently Asked Questions) and userfriendly interfaces, but at present there is still a large demand for telephonic consultations when problems arise. Often the taxpayer is not able to describe the problem clearly enough to the tax official. Detailed technical questions on taxation and usability problems cause tax officials to be confronted with increasing complicated information processing problems. The tax official needs detailed insight into the situational context (Tulving, 1983). While tax payers need to understand a tax official s instructions and need to obtain insight into the different complex interactions that arise. Combining meaningful images can help the problem owner to understand (Ellis, 1992), and recall information given to them (Gardener, 1987) for future use. A proposal was made to introduce ICT supported visualization techniques Audio Supported Application Sharing services - to help tax payers and tax officials to visualize problems and solutions directly (Beumer, 2000) Testing the hypothesis methodology The hypotheses are divided into two types (see above). Within the design the stakeholders judged the hypotheses on their realistic expectations and their changing expectations to test hypothesis 4 using an experimental setting, expectations about individual performance were tested with respect to hypotheses 1, 2, 3 and 5. The methodologies used are explained in detail below. 1. Technology related hypothesis: Efficiency, effectiveness and satisfaction are higher if an ASAS service is added to traditional ways of consulting. 2. Task related hypothesis: The differences found in hypothesis 1 are larger for complex problems compared to simple problems. 3. User related hypothesis: The improvement in efficiency, effectiveness and satisfaction (Hypothesis 1) is larger for subjects who are confronted with constraints, due to e.g. computer illiteracy or handicaps, in comparison to subjects who are not confronted with these constraints. 4. Interaction intensity or communication related hypothesis: Using ASAS services, subjects will communicate less than those without ASAS. 48

49 Chapter 3 Method: ASAS service design and performance 5. Design related hypothesis: Stakeholders will evaluate the service performance effects more positively at the end when compared to the beginning of the service design process Organisational performance related In order to judge the hypotheses on their realistic expectations and to test hypothesis 4, the following steps were taken. First a description was made of the current and possible future contexts of use envisioned according EXSE phase 3, and building on EXSE phases 1 and 2. Second, a future context of use was further refined and developed following EXSE phase 4. This future context of use was then evaluated in EXSE phase 7, on the base of the information collected in EXSE phases 5 and 6. The details are described below. Developing a draft of the current and future context of use The information collected in phase 2 was summarized as preparation for the interviews, and the current work activities and work processes of the organisation were described briefly. The information about current interaction processes was refined using field observations. Second, the information collected in phase 1 was summarized. This summary included an orientation on the ASAS service technology and on its main features, and explained the ASAS service concept and its generic functions. Refinement and elaboration of the current and future context of use The refinements and elaborations of the current and future context of use were determined during the interviews performed in EXSE phase 4. In the interviews, the aim was to collect stakeholders opinions about the conditions under which using an ASAS service coould be expected to increase task and organisational efficiency, effectiveness and satisfaction, and to determine how communication processes might change. The draft of the current and future context of use as described above, was used as the basis of the first interview. In this interview, I first gave the interviewee a summarized description of the current work activities and work processes of the organisation. Then, with the help of the interviewee the current work processes were refined and corrected. Then the ASAS service summery extracted from phase 1 was discussed. I asked the interviewee to think about the integration of the ASAS services into current communication processes. The interviewee was encouraged to envision what these scenarios might be and how they might work for their situation. Each new interview began with a description of the current work activities and organisational communication processes as developed in the previous interview. The interviewee was asked to evaluate this information. He or she was asked to discuss the correctness of the information to help the interviewer change incorrect information and to give additional information if needed. Then in a following interview the ASAS service summery was presented, followed by a presentation of future scenarios as developed in the previous interview. Again, the interviewee could discuss correctness and refinements. Sometimes a technical enquiry searching for ASAS product information from suppliers and or an Internet search for more information on the new ideas of interviewees was needed to determine how these ideas could be realized technically. That is, matching the interviewee 49

50 Application Sharing Services: Design and Performance technology wishes with the current available ASAS technological points of attentions found in suppliers information on paper, or on the Internet. Subjects The first interview was conducted with the key player of the project, i.e. a member of the management team of the organisation to obtain a list of which stakeholders should be interviewed. This list was elaborated using an analysis of the current communication processes in the organisation, developed in the first phase of every interview. The stakeholders that were interviewed were those that were expected to be confronted with work process changes if an ASAS service was introduced. A balance choice was made to obtain a range of different stakeholders: from management to executives and stakeholders who viewed different aspects of ASAS service introduction. Analysing stakeholders opinions about current and future context of use There were two reasons to analyse stakeholders opinions about ASAS service performance. One, the optimal ASAS service characteristics for the organisation had to be distinguished. That is, the circumstances under which the ASAS service would perform in a most optimal manner. Two, the analysis had to deliver stakeholders opinions regarding how they thought an ASAS service would influence performance in their situation compared to the current situation. The analysis of the interviews had to cover the whole range of stakeholder comments concerning factors determining and affecting ASAS service use. Miles and Huberman (1994) argued that this analysis has to cover the essence of what has been said. It has to cover the greatest common denominator, which ignores individual opinions, and enhances multiple comments to be part of a rich context-of use description. Miles and Hubermann (1994) and Yin (1994) discussed how this analysis can be supported by making a matrix of categories and placing the data within categories. The database was split into two parts. One part had to cover the characteristics of the different features of the ASAS service and a second part had to cover their contribution towards efficient, effective and satisfactory ASAS service use, and how communication processes will change. Moreover, the database has to chart with which characteristic features the ASAS service will give optimal efficiency, effectiveness and satisfaction. A matrix of categories was developed in which the different factor characteristics were denoted i.e. technology, tasks, users, and organisational characteristics. and the propositions describing a relation based on efficiency, effectiveness, and satisfaction and communication processes. These categories and propositions were elicited from the literature (see chapter 2). De Vreede (2001) applied a similar method to analyze effective group facilitation using group decision support systems. He describes this method as follows: This approach consists of the following steps: stakeholders are interviewed regarding the phenomenon under investigation. From the transcribed interview, statements are extracted and grouped into clusters. The clusters may form a sequence of categories, which are finally described in English to provide for a rich description of the phenomenon. 50

51 Chapter 3 Method: ASAS service design and performance In this study an simular method of analysis was used. All stakeholder interviews and the group interviews were recorded on audiotape and a person who did not attend the interviews and workshops transcribed these literally. Afterwards all comments were labelled. The labels were refined after two interviews. That is, some labels were too general and could be worked out more specifically. Sometimes it appeared that remarks positioned under one label could be divided into two distinct groups. For instance: technology could be split into technology costs and technology usability. The labels were based on two types of information (see table one): one, the current and future communication processes and two expectations about changes in efficiency, effectiveness and satisfaction of ASAS service use. The comments sometimes consist of a few words or a whole paragraph. All comments with the same labels were clustered in the same category using an Access database. To increase the reliability of this clustering, some parts of the transcribed data were analyzed by a second person. Differences between the coded data were discussed and category descriptions were reformulated. 51

52 Application Sharing Services: Design and Performance Table 3-2 The two types of information in which the comments are clustered Type of information 1 A - ASAS technology features: technological requirements and wishes Ab - Security related issues and ASAS technology Ax - Complexity of ASAS technology Ap - Costs of ASAS technology T - Task characteristics: comments about which tasks can be used E - Environmental characteristics Wo How do the current communication processes without ASAS services - are envisioned Wn - How can the new communication processes - with ASAS services -be envisioned Wv - What is the question, which services and communication processes Wc - Changes in the communication processes of the client Wd - Changes in the communication processes of the official O - Foundation why ASAS services should be used S - In which use situation the ASAS service can be helpful Type of information 2 QFi - Quotes about individual efficiency; faster answers due to ASAS Services QFo - Quotes about organisational efficiency, faster work of the organisation due to ASAS services Fpi - Problems related to changes of individual effectiveness due to ASAS services. Fsi - Solutions related to changes of Individual effectiveness due to ASAS services Fpo - Problems related to changes in organisational effectiveness due to ASA services Fso - Solutions related to changes in organisational Effectiveness due to ASAS service Si - Individual satisfaction due to ASAS services Looking at the clustering it can easily be observed which, and how, the ASAS service characteristics affects efficiency, effectiveness, satisfaction and communication processes. Detailed written scenarios were worked out using this information. Questionnaire In order to test hypothesis 4 interviewees were asked to judge stakeholders opinions about ASAS services efficiency, effectiveness and perceived usefulness. The questionnaire was administered in two phases of the design process: once in EXSE step 4 and once in EXSE step 7, with the aim to discover how stakeholders opinions changed throughout the design process. 52

53 Chapter 3 Method: ASAS service design and performance Subjects Relevant stakeholders were interviewed for the scenario development (see chapter 4). In the support for VICWs case (visually impaired computer workers) 9 stakeholders were selected. In the support for taxpayers case 15 Dutch Inland Revenue stakeholders were selected from different sections of the Revenue Service (see chapter 4). In the support for VICWs case 24 stakeholders participated in the evaluation workshop and in the support for taxpayers case 10 stakeholders participated. More stakeholders were involved in the casestudy 1 evaluation workshop, because in this casestudy the evaluation was more important than for casestudy 2. Casestudy 2 focused more on the ASAS service scenario development than on its evaluation (see, for more details, chapters 4 and 6). In the support for VICWs case 7 stakeholders filled in the pre-test questionnaire and 15 the post-test questionnaire, of which 5 stakeholders were the same in the pre- and post test space. In the support for taxpayers case 5 stakeholders filled in the pre-test questionnaire and 6 the post-test questionnaire, of which 5 stakeholders were the same in the pre- and post test. Procedure In both casestudies the scenario-based evaluation was performed in a workshop. In these workshops the concrete storyline and the scenarios were presented first. Then some video fragments of the experiment were presented together with the experiment results. Then contextual factors remarks concerning performance extracted in phases 2 and 3 were evaluated. The factors were the preferred ASAS service features, problem (task) features, user features (client and official), environmental features and introduction features. Their expected effects on efficiency, effectiveness, satisfaction and communication processes were discussed. Employers, managers, developers and other relevant stakeholders participated in this workshop. Hypotheses Hypotheses 1, 2 and 3 were judged on their realistic expectations and put in a broader context in the interviews and workshops. These hypotheses concerned the efficiency, effectiveness and satisfaction improvement due to using ASAS services in comparison to traditional ways of consulting. This improvement will be greater for complex tasks and users working with constraints. Hypothesis 4 was tested: that stakeholders will evaluate the service performance effects more positively at the end of the service design. Technology and setting The technology and setting of the single interviews are discussed in EXSE phase 3. The workshops were videotaped. In the workshop all stakeholders sat in a circle. The same Access database as described in EXSE phase 3 was used to cluster stakeholders remarks. Observations and measurements The observations and measurements of the single interviews are discussed in EXSE phase 3. An equal method was used for the workshops. 53

54 Application Sharing Services: Design and Performance The questionnaire was administered in two phases of the design process: one in EXSE phase 4 and one in EXSE phase 7, with the aim of testing hypothesis 4: how stakeholders opinions changed throughout the design process. The questionnaire consisted of three parts. The 9 questions were towards: 1. Efficiency of the ASAS service. Efficiency concerns the actual time needed for training and support, cost aspects, and man power (officials) needed to give training and support. 2. Effectiveness of the ASAS service. Effectiveness concerns the quality of training and support, quality of problems handling and subjects ability to solve repetitive problems themselves later on. 3. Usefulness of the ASAS service. Usefulness concerns the ability to give more accessible, better, more accurate, faster and easier training and support. The questions were answered on a 5 points scale (1 = very bad/not agree, 5 = very good/agree). The same procedure for both measurements was followed: The different answers around one topic, efficiency, effectiveness or perceived usefulness, were counted. This delivers a value for every stakeholder. Then, all stakeholders values were aggregated. The reliability of the three scales was good, because the Crombach s alpha was > The reliability of the efficiency scale was 0.885, the reliability of effectiveness scale was 0.883, and the reliability of the usefulness scale was The Crombach s alphas were calculated using the collected data of both casestudies. Analysis The analysis of the single interviews is discussed in EXSE phase 3. An equal method was used for the workshops. The questionnaire was analyzed using a T test for unequal variances, due to the different amount of stakeholders. (see chapter 6). In both the support for VICWs case and the support for taxpayers case 5 stakeholders were equal in the pre- and post test. Because most stakeholders filled in the questionnaires anonymously, no special analysis could be performed on this subject group Individual performance In order to test hypotheses 1, 2, 3 and 5 the following steps were taken. First the future context of use developed in EXSE phase 4 was used to develop the experimental design, according EXSE phase 5. Next, the experiment was performed as described in EXSE phase 6. The detailed experiment design in the two casestudies (EXSE phase 5) and how the experiment was performed (EXSE phase 5) will be discussed below. Experiment design The start of the experiment design in both casestudies consisted of a discussion with the key player of the project, a management team member of the organisation, regarding which scenario or combinations of scenarios could be used as input for the experiment. The 54

55 Chapter 3 Method: ASAS service design and performance technology, tasks, users and organisational settings of the experiment were determined on the basis of this scenario. Technology The ASAS service was simulated using NetMeeting (see chapter 5) in both casestudies. NetMeeting is a standard installation on Windows 2000 PCs and higher version numbers and it is freely available. Furthermore, no problems were expected with running NetMeeting together with the software adaptation aids (casestudy 1) or the digital tax return software (casestudy 2). See for more details chapter 5. The telephone connection needed to simulate the ASAS service was realized differently in both casestudies. In casestudy 1 the internal telephone switchboard was used. In casestudy 2 this was not possible, therefore two microphones and boxes were used to simulate the telephone connection. Problems In casestudy 1 no special problem-cases were developed for the experiment (see chapter 5). It was decided that the subjects should perform 2 hours of their normal computer training via the ASAS service, instead of performing it in a classroom. The problems being solved were the problems subjects met during their course. These were softwarehardware- and physical environment related problems (see chapter 5). In casestudy 2 problem cases were related towards taxpayers frequently asked questions about tax returns. These questions were chosen with the help of tax officials and they were further refined during a pre-test (see chapter 5). Subjects Two subject types can be distinguished for both casestudies: one, the subject with the problem (client) and two, the subject who solves the problem (official). In casestudy 1, the clients were visually impaired computer rehabilitants. The official was an official employee of the rehabilitation center. In casestudy 2, the clients were a sample taken of the tax paying population. The officials were official helpdesk employees (tax officials) of the tax agency. In casestudy 1 three subjects participated and the experiment results were analyzed using qualitative analysis. Two reasons can be given for why this experiment was performed with only 3 subjects. One, the experimental results were expected to obtain information about the individual performance of a visually impaired person. According to the officials at the rehabilitation center and the members of the management team every VICW will have different possibilities, different questions and different handicaps, which cannot be generalized. Two, this experiment was aimed at envisioning a future scenario. More observations were not necessary to obtain this goal. In the taxpayers casestudy eighty subjects participated. This number was required because the results of the experiment were designed to serve as a basis for helping the stakeholders to envision a future scenario, and they were the basis for a systematic analysis of (expected) future performance of tax payers. 55

56 Application Sharing Services: Design and Performance Because casestudy 1 was a field experiment and exploratory in nature, only 3 VICWs participated as volunteers. An experienced computer official from the centre acted as the problemsolver for the 3 subjects. Casestudy 2 was a controlled experiment, 72 subjects were tested in the experiment proper and 3 participated in the pilot. Nineteen experienced tax officials also participated in the experiment. Tasks Due to the exploratory nature of the field study performed in casestudy 1, no problems (tasks) were pre-defined. The problem types in casestudy 2 consisted of three types: diagnostic problems, navigation problems and information problems. Procedure In casestudy 1, subjects performed 2 hours of their normal computer training via ASAS technology. In other words, the ASAS service conditions and problem types were not systematically manipulated. In casestudy 2, two variables were manipulated. The ASAS service conditions were three consultancy conditions: intervention based ASAS service, observation based ASAS service and plain telephone. The problem types were also of three types: diagnostic problems, navigation problems and information problems (see chapter 5 for more details). The combination of these two conditions implied that nine types of stimuli were given. Questionnaires were used to measure subjects and officials opinions about ASAS use after every new technology condition. Hypotheses In both cases hypotheses 1, 2, 3 and 5 were tested (see chapter 2). Technology and experimental setting In casestudy 1 the experiment was performed at the rehabilitation centre. In casestudy 2 the experiment was performed at the Work and Interaction Technology laboratory at the Delft University of Technology (the Netherlands). In both casestudies three rooms were used. The clients, officials and observers had places in separate rooms, from which they could not see each other. All subjects had their own personal computer. These computers were connected via a network with the help of Microsoft NetMeeting (10 Mbps). Two video cameras were used to film the faces of the clients and officials. A screen-cam, taped the screen of the client. The images and sounds recorded by the video cameras and the taped client screen recorded by the screen-cam were transmitted to an observatory room by respectively the network and video connection. Observations and measurements The analysis of both casestudies was based upon digitally mixed data. The mix contained the voices and camera images of the faces of client and official, and the information of the screen-ca. This allowed direct observation of how the subjects reacted in a certain situation, because a direct link was made between spoken words, errors observed at the screen and how subjects reacted visually in a specific situation. 56

57 Chapter 3 Method: ASAS service design and performance Four dependent variables were measured: efficiency, effectiveness, satisfaction and communication processes. The dependent variables within the experiment setting were as follows: Efficiency is defined as the time needed to receive and understand a valid and satisfactory answer to problems posed by the subjects. During the experiment performed in casestudies 1 and 2, an observer who was sitting in the observatory room, noted the start and end of the handling of a problem. The start was the moment the client said his or her first Hello. The end of the problem was defined as the last word during the session. Most of the time these were remarks such as Yes, I understand. and Thanks for answering my questions. Effectiveness was defined as the number of correctly solved problems per client. The observer also noted this down during the experiment. For casestudy 1 a correctly solved problem was deemed to be so when the client was satisfied with the answer. For casestudy 2 the correct solution was determined during the experiment preparation. Tax officials indicated the correct answers. Satisfaction was defined as the subjects confidence in the ASAS service in comparison to the traditional service via telephone. Interaction intensity and interaction focus were defined as the observed changes in the number of questions and time spent by clients and tax officials to deal with certain information categories. These categories are listed below. Tax related information: communication about tax matters, such as explanations about taxes. Diskette related information; information on how the diskette should be manipulated, for instance how to fill in certain information. Situation awareness: communication about the location of some needed information. Informal communication: communication about personal matters, for instance telling something personally or saying Hello or Goodbye. This information was not tax related, diskette related or about situation awareness. The amount of turn taking was noted. This was measured by counting the switches in communication: the switch from official to subject and vice versa (see for more details chapter 5). Analysis Efficiency was measured using observation and video recordings. Efficiency was defined as the time needed to solve a problem. It is the time between when the subject picks up the telephone to phone the official and the moment the connection is closed. Effectiveness was measured by denoting if a problem was solved correctly. The official defined a correct answer, note in casestudy 1 it was decided after the experiment, in casestudy 2 it was determined before the experiment. Satisfaction was measured differently in both casestudies. In casestudy 1 it was not measured using a questionnaire, but by noting the spontaneous remarks. These remarks 57

58 Application Sharing Services: Design and Performance were obtained from the videotapes. In casestudy 2 satisfaction was measured using a closed questionnaire (see for more details chapter 5 and appendix 2). After the observations in both casestudies, all information registered during the observations was collected and stored in an Access database. Due to the exploratory nature of casestudy 1, the analysis can not be supported by statistical evidence. In casestudy 2 the data was analysed statistically using SPSSx. The statistical analysis of casestudy 2 was carried out as follows. Multi-variate analysis of variance was used for efficiency and satisfaction measures and measures on interaction processes. This was because multivariate analysis gives more robust results compared to an ANOVA when data are not distributed perfectly normally and measures are not completely independent i.e. due to learning effects and fatigue. Dependencies between measurements cannot be excluded completely, although conditions were assigned randomly as were the instances of the different task types (the individual problems) for each session and the method of analysis controlled for systematic effects due to the official; every official answered the questions of 3 tax-payers and 1 taxpayer worked on 9 problem-cases. 3.4 Discussion and conclusions It has been shown in this chapter that the research methodology was divided into two separate parts: the methodology required to design the ASAS service and the methodology required to test ASAS performance after a service has been designed. These two different research methodologies required two different research methods. The research methodologies were presented in this chapter. However, their detailed applications in the casestudies were not described. This is presented in the following chapters. The methodology used for ASAS service design will be explained in chapters 4 and 6; the methodology used for ASAS service performance testing will be explained in chapter 5. With regard to ASAS service design, the Early experiment and Scenario-based Evaluation (EXSE) methodology was used. The power of EXSE lies in the combination of casestudy research and experiments. Casestudy research delivers rich, qualitative data about scenarios for future ASAS services, through making an inventory of stakeholders opinions. The experiment should provide strong visualisations of future ASAS scenarios and show possible performance effects. These visualisations were used to refine the scenarios. In this dissertation EXSE was used to design two different ASAS services: to support Dutch Visually Impaired Computer Workers and to support Dutch taxpayers. The focus of this dissertation is the testing of ASAS service performance. Performance will be tested on the basis of the scenarios developed during the ASAS service design. The main focus is testing performance at the individual, task level. Four of the five hypotheses formulated in this dissertation are related to these individual performance effects. However, it has been argued that ASAS service performance can not fully be related to individual performance: observing at the organisational level will deliver additional information that is necessary to predict ASAS service performance in organisations. 58

59 Chapter 3 Method: ASAS service design and performance Some critical remarks have to be made with regard to performance testing. Firstly the involvement of the researcher within the design processes can be criticized. The researcher developed a draft of current and future interaction processes using information collected in interviews. Although the stakeholders were asked to refine and correct this information, the way in which this information was presented to them possibly influenced the way in which the scenarios were developed. A researcher with another perspective may have helped to develop the scenarios in another way. A second critical remark concerns the choice of stakeholders and their availability. First, not all of the stakeholders who participated in the pre-test could join the post-test. Furthermore, a lot of stakeholders filled in the questionnaires anonimously. This made the comparison between the pre- and post-test difficult. Of course the data was corrected for the different numbers of stakeholders in the pre- and post-test using a T-test with unequal variances, but these results are less robust than the results found with equal and known stakeholders. So limited significant results could be drawn with regard to hypothesis 4. The third critical remark concerns the amount of interviews. First, the stakeholders were judged upon information given by the key player of the project during the first interview. The amount of stakeholders and their expertise was mainly based on the information obtained in this first interview. However, the results are probably influenced by this stakeholder choice. Second, new interviews were not conducted if all, according to the key player, important stakeholders were interviewed and no new information was elicited for scenario design in the next interview. However, this could be a local maximum: in advance it cannot be concluded that all really relevant information is collected. Probably more interviews would deliver new information for the scenario. The fourth critical remark concerns the collection of stakeholder remarks and how these remarks should be interpreted. It is difficult to find a good method for comparing different opinions. Sometimes, unimportant stakeholders repeatedly make an unimportant remark. Sometimes, however, a very important remark is made only once by a very important stakeholder, and this remark is supported by the whole stakeholders group. However, this support is not made explicitly clear with spoken remarks, but only by nodding or other visual cues. In this case, a qualitative measurement of stakeholders opinions fails. This is the reason why the numbers of a certain remark fail in chapter 3 and 6. A fifth critical remark concerns the collection of the first opinions. All stakeholders were asked to fill in the questionnaire after the first interview. These interviews, however, were not carried out synchronously and the first interviews were based upon less detailed data than the last interviews, because every interview was based on information collected in the previous interviews. This could lead to misrepresentation of the collected opinions. Note that the post test, the last collection of opinions was for all stakeholders at the same time. The sixth critical remark has to do with the equal questionnaire that was used for the preand post-test. This could have influenced the way in which stakeholders filled in the questionnaire. Some large differences were observed between the two experiment based evaluations. This influenced the way in which the results can be interpreted and generalized. The experiment based evaluation for the support for Dutch Visually Impaired Computer workers casestudy was an exploratory study. Only 3 VICWs were observed and 1 official. Furthermore the 59

60 Application Sharing Services: Design and Performance conditions were not systematically manipulated: face-to-face support was not systematically compared with support via ASAS services, nor were problems manipulated with respect to different complexity. This makes it impossible to generalize the results to other VICWs. From this experiment one can not predict whether an ASAS service really increase VICWs performance. The support for Dutch tax payers experiment based evaluation results could be generalized. Due to the controlled conditions and the large number of taxpayers and officials, participating in the experiment predictions could be presented about performance changes in relation to the use of ASAS services. Generalizations could be made with regard to different technology conditions (telephone, ASAS services conditions), task types (problems) and user groups, and the experiment allowed us to determine in which situation(s) ASAS services will influence performance optimally. Some critical remarks are necessary with respect to the experiment based evaluation of the support for Dutch tax payers casestudy. The first remark concerns the predefined problems used in the experiment. These predefined problems created an unnatural situation. It is doubtful if taxpayers reacted in the same way to these predefined problems as they react in a natural situation, for instance if emotions are influencing their behaviour. The second remark is related to the usability of the ASAS service. In reality, contact will start differently: not via microphones and an up and running ASAS connection. The actual way of establishing the ASAS service connection will influence ASAS service performance. The third remark concerns the choice of technology conditions, in the experiment, the choice to use ASAS services or telephone was predefined. In a natural situation, the choice will be free. This will possibly change performance. 60

61 Chapter 4: Scenario s for future ASAS based services 4 Scenario s for future ASAS based services A report of the core of the design approach is given in this chapter, this includes a description of the main aspects of the current services and the adjustments that need to be made to the main aspects of the concept to integrate the ASAS service concept in future settings. The future settings are visualized using scenarios. These scenarios are possible design alternatives. They give the directions as to how future settings can be envisioned. This allows us to provide an answer for the second part of sub question A1: What are the existing ASAS service concepts and in which contexts can these services be used? 4.1 Method The ideas for the new design were gathered using a series of interviews, in which data was collected on aspects of current services and aspects that interviewees would like to see in future ASAS based services. The following issues were covered in the interviews. Organisational setting of the organisation under consideration and how client problems were solved at the time of the interview. The interviews covered the following topics: o The interviewees preferred ASAS technology features. o A discussion of the kind of problem(s) that might be solved using ASAS services. o A discussion of possible changes in organisational roles due to implementing ASAS services. Scenarios for ASAS based services were then developed based on the data gathered during the interviews Procedure The main aspects of present and potential future work processes were determined using an iterative process during a series of interviews (see chapter 3). At the end of the first interview a first draft of the current and predicted future service(s) was made and extended or withdrawn with on the basis of internal reports and a technology analysis. These service descriptions were submitted to the next interviewee who in turn could extend it, or give his or her objections to the descriptions. Once all the interviews had been held, the final scenario was evaluated in a workshop setting. The first interview was conducted with a key informant of the project, i.e. a member of the management team of the organisation. This interview was used to identify the relevant stakeholders and these stakeholders were then interviewed. This list was elaborated using an analysis of the current workflow which was conducted for every interview (see below). A balance was made between different stakeholders who came from various hierarchical levels and parts of the work processes. In the support for VICW (visually impaired computer workers) casestudy nine stakeholders were selected. Six of them worked at the national 61

62 Application Sharing Services: Design and Performance rehabilitation centre: two members of the management team, two computer trainers and two policy advisors. Further two badly sighted computer rehabilitants of the center were interviewed and one blind computer user who was already working in a regular job. In the support for taxpayers casestudy fifteen stakeholders drawn from the Dutch Inland Revenue were selected from different sections of the Revenue Service (see section The Dutch Inland Revenue Services ). Five interviews were conducted at the Belastingdienst Centrum voor Proces en Productontwikkeling, BCPP 1. These interviews were conducted with a lawyer, a technician, a designer and two managers of product departments of the Revenue Service, one interview with a manager of a district office, two managers of the BelTel center- a national call center dealing with tax return problems of individual taxpayers - and two executives (tax officials). The main aspects of the scenarios were evaluated and refined in a final workshop in both casestudies. The scenarios were presented on paper, while video fragments of the ASAS service experiment (see chapter 5) were demonstrated. The participants were asked to refine the scenarios on a basis of the topics presented in chapter 2. Twenty-four stakeholders participated in the final workshop of the VICW casestudy. Twelve of them worked at the national rehabilitation centre: one member of the management team, four computer trainers, three policy advisors and four technology experts. Further there were two badly sighted computer rehabilitants from the centre, five persons drawn from special interest groups of VICWs, two trainers from Sonneheerdt (a rehabilitation centre for VICWs supporting vocational training) and two from a regional centre, and one stakeholder from the sponsoring organisation SENTER. Due to the complexity of the scenarios in the taxpayers casestudy the specific client problems had to be further specified as part of the preparation for the experiment, the main aspects of the experiment also had to be determined in more detail. Three extra interviews were used to determine client problems, which were expected to be of use in the new ASAS service. Two interviews were performed at BelTel locations in Utrecht (the Netherlands), and one at the BelTel location in Zaandam (the Netherlands). A short explanation was given about possible ASAS service implementations in the future in every interview. Then interviewees were asked to state what client problems they expected using the service might help them to solve. The answers were combined to form a micro scenario. These scenarios were refined and corrected during a pre-test in which five different taxpayers and two different tax officials participated. Ten stakeholders participated in the final workshop of the Taxpayers casestudy. Six members from the BCPP centre were present: a lawyer, a technician, a designer and three managers of product departments of the Revenue Service. One manager from a district office participated, one manager from the BelTel centre and two managers from the B/SEB The interviews Every interview consisted of a number of different steps. In the first step the interviewee had to describe how current support was given. A first draft of this support was developed using internal reports. The description was refined, elaborated and corrected in every new interview. In the second part of each interview, the interviewees were told about the ASAS 1 Center of the Revenue for Process and Product development 62

63 Chapter 4: Scenario s for future ASAS based services service concept and the results of the technology analysis (see chapter 3). Then, the interviewees were asked what problems they had with their present form of support and how they felt the ASAS service could be used to overcome these problems. This allowed me to identify potential future services, which were then worked out in scenarios. Each interview was part of an iterative process and the results of an interview were refined and corrected for every interview. 4.2 Casestudy one: Support for Dutch Visually Impaired Computer Workers I will discuss the development of support for Dutch VICWs 2 in this section. A VICW is defined in this study as a visually impaired computer worker between the ages of 18 and 65. They are visually disabled to such an extent that they need special computer aids or special magnifiers to read the screen of a computer. Computer support for VICWs in the Netherlands is coordinated as follows. If a VICW wants to use a computer they can take three different courses (see figure 4-1). First, they can contact a rehabilitation centre. If they want to use a computer for professional purposes, they have to contact Stichting Het Loo Erf, the national rehabilitation centre for visually impaired persons. They will be asked to visit the centre for (a part of) a day during which tests will be conducted to determine the right computer aids and training for their needs and visual disability. The centre will assist the VICW with writing letters to possible sponsors that might be willing to pay for the computer aid (most of the time an insurance company), and will discuss with the VICW the possibility of following a computer course at the centre using the types of aids best suited to their needs. These courses are given intramurally: the VICWs stay some days at the centre and receive lessons in small groups during which they learn to work with the special aids, and if needed, with some applications. Stichting Het Loo Erf can help VICWs to return to work, or to find a new job. This last possibility is outsourced to Kliq, an organisation that deals with placing and reintegrating people with handicaps. If VICWs have a more general problem with working with a computer, they can follow standard vocational training at Sonneheerdt. The lessons are given in small classes. If a VICW wants to use the computer for private purposes, they have to phone the regional centre in their neighbourhood. As at the national rehabilitation centre they will undergo tests, and receive assistance with obtaining funding for their special aids. They may also take a computer course at the centre using the aids deemed suitable for their needs. These courses are given externally: VICWs travel once or twice times a week to the centre. VICWs can also contact a supplier of computer aids directly, this is the second path a VICW can take to obtain computer aids. In this case a VICW has to know which adaptation they need and has to write the requests for sponsorship letters themselves. The supplier can deliver the computer and the aids together with individual training at the VICWs home. Thirdly, VICWs can contact VICW special interest groups, the members of these groups can give advice concerning the availability and appropriateness of such aids and on how to write 2 These steps are not chronological in time: every interview starts reiteratively with step one. 63

64 Application Sharing Services: Design and Performance requests for sponsorship. Bulletin boards and the telephone are used to support mutual help. Figure 4-1 Support for Dutch VICWs Based on the results of the interviews described above I will now discuss, briefly, the types of preferred ASAS technology Summary of preferred ASAS technology features The technology analysis (see for more detail chapter 2) revealed that VICWs participating in the rehabilitation casestudy would profit most by an ASAS service consisting of a combination of Microsoft NetMeeting and the telephone. The Internet should be used for sending screen information, collected by Microsoft NetMeeting (sharing applications), and for sending the audio signal from a speech synthesizer (if the VICW uses one). A normal telephone line should be used for contact between the VICW and his or her advisor, i.e. the person working at one of the care organisations. ISDN, ADSL cable or a mobile telephone in combination with a plain telephone would be needed, because two contact lines are required for sending data and speech. A personal computer with Windows 2000 or higher would be required for safe, reliable and fast data traffic. The problems VICWs had in the present setting were discussed, after discussing this ASAS technology concept and its generic functions with the interviewees. Interviewees were asked how the service could be used to overcome these problems. This resulted in the 64

65 Chapter 4: Scenario s for future ASAS based services identification of two potential future services: rehabilitation at a distance and a national helpdesk. A scenario was developed for both services (see next sections) Services and Additional Scenarios for VICWs Support with ASAS Services Service one: Rehabilitation at a distance This service has to deal with the first problem VICWs meet when working with computer applications in combination with their visual aids. After VICWs have learned to work with the new aids they may start to learn a specific computer application. When learning a new application, VICWs will constantly be hindered by their handicap: they always 'see' only a small portion of the screen at any one time. In Braille, they see a half to a full line; when using a speech synthesizer, they hear all the information in succession, and with a magnification package they see only a small part of the screen. This means that VICWs have little or no overview of the screen, so that certain actions are less self-evident than they would be for a well-sighted computer user (Beumer, Jameson, & Waterham, 2000). These problems are increasing due to the rapid growth of new applications and adaptations, and the continuously changing work processes in which many computer workers in organisations also VICWs - are involved. These changing processes include steps in the career of the VICW, changes in the operating systems of a computer, software changes or a new (version) of a computer aid. Sometimes complete training in a new program or aid is needed; sometimes a single problem has to be solved. Throughout the interviews it appeared that ASAS services could be used to support VICWs during their whole (working) career (see figure 4-2). In this case, rehabilitation centers will have to develop a special ASAS service, where VICWs can ask for distance training in a new application or new aid, or obtain help with solving problems. Imagine for instance a complex or big document. VICWs sometimes have difficulties with understanding the structure of such a document, due to their lack of an overview. A second type of problem may occur during the log on procedure. Most computer aids start to work under Windows, and do not facilitate actions before Windows is up and running. Interviewees suggested that rehabilitation centers could support a view in this situation by looking over their shoulder briefly to give instructions. According to the interviews, rehabilitation support at a distance needs to be developed by different organisations. Stichting Het Loo Erf is the most appropriate party to support computer rehabilitation related to a working situation. However, regional rehabilitation centres are the most appropriate party to assist with computer training for private situations use. Interviewees mentioned that computer training may be strongly influenced by the way in which (new) computer aids work with the (work specific) software. In their view the software suppliers of the computer aids would be the most appropriate parties to assist them with training in these cases. 65

66 Application Sharing Services: Design and Performance Figure 4-2 Rehabilitation at a Distance Scenario 1: Rehabilitation on Distance The first elements of a scenario for the solution of the rehabilitation-at-a-distance were developed during the interviews. A first draft was made of some scenarios in the first interview, and in the following interviews these scenarios were elaborated further (see figure 4-3). Scenario developed in the first interview Jan Smit is 45 years old and worked as staff member at a service desk. One year ago he went blind. Jan went to Stichting Het Loo Erf where he was given training in personal care, was taught Braille and received computer training where he learned to work with Microsoft Windows and NetMeeting in combination with Infofox, a speech module. Jan can continue with his rehabilitation at distance. The courses are on a cd-rom which he can follow aurally. For problems he can use the ASAS service: the trainer at Stichting Het Loo Erf will answer his questions. 66