A user-driven workplace software certification process

Transcription

1 A user-driven workplace software certification process ÅKE WALLDIUS TORBJÖRN LIND YNGVE SUNDBLAD BENGT SANDBLAD LARS BENGTSSON Human computer interaction (HCI) Computer science and communication (CSC) Royal institute of technology (KTH)

2 HCI-58, TRITA-CSC-MDI 2007:1 ISSN , ISRN-KTH/CSC/MDI--07/1--SE {akew tojje yngve bengtsand Human computer interaction (HCI) Computer science and communication (CSC) Royal institute of technology (KTH) S Stockholm, Sweden URL:

3 USER CERTIFIED 2006 A USER-DRIVEN WORKPLACE SOFTWARE CERTIFICATION PROCESS Introduction 3 Aim of certification and methods used 4 What kind of software can be awarded the USER CERTIFIED label? 5 Who gets involved in the certification? 5 How is the certification process carried out? 6 How is the user satisfaction assessed? 6 Required levels of user satisfaction 7 To what does the certification apply? 7 What the certification entails 7 Quality criteria 9 Relation to international standards and other certification processes 10 Differences between UC 2002 and UC Future work 12 Acknowledgements 13 References 14

4 2

5 USER CERTIFIED 2006 A User-driven Workplace Software Certification Process Introduction This is a description of the USER CERTIFIED 2006 programme, a programme for userdriven certification of software used at workplaces. It updates the earlier report [17] which presents the first version of the certification programme, USER CERTIFIED The initiative to certify the quality of workplace software was taken by a team of ombudsmen and investigators Ove Ivarsen, Peter Forss, and Torbjörn Lind at LO (The Swedish Trade Union Confederation) in The method and institutional framework for software quality assurance were developed in cooperation with researchers in the ITQ project, Quality assurance of IT Support for the Developing Work. This project was coordinated by Yngve Sundblad, CID (Centre for User Oriented IT Design) at KTH, Stockholm and the other participants were Åke Walldius, CID; Bengt Sandblad, Jan Gulliksen, Iordanis Kavathatzopoulos, and Mats Löfberg, Human Computer Interaction department at Uppsala University; Lars Bengtsson and Martin Ljungström, Industrial Economy department at Gävle University College; and Torbjörn Ilar, Production technology department at Luleå Technical University. The methodology underlying the USER CERTIFIED 2006 programme (UC 2006) is based on experiences gained from a number of pilot projects performed in the ITQproject, described in Swedish in references [3, 16] and in English in [19]; from extensive surveys of users satisfaction with their IT-support in industry, healthcare, and banking [8, 9]; and from quality assessments done in connection with the yearly Users IT Prize Contests The TCO Certification programme of computer hardware (TCO'92, '95, '99, '03) is an important source of inspiration for the UsersAward programme [1]. After extensive work and use of a step-wise developed list of about 30 user quality criteria in a questionnaire, filled in and commented by hundreds of users attending yearly users conferences , the USER CERTIFIED programme was ready for use in two certifications of workplace software, performed in The deployment and use of the time management system Time Care (Time Care AB) was assessed at three workplaces (a department store in Stockholm, a police department in Katrineholm, and an emergency ward in Falun). Similarly, the deployment and use of the enterprise resource planning system Monitor (Monitor AB) was assessed at three manufacturing companies (Mekanotjänst AB in Järvsö, Stacke Hydraulik AB Skillingaryd, and Sjölanders Verkstäder AB in Västerås). These two certifications mark the transformation of the UsersAward/ITQ-project into a permanent certification programme based on systematic assessments of user satisfaction. The USER CERTIFIED certifications are conducted by the non-profit development company UsersAward AB, founded by LO in 2003 and jointly sponsored by LO and the Swedish Governmental Agency for Innovation (VINNOVA). A research panel, with 3

6 researchers from KTH (coordinators), Uppsala, and Gävle, is tied to UsersAward with the responsibility to do and report follow-up studies of the certifications and surveys performed and to further develop the underlying method for successively new versions of the programme. To this end, UsersAward also has cooperation agreements with KTH and with TCO (Swedish Confederation for Professional Employees), the initiator of the TCO Certification programme. The programme's firm link to publicly sponsored research means that its criteria and assessment procedures will be updated in new versions based on experiences gained from three to four years of assessment activities. These experiences will be presented to the public at conferences, in articles and on the UsersAward website and the feedback obtained from these presentations will help keep the programme informed on the latest developments in the software industry and on new ways of deploying and using software in different branches of industry, healthcare and services. Additional information can be found on the web sites: and in references [12, 19] and (in Swedish) [8, 13]. Aim of certification and methods used The aim of the certification is to give prospective buyers and users an understanding for the overall quality aspects of the software as it is used in different workplaces. Therefore, the assessment is not reported in the simple terms of passed or not passed. Instead, the relative strength of the user satisfaction with all the investigated quality aspects is reported and briefly commented by the certification team. This qualitative aspect of the assessment, with focus on the context of use and the inherently relational character of use quality, is in turn supported by the ongoing research and development effort which is a constituent part of the UsersAward programme. The user certification gives a thorough and unbiased report on what a representative sample of users from at least three workplaces (or more, for systems with few users) have experienced after at least one year of daily use. A certificate thus means that it is possible to install and use the software in such a way that the users are satisfied. However, the relational character of use quality makes it impossible to issue any guarantee that a user certified software will always be regarded as a success by its other users. The way the label USER CERTIFIED 2006 certifies that the software has satisfied users is expressed on the first page of the certification protocol: User Certified is a quality certificate for software issued by UsersAward. The assessment is carried out on several workplaces where the software users in interviews and questionnaires are asked to confirm that the software provider lives up to the claims made in the self declaration part of the certification. 4

7 The certification process consists of an assessment of information obtained through the following investigation and reporting methods: * a self declaration questionnaire to the software provider * a questionnaire on quality aspects, answered by at least 10% of the users in at least three workplaces, and at least 36 users altogether * an interview with three user representatives in each workplace based on the questions in the same questionnaire * a corresponding interview with two to three management representatives * a public certification protocol in which the user satisfaction is reported * a presentation of a certification report to the software provider. What kind of software can be awarded the USER CERTIFIED label? The set of criteria developed for the certification programme is designed to comprehensively measure the usability of software systems which are used by or involve a large number of employees at the worksites where it is deployed. A prerequisite for software providers who wish to apply for a certification, national or international, is that the system has been in daily use for at least one year in the workplaces where it is assessed. Since the process of deployment is a decisive factor for the overall level of usability at the individual worksite, the assessment not only includes technical and organisational aspects of the system but also the design and performance aspects of at least three individual deployment processes. Who gets involved in the certification? The assessment is always carried out in at least three different workplaces. When the assessed system has only few users at each workplace (e.g. financial systems), additional 5

8 workplaces are assessed to reach a minimum number of 36 participating users. The assessment of multiple workplaces is done in order to reduce the dependency of specific preconditions for deployment at the individual worksites. The software provider selects the stipulated number of workplaces where the assessment will be carried out. The user certification is based on the participation of representative end-user groups at the investigated workplaces. The group should comprise at least 10 percent of the workforce. The UsersAward organisation contact person at each workplace, in general a representative from the employee organisation, is responsible for making the panel representative for all end-users at their respective workplace. The questionnaire is answered anonymously on a paper or a web based form. At each workplace, three end-user representatives are interviewed based on a questionnaire, which is also filled in by all the participating users. As supplementary information the same questionnaire is used in an interview with two to three representatives from management at the workplaces (typically responsible for human resources, IT and communications, and economy). Supplementary information in the form of a self declaration of services provided is obtained from the software provider. How is the certification process carried out? Software providers who apply for certification of a software package are asked to fill out a self declaration regarding the package and its intended use. It consists of a form which is structured like the user questionnaire but that also include questions about what principles that guided the software developed, how it is deployed and supported at workplaces and how it is continuously enhanced according to feedback from users. In this declaration the provider selects the stipulated number of workplaces at which the actual use of the package can be assessed. After that, the evaluator team carries out interviews and questionnaires at the workplaces. At this point, the management at all worksites are also asked to read and comment the supplier selfdeclaration. Based on this information, the certification protocol is written by the UsersAward certification team in cooperation with the research panel which verifies the interpretation of the results. The last step of the certification process is the presentation of certification report to the software provider in which the assessment of user satisfaction is commented by the certification team. As a whole, the typical certification is carried out in less than two months. How is the user satisfaction assessed? In the interviews and questionnaire, each criterion of the defined set of 38 quality criteria is presented in the form of a statement to be confirmed on a Likert scale between 1 (total dismissal) and 6 (total agreement). These statements are preceded by a series of questions 6

9 on the respondent s personal data and his/her use of the software (age, gender, profession, modules used etc.), and followed by a series of statements regarding the overall software environment at the workplace. The assessment is done at the workplaces selected by the software provider out of which at least two thirds of the workplaces, and at least 50% of all users, have to meet the stipulated level of user satisfaction (se below). At each workplace, three end-users are interviewed and asked to 1) confirm the statements in the questionnaire with a value between 1 and 6 and, 2) give a short motivation to their level of agreement. Then three representatives from management are interviewed in the same way. (All interviews are sound-recorded.) These interviews are then complemented by a questionnaire to a panel made up of ten percent (or at least ten users) out of the end-users at the workplace. Required levels of user satisfaction The users are considered satisfied as a whole when the following two minimum conditions are met: - on at least two thirds of the workplaces the users have to agree to the level 4.0 or more on at least 67 percent of the statements - at least 50% of all users have to agree to the level of 4.0 or more on at least 67 percent of the statements. To what does the certification apply? - The certification only applies to the deployment and use of the software version specified by the provider and assessed by UsersAward. Hence, the certificate is not applicable to earlier or later versions of the software. - The certificate is only valid for the modules specified by the provider. More extensive systems with a large number of modules require multiple certification processes. - Certifications can apply to specific user categories. When the system typically is used only by specific professions with well defined tasks and activities the assessment is done with those users. - The certification applies only to the country/region in which the software deployment and usage has been assessed. - The certification is valid for three years. After that it can be renewed through a process which involves three or more workplaces (at least 36 participating end-users) of which one may be the same as in the original assessment. What the certification entails UsersAward is committed to publish information about certified software packages: 7

10 - at the website - at Swedish and international trade fairs and research conferences - in consultations for consumer, employee and management organisations. The software provider is entitled to use the UsersAward logotype in advertisements that apply to the certified software packages. 8

11 Quality criteria The project has identified the following six fields of relevance in which the user satisfaction is measured. 1. Total benefit (5 criteria): The benefit of the deployed IT support is satisfactory in terms of its support for quality of the organisations overall operations, productivity, competence development, for customers/clients, for the organisations fulfilment of its overall goals. 2. Deployment process (4 criteria): The IT support was deployed based on a clear and thoroughly discussed idea, the users ideas were taken into account in the process, their ideas regarding the new work tasks and processes were also taken into account, the users were given adequate education and training. 3. Technical design (13 criteria): overview of functionality, learnability, navigability, help functions, undo/history log, appropriate functionality, adaptability, integrity of customers/clients, integrity of users, interaction with other IT-support, overall stability, overall accessibility, response times. 4. Support for work tasks (8 criteria): overview of work tasks, supportive in getting job done, engaging to use, flexible to use, helps avoid work incidences, helps avoid stressful work, helps making follow-ups of work results, helps learning at work. 5. Support for communication and cooperation (5 criteria): supports the degree of selfgovernance of the individual s work, supports collaboration in the work group, with other work groups, with customers/clients, with external suppliers/partners. 6. Feedback and ongoing development (3 criteria): continuity of user requirement followups, visibility of follow-up results, adequacy of system support. The development of a set of quality criteria and a formal process for their use in certifications was the prime goal of the six pilot studies of the ITQ-project and participation in LO:s user community activities. Thus, the criteria have been developed and tested iteratively in pilot studies, in discussions within the projects User panel, in four major national questionnaires, in some 60 visits, typically eight per year, to nominees of the yearly IT Prize Contests and in discussions with the jury for that contest. The quality aspects found relevant to assess in the USER CERTIFIED programme apply to both the software as artefact and the method in which the software has been deployed in the client organisation. Hence, the aim of the set of criteria is that it should embrace both tangible characteristics of the software package and its associated documentation (the software provider s declared design principles, principles for deployment, functionality, cost, documented efficiency measures etc) and end-user experience of how well the software provider has managed to live up to the declared levels of performance. 9

12 Relation to international standards and other certification processes The USER CERTIFIED programme has common features with many existing standards and assessment programmes. The TCO labelling programme [1, 12] and the Swedish Software Industry SPI2000 programme [2] build on tangible characteristics of the investigated hardware and software while the Swedish and the American Customer Satisfaction Index, NKI [10, 11] and ACSI [14], build on experience expressed by representative samples of customers/users. The USER CERTIFIED set of criteria is based on quality criteria derived from the product specific ISO standards for Software Quality (ISO 9126), for Ergonomic requirements for office work with visual display terminals (ISO 9241: 10-17), for Multimedia user interfaces (ISO 14915) and Accessibility of human-computer interfaces (ISO 16071). Important input for the process oriented criteria are gained from the ISO standards for Human-centred design (ISO 13407) and for Ergonomics of human-system interaction (ISO 18529) which provide means for assessing processes in respect to all stakeholder interests involved during the full lifecycle span of the system under consideration. The Swedish and the American Customer Satisfaction Index involve a large number of respondents in order to assess the relative customer satisfaction with products and services in a wide set of industries and service sectors. The UsersAward on the other hand aims at involving representative panels of end-users from three or more workplaces in which the software to be certified has been deployed in comparable configurations. The long-term goal of this strategy is to initiate sustainable dialogues on software quality both within and between these (and other) workplaces. The idea of assessing at least three independent deployments, of which at least two should pass the certification requirements, is inspired by the Internet Engineering Task Force (IETF) which has as a basic requirement that complete implementations of a candidate protocol/standard/application shall be demonstrated on at least two different platforms. UsersAward s emphasis on sustainable workplace dialogues, based on clear statements of requirements, is derived from the findings of numerous research projects which all verify that user participation and management engagement are two of the most important factors for a successful deployment process. For example, the 1996 update of the Chaos Report, a survey compiled since 1994 by the Standish Group [15], is particularly revealing in this respect: "The three biggest contributors to project success are user involvement, executive support and a clear statement of business objectives. Together they account for 50 % of a project's chances for success." The fact that user involvement is found to be the most important factor for success confirms the UsersAward strategy for sustainable quality work. The report's estimation that "user involvement" has a weight of 20 points, while "executive support" and "a clear statement of business objectives" has 15 points each, further verifies the necessity of not only involving end-users in the certification process, but to make the process user-driven in respect to criteria articulation and institutional affiliation. 10

13 Differences between UC 2002 and UC 2006 The USER CERTIFIED 2006 version of the programme is an upgrade of the USER CERTIFIED 2002 (UC 2002) and is based on experiences from three UC 2002 certifications and three national surveys of the industrial, health care, and banking sectors IT Maps founded on the UC method and carried out by UAAB (Lind 2002, 2004, UsersAward 2005). These experiences have been analyzed and discussed in the research panel and in a series of workshops, conducted in Stockholm and in Germany, with a group of German and Austrian researchers and trade union leaders who were familiar with the TCO programme in Germany and who saw UsersAward as a viable and important extension of it. In the new UC 2006 version, three main problems are addressed. Firstly, a further step is taken towards designing the instrument for an extended adaptability to specific software categories and thereby to make it more conclusive in respect to workplace context awareness. This is done by including more specific questions about context of use and supported work tasks, both in the questionnaires underlying the software provider self declaration, and in the user questionnaire. Secondly, the German group found some of the criteria in UC 2002 too closely tied to Swedish labour traditions and proposed a more general orientation in content. Some of these proposals have been implemented in UC One important change for making the instrument applicable for a wider European use is that the basic norm for passing the certification was lowered from four out of five satisfied users (80% with the quite demanding agreement level!4 out of 6) to two out of three satisfied users (67% with the same agreement level). Another change to that effect is that the stipulated level of user satisfaction is not applied to subdivisions of the respondents into user categories (management/employees, professions, male/female) as was done in the initial UC 2002 certification. Discernable differences between different user categories may be commented on in the certification protocol. Thirdly, the UC 2002 was not fit to handle certain kinds of dedicated software that had few users in each workplace but that had critical impact on operations, typically financial or technical applications. This caused a revision in the number of workplaces, allowing for several smaller workplaces to be involved in the investigation, in order to fulfil the requirement of the minimum number of users to be involved in the certification process. Parallel to these revisions on the structural level, a series of changes has been made to the user questionnaire, the certification protocol, and how they are used in the process: in order to enhance the validity and the international applicability of the criteria set, the number of quality criteria is extended from 30 to 38 (resolution of composite statements, fewer statements regarding the relative autonomy of work groups, further statements on technical performance and context of use) as a consequence of not subdividing the user community into user categories, the same version of the questionnaire is used for management and employees in order to tighten the interplay between the software provider s self declaration and the user feedback, the management on all workplaces are asked to comment the self declaration and these comments are reported, and in turn commented by the certification team, in the final certification protocol 11

14 the certification protocol is given a more concise reporting style (separate diagrams and comments for each of the basic certification requirements and quality areas) while the more qualitative reporting is reserved for a special certification report to the software provider (a separate document) the hearing, performed with the software provider by the certification team for gaining a deeper understanding of the services offered, is replaced in UC 2006 with an initial meeting with the software provider and a concluding presentation of a certification report. Future work An important underlying principle for all future research and development work is that new versions should, as a matter of necessity, comply explicitly with the standards issued by the industry based standards organisations, notably ISO/CEN. Thereby it can complement, e.g., ISO (and upcoming updates of ISO 13407) with its focus on process management, ITIL-ISO with its focus on technology and control of configurations, and ISO/IEC Common Industry Format for Usability Test Reports with its focus on situated usability tests. The research group and UsersAward investigators also agree that new versions should have explicit provisions for future extensions of the programme. In the UC instrument, this is mainly done by successively more demanding self declarations in which software providers, who sincerely strive to satisfy their users, can specify their competitive edge. The cooperation of each applying software provider through their self declaration assures that the users judgement can be compared to the claims about technology and context of use made by the provider in the certification protocol. This is where a systematic extension of the User Certified instrument s adaptability to different software categories can make a difference, for understanding the relative importance, in terms of competence building and productivity, of different kinds of workplace applications. The ongoing discussions with researchers in Denmark, Finland, Germany and Austria, in common seminars and in the UITQ 2005 and 2007 workshops, have made it clear that new version should develop the applicability to European countries further, e.g. by making the complementary nature of the certificate more explicit and by adhering to new European standards and new vocabularies. This can only be done by long term collaborative efforts between researchers and the labour market partners. An interesting venue for research and the certification effort is to further explore the importance of the software provider s long-term personal experiences from the application domain for which they provide software support. This aspect is covered in the self declaration with questions on how knowledge from the application domain has been acquired. And its impact on user satisfaction has implications for both research to cast light on issues of language and cooperative work patterns of successful software providers and for design to elaborate questions in the self declaration, e.g. about how those cooperative work patterns are sustained by the professional languages used. 12

15 Closely linked to the issue of the software provider s personal experience from the application domain is the sometimes prominent role played by consultants in the area of interaction design and software deployment. As enterprise resource planning software tends to become more standardized, consultants emerge as the decisive agents of software integration and deployment. This means that consultants assume responsibility for integrating software from different providers and carrying through a deployment process, that not only takes account of particularities in the context of use in the buying organizations, but also of the particularities of the different software packages employed. Here, a set of carefully thought out questions in the self declaration could help gather valuable empirical data on best practices e.g. cooperative work patterns for consultants in interaction design, software integration and software deployment. Acknowledgements We want to thank Peter Forss and Ove Ivarsen for inviting us to take part in the user community and networking work at LO, which has provided the ground for the research and the development of this certification programme. We also want to thank Jan Gulliksen, Robert Ferm, Torbjörn Ilar, Christian Koch, Paula Kökeritz, and Martin Ljungström who have made important contributions to the pilot studies and the assessment method and Irene Odgaard, Peter Carstensen, Timo Jokela, Sabine Hegner, Peter Brödner, Jochen Prümper, Reinhard Linz, Jörn Hurtienne, Thomas Vöhringer- Kuhnt, Stefanie Floegel, Thomas Riesenecker and Alan Borning, who have given valuable comments on our research from an international point of view. Last but not least we want to thank Per Erik Boivie and Tomas Berns who were both heavily involved in creating the TCO certification program and who have inspired and followed the ITQ project throughout. The ITQ project and its currently running follow-up project have been funded by VINNOVA, the Swedish Governmental Agency for Innovation Systems. 13

16 References 1. Boivie, P.-E., Asplund, M., Ekström, B., Fransson, K., Frey, C., Northman-Alm, T., Petrelius, T. and Prieto Beaulieu, M. (1997): TCO 95 for the professional user and the external environment, in Proceedings of WWDU 97, Tokyo, November Håkansson, B. (2000): Egendeklaration av programvaror. Stockholm: Teldok. 3. Ilar, T., Lind, T. och Walldius, Å. (2002): Simulering som verktyg i förbättringsarbete pilotprojekt på Arvika Gjuteri. Teknisk rapport TRITA-NA-D0202 och CID International Organization for Standardization (1998): ISO 9241 Software ergonomics with visual display terminals (VDTs). International standard. (Switzerland: International Organization for Standardization). 5. International Organization for Standardization (1999): ISO Human centred design process for interactive systems. International standard. (Switzerland: International Organization for Standardization). 6. International Organization for Standardization (2004): ISO Ergonomics Ergonomics of human-system interaction Human-centred lifecycle process descriptions. International standard. (Switzerland: International Organization for Standardization). 7. International Organization for Standardization (2005): ISO Ease of operation of everyday products. International standard. (Switzerland: International Organization for Standardization). 8. Lind, T., (2002): IT-Kartan, LO: Användare och IT-system i svenskt näringsliv. 9. Lind, T., Sandblad, B., Johansson, N. and Utbult, M., (2004): Vård-IT Kartan, UsersAward. 10. SCB, SCB:s analysmodell med Nöjd-Kund-Index för offentlig sektor, asp 11. SCB, SCB:s Analysmodell: Fördjupad beskrivning, asp 12. Sundblad, Y., Lind, T., and Rudling, J. (2002): IT product requirements and certification from the users perspective Proceedings of WWDU 2002 Conference, Bavaria, pp , May 2002, Also available as report CID-176, Sundblad, Y., Walldius, Å., Lind, T., Sandblad, B., Gulliksen, J., Löfberg, M., Bengtsson, L., Ljungström, M och Ilar, T. (2002): ITQ - Kvalitetssäkring av IT-stöd för det utvecklande arbetet. Teknisk rapport TRITA-NA-D0202, available as report CID- 170, KTH, The American Customer Satisfaction Index, accessed

17 15. The Standish Group International (1996): Unfinished Voyages - A Follow-Up to The CHAOS Report, accessed Walldius, Å., Ljungström, M. och Bengtsson, L. (2002): Umeverkens IT-brygga - Webbstöd för överblick och dialog kring arbetsplatsens systemintegration Sammanfattning av intervjuundersökning och prototyputveckling. Teknisk rapport TRITA-NA-D0201, available as report CID-169, Walldius, Å., Sundblad, Y. and Lind T. (2003): A User-driven Workplace Software Certification Process, Teknisk rapport TRITA-NA-D0301, KTH, available as report CID-221, Walldius, Å., and Sundblad, Y. (2002): User-driven software quality labeling, Proceedings of DIAC 2002 Conference, Seattle, May 2002, available as report CID-177, CID, Walldius, Å., Sundblad, Y., Sandblad, B., Bengtsson, L., & Gulliksen, J. (2007), User certification of Workplace Software Assessing both Artefact and Usage, accepted for publication in BIT (Behaviour & Information Technology), Vol. 26, 21 pages. 15