A Performance-Oriented Approach to E-Learning in the Workplace

Transcription

1 Wang, M., Ran, W., Liao, J., & Yang, S. J. H. (2010). A Performance-Oriented Approach to E-Learning in the Workplace. Educational Technology & Society, 13 (4), A Performance-Oriented Approach to E-Learning in the Workplace Minhong Wang 1, Weijia Ran 1, Jian Liao 2 and Stephen J.H. Yang 3 1 Faculty of Education, The University of Hong Kong // 2 E-Learning School, South West University, China // 3 Department of Computer Science and Information Engineering, National Central University, Taiwan magwang@hku.hk // ranwj@hku.hk // leojames123@gmail.com // jhyang@csie.ncu.edu.tw ABSTRACT Despite the ever-increasing practice of using e-learning in the workplace, most of the applications perform poorly in motivating employees to learn. Most workplace e-learning applications fail to meet the needs of learners and ultimately fail to serve the organization s quest for success. To solve this problem, we need to examine what workplace e-learning requires and how workplace e-learning systems should be developed in line with those requirements. We investigated the problem by identifying the fundamental elements of the workplace learning environment including the learner, organization, learning content and social context, and their relationships. We found that workplace e-learning should align individual and organizational learning needs, connect learning and work performance, and support social interaction among individuals. To achieve this, a performance-oriented approach is proposed in this study. Key performance indicators are utilized to clarify organizational goals, make sense of work context and requests on work performance, and accordingly help employees set up rational learning objectives and enhance their learning process. Using this approach, a prototype system has been developed and a set of experiments have been conducted to demonstrate the effectiveness of the approach. Keywords E-Learning, Workplace, Web-based Training, Performance, Learning System, Ontology Introduction E-learning refers to the use of computer network technology, primarily via the Internet, to deliver information and instructions to individuals. Due to its access flexibility and just-in-time delivery, e-learning is emerging as a popular approach for learning in organizations or workplace settings (Rosenberg, 2006; Sambrook, 2003). Despite the everincreasing practice of using e-learning in the workplace, most of the applications perform poorly in motivating employees to learn. Significant gaps exist between corporate interests and learner needs when it comes to e-learning (Brink et al., 2002; Servage, 2005). For individuals, although knowledge can be learned by participating in e-learning programs, more often they do not think e-learning is helpful since the knowledge learned cannot help improve their work performance. For organizations, e-learning is generally designed without meeting the organizational vision and mission. Moreover, current e-learning development tends to focus on technical issues of design and ignores pedagogical and organizational issues that are necessary for effective e-learning programs to address (Tynjälä & Häkkinen, 2005). The dominance of technology-oriented approaches has made e-learning practices less goaleffective, and they are therefore perceived to be poor in quality and design. On further review of the root of the problem, it seems that much of e-learning research is based on formal courses in educational institutions. However, corporations as learning arenas are different from schools. Workplace learning is built on practical tasks and work situations with the aim to serve organizational goals. Learning in the work environment takes place in the context of use and application, and as a result is often embedded in work practices. Moreover, learning is more collaborative in workplace settings, where sharing individual knowledge with co-workers is an important part of the learning practice. The aforementioned problem highlights the need to design learning activities that address corporate interests, individual needs, and work context. The development of workplace e-learning should consider the alignment of individual and organizational learning needs, the connection between learning and work performance, and communication among individuals (Wang, in press). To solve the problem, a performance-oriented approach is presented in this study. A set of key performance indicators (KPIs) has been set up to represent a set of measures focusing on the aspects of organizational and individual performance that are critical for the success of the organization (Ran & Wang, 2008; Ran et al., 2008). The KPI framework provides a clear picture for everyone in the organization of what is important and what they need to do and learn. The mechanism of the approach is explored and elaborated with conceptual frameworks and implementation details. To demonstrate the effectiveness of the ISSN (online) and (print). International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 167

2 approach, a prototype of a workplace e-learning system has been developed with relevant experiments to evaluate the effectiveness of the approach. Theoretical Background Workplace learning refers to learning or training activities undertaken in the workplace, with the goal of enhancing individual and organizational performance (Rosenberg, 2006). Attention to workplace learning has greatly increased due to the significant role of professional skills and expertise in organization development. Theories specific to workplace learning can be categorized into adult learning, organizational learning, and knowledge management (KM). Adult learning theories form the basis for the design of e-learning practices in work environments. Andragogy (learning strategies focused on adults) and self-directed learning are two fundamental parts of adult learning. The implication of adult learning theory in the workplace context is that learners would be motivated once learning objectives have been rationally set that would meet their needs (Knowles et al., 1998). According to self-directed learning theory, learning programs should be designed to give emphasis to self-directed learning so as to help learners make sense of the workplace and their experiences at work (Merriam, 2001). Organizational learning concerns both the ways individuals learn in an organizational context and the ways in which organizations themselves can be said to learn (Easterby-Smith et al., 1999). Organizational theory implies that learning occurs and should be addressed beyond the individual level. Its pedagogical focus is on organizational systems, structures, and policies, along with institutional forms of memory to link individual and organizational learning. In relation to organizational learning, knowledge management (KM) represents another discipline. It refers to a range of approaches and practices used by organizations to identify, create, represent, and distribute knowledge for reuse, awareness, and learning (Nonaka & Takeuchi, 1995). Recent research has motivated the integration of knowledge management with e-learning for organizational development (Wang & Yang, 2009). How knowledge management and e-learning apply to and affect organizations is a complicated, yet important question that requires a variety of conceptual, methodological, and technical approaches. Preliminary Investigation Diverse theories exist related to workplace e-learning, each emphasizing a different perspective. Due to the lack of holistic understanding for interdisciplinary study and the lack of appropriate conceptual and methodological tools for implementation, e-learning in the workplace remains a fragmented, complex, and challenging area of research and practice (Collin, 2006; Servage, 2005;). Researchers in systems theory conceptualize learning in organizations from a macro-systems perspective, i.e., organizational learning is a system that must adapt or react to environmental pressures. If effective, it can lead an organization to change or learn to change faster to increase the organization s chances of survival. System thinking is a discipline for viewing things as a whole. It is a framework for seeing interrelationships rather than individual parts, and for seeing the structures that underlie complex situations. Based on system thinking, it is found that workplace learning is confronted with a highly complex set of variables such as learners, activities, outcomes, organization, and contexts, as well as interactions between the variables such as the effect of organizational context on learners motivations (Smith & Sadler-Smith, 2006). In order to have a better understanding of workplace learning, we refer to the fundamental elements of a learning environment addressed in Illeris (2003): (a) learners, who are the chief actors in the learning environment, e.g., employees in the organization; (b) learning content, e.g., the knowledge and expertise required in work practice; (c) social context, which considers groups and teams in the workplace; and (d) other learning stakeholders, such as the organization, society, or parents. An effective workplace learning application should take the four elements and their interactions into consideration. First, in workplace settings, employees are adult learners with distinctive learning characteristics. They have distinct job responsibilities, which require different types and levels of expertise. Even assigned with an identical task, employees would have different learning needs and expectations as a result of different educational backgrounds, working history, and learning performance. Second, different from formal learning in educational institutions, learning in the workplace serves organizational goals and needs, and it focuses on organizational systems, structures, policies, and institutional forms of knowledge to link individual and organizational learning. Third, learning content in the workplace is more contextual and dynamic than that in typical school settings. The learning content is contextual in that knowledge in the workplace is disseminated within an organization and arises from employees daily activities and interaction with the working environment (Raelin, 168

3 1998). Fourth, the workplace learning environment is a knowledge society that builds upon a community of practice. Learning in the workplace can be understood as social networking between learners, which allows the creation and transfer of knowledge among individuals and groups. Based on the above analysis, we should direct learning activities in the workplace to address corporate interests, individual needs, work performance, and social context. The development of workplace e-learning applications should consider the alignment of individual and organizational learning needs, the connection between learning and work performance, and the interaction between individual learners, as shown in Figure 1. First, the system should align individual and organizational learning needs. This can be achieved by (a) identifying the training requirements of an organization based on its mission and vision, job design, and reward mechanism; and (b) helping employees determine their learning objectives based on the organizational request and individual performance. Second, the system should link learning with work context and the performance request. This can be achieved by (a) identifying the expertise required for a job position, and (b) providing suitable learning resources and instructions for learners to develop the required expertise. Third, the system should be able to facilitate social interaction and knowledge exchange in the learning community. The knowledge sharing and transfer processes can be directed by linking them to the work context and practices and by linking learners of similar background or learning interest. Figure 1. Learning in the workplace Performance-Oriented Workplace E-Learning Design There is no doubt that the goal of e-learning in the workplace is to enhance individual and organizational performance (Rosenberg, 2006). However, there is a lack of concrete strategies or approaches for achieving this goal in e-learning development. To solve this problem and meet the aforementioned requirements of e-learning in the workplace, a performance-oriented approach is presented in this study. Performance measurement is crucial for organization development, and therefore it is a main driver of learning in the workplace. In this approach, a set of key performance indicators (KPIs) can be set up to represent a set of measures focusing on the aspects of organizational and individual performance that are critical for the success of the organization. KPI is a flexible and popular approach to conducting performance measurement in organizations. The mechanisms of why and how we use this KPI-oriented approach to workplace learning are elaborated as follows. Performance-Oriented Approach Performance measurement is used by organizations as a procedure to improve performance by setting clear objectives, assessing performance, collecting and analyzing performance data, and utilizing performance results to drive performance development (Baker, 1995; Parmenter, 2007; Slizyte & Bakanauskiene, 2007). KPIs are financial and nonfinancial metrics used to help an organization define and measure progress toward its goals. KPIs can be used to assess almost any aspect of work performance, depending on an individual organization s design. KPIs are typically tied to an organization s overall strategy, and they differ according to the nature of the organization and its strategy. In a KPI system, organizational vision and mission are converted into clearly defined key performance 169

4 targets for the organization and its business units; based on the unit s goals and objectives, official expertise and capabilities required for each position in the unit can be defined. A KPI framework provides everyone in the organization with a clear picture of what is important and what they need to do. The KPI-based performance measurement approach has special meaning for learning in the workplace. KPIs bridge the gap between an organization s mission and vision and its employees targets, and making organizational goals accomplishable. KPIs can be used to help employees set up rational learning objectives based on their job position and knowledge gap. It can be used as a systemic scheme to organize and manage learning resources in line with work context and performance requests. KPIs can also be used to facilitate social interaction among individuals by identifying employees work context, expertise, and performance proficiency. In brief, KPIs can be used to support (a) the alignment of individual learning needs and organizational interests, (b) the connection between learning and work performance, and (c) social communication between individuals. Performance-Oriented System Design A KPI framework encompasses an organization s structure and job system. It consists of three levels: the organizational level, business unit level, and position level. KPIs on the organizational level are defined according to organizational goals and strategies. Derived from the organizational KPIs, the KPIs for each business unit are specified. Based on the unit KPIs, the KPIs for each job position within the unit are then defined. For performance measurement to be effective, the measures or indicators themselves must be accepted, understood, and owned by employees as well as their managers. Therefore, the building of a KPI framework requires cohesion and integration of different strategies as well as tight cooperation among managers and employees from different units and at different position levels in the organization (Parmenter, 2007). KPIs for a position in one unit can be reused in other units for a similar position, or where similar capabilities are required. In this study, due to the space limitation, we focus on KPIs at a position level that has a close relationship with learning or training programs in the workplace. Employee ID Job Position Junior Tester Senior Tester KPI Item (Capability) Bug Reporting Table 1: A KPI Framework at the Position Level Rating Criterion Test (Weight: 1/3): Level 1: score [0,20) Level 2: score [20,50) Level 3: score [50,70) Level 4: score [70,90) Level 5: score [90,100] Peer Assessment (Weight: 1/3): Supervisor Assessment (Weight: 1/3): Levels and Criterion defined: 0: Do not know 1: Know little about this area 2: Know basic knowledge about this area 3: Have substantial knowledge about this area 4: Use related knowledge to accomplish tasks 5: Use related knowledge to achieve sound effect KPI Value (Assessment Result) Score obtained: 65 Level 3 Rating: 3 Peer Assessment: Level 4 Rating: 4 Supervisor Assessment: Level 3 Rating: 3 Overall 3*(1/3) + 4*(1/3) + 3*(1/3) = 3.33 Test Execution 170

5 The KPI at the position level consists of three components: KPI item, rating criterion, and KPI value. KPI items are a set of performance indicators specified for a job position. For example, oral and written communication skills might be two KPI items defined for a sales job position. For each KPI item, a rating criterion is set up to assess performance. The proficiency level achieved by an employee on that item is called a KPI value. An employee s performance measure result is a set of KPI values for his or her job position. Tests or quizzes can be used to assess how an employee performs with a certain KPI item. To preserve impartiality and objectivity, most organizations use 360-degree feedback to assess employees performance. This means that the employee s performance can be assessed by the employee him- or herself, the employee s supervisor, his or her subordinates, and peers, in addition to taking standard tests. Each appraiser gives the employee a set of KPI values, and each appraisal is given a certain weight. As a result, a set of KPI values will be calculated to evaluate the employee s work performance. An illustration of the KPI framework at the position level is shown in Table 1. Figure 2. Performance-oriented workplace e-learning architecture System Architecture KPIs help employees make sense of their work context and required expertise, and accordingly help them set up rational learning objectives, access relevant knowledge resources, and communicate with peers or experts to enhance their work performance. The mechanism for how the KPI framework works as a driver for e-learning in the workplace is depicted in a system architecture. As shown in Figure 2, the learning system provides interfaces to the learner, domain expert, and training manager. The Learner interface enables a learner to maintain personal information; access, share, and evaluate learning resources; assess his/her learning performance; and interact with other learners. The Expert interface enables a domain expert to maintain the KPI framework, process and maintain 171

6 learning materials, generate and update learning objects based on learning materials, and coordinate discussions. The Training Manager interface enables a training manager to manage learners profiles, maintain the assessment base, define instructional rules, and manage the KPI framework with the domain expert. Moreover, the system can guide individual learning processes according to the learner s performance gap and learning progress. This can be achieved by measuring and analyzing individual performance, managing and monitoring individual learning processes, recommending learning activities and resources to individuals, and sending personalized learning instructions and alerts. A KPI framework is constructed based on the organization s vision and mission, as well as its organizational structure and strategy. Based on the KPI framework, a KPI ontology is then defined. This includes the specifications for all the positions; the KPI items for each position; the capabilities required for each KPI; as well as the relations between the positions, KPI items, and capabilities. As shown in Table 2, the KPI ontology specifies the relationships among all the capabilities including prerequisite, composition, relevance, and inhibitor. A portion of an ontology, which describes the capabilities and their relationships required for the position Junior Tester in a software development company, is presented in Figure 3. In this example, if a learner intends to acquire the capability of Testing Specific Skills, he or she needs to acquire two composite capabilities, Bug Reporting and Test Execution ; before acquiring the two capabilities, he or she must acquire the capability Testing Basic Concepts and Definitions, which involves three composite capabilities: Testing-Related Terminology, Theoretical Foundation, and Relationships of Testing to Other Activities. Table 2. Relationships between capabilities Name of Relation Representation of Relation Representation of Relation Composition Par (a, b) Capability a is a part of capability b. Prerequisite Seq (a, b) Capability a is a prerequisite of capability b. Relevance Rev (a, b) Capability a is relevant to capability b. Inhibitor Inh (a, b) If capability a is learned, capability b is not necessary to be learned. Figure 3. An example of the ontology for capabilities The KPI ontology can be used to determine individual learning content and learning paths according to the employee s position and the required capabilities with their relationships. Relevant learning instructions or rules can be specified by the training manager or experts to support the reasoning process. For example, if a capability a has an 172

7 inhibitor or alternate b, i.e., Inh(a,b), there is no need to learn both a and b. The rule for such refinement is specified as follows. where Req(a) denotes that capability a is needed for the learner, Mas(a) means that capability a has been mastered by the learner, and Hsc(a, b) denotes that the learner s assessment score of a is higher than that of b.. Furthermore, an individual learning process should also consider the learner s knowledge gap. As in the example shown in Figure 4, if the learner has already acquired the expertise of Theoretical Foundation, he or she may skip this item in the learning process. The knowledge gap can be identified by using assessment methods provided by the system or by self-estimation of the learner. Figure 4. An example of a learning process While the system provides instructions to learners on their learning progress, individuals can adjust their learning scope and learning path according to their preference. In this context, the ontology, which is visualized as a knowledge map in the system, can be used as a reference for learners to navigate the complex learning environment and construct their knowledge according to their own request. In this study, the KPI ontology drives learning in the workplace as follows: (a) Each positions links to one or more KPI items. (b) Each KPI item links to one or more required capabilities. (c) Each capability links to one or more learning objects, which are categorized into different types such as articles, books, web pages, video files, and so forth. Learning objects are generated by experts based on learning materials contributed by learners, or directly uploaded by the training manager or experts. (d) Each capability also links to an assessment package. The package includes assessment methods (e.g., tests, quizzes, and peer evaluation) and rating criteria for assessment. (e) Moreover, each employee is provided with a KPI profile, i.e., a set of KPI values that indentifies his or her work context, expertise, and proficiency level, which is stored in the learner s profile. At the same time, social communication between learners can also be facilitated by KPI. Learners, including experts, are able to know about each other based on their job position, expertise and proficiency, and their participation in the learning community. Learners are able to share and contribute learning materials, including discussion messages, by using capability as the index. Moreover, learners are able to conduct peer evaluation of their work performance and evaluate learning resources for interactive learning. In this way, self-directed and socially constructed learning 173

8 activities in the workplace are effectively directed via the integration of organizational interests, individual needs, work performance, and social context. Prototype System To demonstrate the effectiveness of the performance-oriented approach, a prototype system has been developed for Peanut, a medium-sized software company. The organization structure of Peanut and its job positions are outlined in Figure 5. In this prototype, the e-learning development is focused on the Testing Unit. Testing is an important and mandatory part of software development; it is a technique for evaluating software product quality by identifying defects and problems. The Testing Unit of this company has defined Bug Found and Bug Returned KPI as the standards for measuring productivity and quality of software development. The KPI framework of this prototype is constructed based on the company s strategy as well as IEEE standards of software testing introduced in Bertolino (2001). Figure 5. An organizational structure with positions The prototype is built using Java programming tools. In the prototype, two platforms are provided, one for the training manager and expert, and another for individual learners. User interfaces enable users (learner, expert, training manager) to access the learning system via the Internet. The interfaces and operations for different roles of users are outlined in Figure 2. To synchronize the operations between the two platforms, the Java Database Connectivity (JDBC) technique is used. To develop the KPI framework, Protégé is used for designing and constructing the KPI ontology. Protégé is a free, open-source Java tool developed at Stanford University for the editing of ontology and knowledge frameworks; it provides a powerful environment and plug-in Application Programming Interface (API) for developing knowledgebased applications (Noy et al., 2000). With the help of an XML parser, the ontology is able to link to the capability items stored in the database. Each capability is associated with relevant resources including learning objects, assessment packages, and discussion items stored in the database. A semi-open-source component, JGraph, is also used for display and auto layout of the ontology in graphs. In this prototype, Ontology Web Language (OWL), specifically, Description Language (OWL-DL) (Horrocks et al., 2005) is used to define the KPI ontology. To support the reasoning services, instruction rules are bound with the ontology using DL-safe Semantic Web Rule Language (SWRL). The OWL ontology and SWRL rules are used to generate individual learning processes according to the position and the relationships between the required capabilities defined in the ontology as well as the knowledge gap of individuals. To implement both OWL ontology and SWRL rules, we use OWL-API to access Pellet (Sirin et al., 2007) as the semantic reasoning tool of this system. Moreover, to enable domain experts and the training manager to construct and maintain the ontology, tools for ontology editing and visualization are necessary. In this study, Protégé together with SWRL tab and Jambalaya tab plug-ins are employed. 174

9 A set of screenshots of the prototype are presented in Figure 6. The ontology is visualized in graphs for easy communication of the learning context. The learner is able to locate learning resources related to a specific capability item by clicking the capability in the graph. The learner s performance can be assessed via tests or by evaluation from peers and supervisor. The performance results can be combined into KPI values. Each KPI value is represented by a color in the graph, which indicates the overall proficiency level of the capability. In addition, learners are able to share and evaluate learning objects as well as participate in discussions or communications. To facilitate the social communication, learners are able to locate peers or experts according to their background and expertise, as well as their contributions to the learning community. System Evaluation Method Figure 6. Screenshots of the e-learning prototype To evaluate the effectiveness of the prototype, we invited a number of employees who currently work or previously worked with the Testing Unit of the company to participate in the experiments. Two parallel systems were used for evaluation the prototype system A developed using the KPI-oriented approach, and another system, B, developed using the traditional approach without KPI support. System B has similar functions to the prototype A in terms of user management, learning and assessment management, and communication tools, but without KPI-based facilities. The participants were divided into two groups of 12 the treatment group that used the KPI-based system and the control group that used the traditional system. The data collection process can be roughly divided into three stages. First, the participants completed a pretest. Second, after using the system for 4 weeks, participants completed the posttest and the questionnaire for evaluation of the workplace e-learning system. Third, interviews were conducted for qualitative feedback from the participants, as well as gaining their opinions on the e-learning system. The methods used for evaluation consisted of a mixture of quantitative and qualitative data-gathering approaches. The data collected included learning outcome-related data obtained through pretests and posttests, and participants perception data obtained through questionnaires and interviews. The evaluation was conducted based on Donald 175

10 Kirkpatrick s model (Kirkpatrick et al., 2006), which is proposed for evaluation of training programs. The model includes four levels: Reaction (how participants react to the learning system), Learning (knowledge learning or skill development by using the application), Behavior (transfer of learning into change of behavior by using the system), and Result (organizational and individual outcome as a result of the training program). Questionnaire items were developed mainly based on Kirkpatrick et al. (2006) and other studies with respect to the evaluation of e-learning systems such as Sun et al. (2008). Pretest and posttest questions were designed based on certification examinations in the software testing profession and adjusted by subject experts. The participants evaluation of the system via the questionnaires was measured using a Likert scale (from 1, strongly disagree, to 7, strongly agree). The scores on the pretest and posttest were measured according to the number of questions that were answered correctly. Results The results obtained from the questionnaire and tests are presented in Table 3. The initial findings show that the KPIoriented system is perceived to be significantly more effective in terms of content management and functional support for learning, more helpful for learners to obtain knowledge, and more helpful in enabling learners to integrate learning into practice and transform individual learning into collaborative learning. On the other hand, the results of the pretest and posttest scores indicate no significant difference between the two groups. The results are understandable, since the duration of the experiment (4 weeks) is not quite sufficient for evaluation, and other factors of the learners (e.g., learning capability and efforts) as well as learning environment (e.g., Internet accessibility, speed, and cost) may have affected the results. Table 3. Evaluation results Level Aspect KPI System Non KPI System Reaction meeting individual learning requirements providing satisfactory learning functions Learning pretest score vs. posttest score 7.3 / / 8.0 increasing knowledge Behavior integrating learning with work practice facilitating social learning Result improving individual work performance providing benefits to the organization The findings from the pilot study are limited due to the small sample size and short period of time of the experiment. In this context, interviews were conducted to collect qualitative comments or feedback from the users. The interviews were conducted after the two groups were asked to switch and use the other system for 2 weeks. A total of 20 participants were ultimately interviewed. Each participant was interviewed individually via telephone or Windows Live Messenger. Each interview lasted approximately 30 minutes. To ensure the reliability of the interview, we conducted 2 3 interviews with each participant; some interview questions were repeated for the same participant at different times. The findings from the interviews are organized in terms of the role of the interviewee employee, expert, or training manager. Most of the employees are concerned about the learning content and the interaction and communication functions supported by the system. In relation to learning content, they believe that the availability of comprehensive and abundant learning materials is crucial to an e-learning system. Most feel the KPI-oriented prototype is more helpful because it provides a broad scope of learning materials and helps learners become familiar with the domain knowledge of software testing in a systemic way. Further, most participants report that a clear and flexible classification scheme of learning materials is very important and could be improved in the prototype. For example, some participants recommend that learning materials be organized around actual projects or cases. In relation to interaction and communication functions provided by the learning system, the participants feel that they are important elements that can motivate learners to learn and get engaged with the system. They gave positive comments to the KPI-oriented system on its facilities for communications, knowledge sharing, and discussion. As for the experts, they are more concerned about how an e-learning system can guide employees to learn. Compared to the employees, the experts gave more positive comments on the KPI-based learning system. They also believe that the construction of the capability framework in the KPI model is an evolving process and that 176

11 cooperation is needed from both designers and users of the e-learning system. They suggested that the construction process be roughly divided into three stages. First, system designers should define the initial framework of the capabilities required for the positions in the workplace according to the KPI model and existing industrial standards. Second, experts in the workplace should modify the framework according to their experience and the context of the workplace. Third, the framework should be continuously modified and improved. With respect to the training managers, the greatest issue of concern is cost. They prefer e-learning systems to traditional ways of training in classrooms. They gave positive comments on the KPI-oriented e-learning system, since they felt that it provided various methods of learning and assessment. Related Work E-learning has been attracting considerable interest, and a variety of perspectives have developed. Although much of e-learning research is based on formal courses in educational institutions such as Chiu et al. (2008), workplace e- learning or web-based training is being studied by a significant number of groups. PROLEARN, a Network of Excellence financed by the Information Society Technology program of the European commission dealing with technology-enhanced professional learning, is working on approaches and guidelines for companies to implement knowledge work and learning management according to business needs and business processes (Zimmermann et al., 2007). PROLEARN has developed a web portal supporting learning performance assessment, including the selection of learning performance metrics, validation of scales, executions of online surveys, and interpretation of results (Seirafi, 2008). Another project, APOSDLE, supported by the European Union, aims to enhance knowledge worker productivity by supporting informal learning activities in the context of knowledge workers daily work processes. They have developed a flexible framework for enterprise modeling that contains formal and informal knowledge and integrates the modeling of domain, processes, and competences within an enterprise (Ghidini et al., 2008). Other related work includes competency-based learning, where learning is driven by development of specific competencies for dealing with needs and challenges in competitive environments. Korossy (1997) used the competency performance approach to extend the theory of knowledge space. Based on Korossy s work, Ley and Albert (2003) investigated a competency method with a focus on organizing learning content in line with competencies. Regarding e-learning systems development, extensive research and practices have been carried out in the analysis, design, and delivery of learning systems, which integrates concepts, processes, and principles of instructional design, software engineering, and knowledge engineering (Paquette, 2006; Xu et al., 2005). In these studies, ontology and semantic web technologies are used as the backbone for e-learning systems. They provide a semantically structured learning space that allows effective search and navigation activities in the learning environment (Knight et al., 2006; Stojanovic et al., 2001; Vassileva, 1995). Most of the current studies of workplace learning are more content-driven. Their e-learning applications place more emphasis on learning resources with their semantic annotations and organization to facilitate intelligent functionalities such as reasoning of learning contents for provision of customized courses or environments (Mahmood et al., 2006). Also, most approaches are technology-oriented, with less attention to the nature of workplace learning, which takes place in an organizational and social context. The pedagogical focus on organizational systems, structures, and policies that link individual and organizational learning has not been well addressed. In many situations, the complexity of the interactions between individual learning and organizational strategy has been underestimated. In this study, we build a performance-oriented learning environment in which job positions and their KPIs, together with the required capabilities, learning resources, and assessment methods, are well specified and integrated in a formal and explicit way. Our learning ontology has gone beyond learning content by including learning objectives (reflected in performance targets) and assessment methods in line with the KPI model. In this way, a sound pedagogical underpinning is provided in the design of the learning ontology to implement a coherent and consistent e-learning environment in the workplace. Discussion and Conclusion This study has addressed the problem of e-learning applications in workplace settings, i.e., failure to meet the needs of learners and the organization s quest for success. Starting from this problem, we examined what workplace e- 177

12 learning requires and how workplace e-learning systems should be developed in line with these requirements. We investigated the problem by identifying the fundamental elements of a workplace learning environment (learner, organization, learning content, and social context) and their relationships. We found that workplace e-learning should be able to align individual and organizational learning needs, connect learning and work performance, and support social interaction among individuals. To achieve this, a systemic approach is required in which considerations regarding pedagogy, organization, and technology are integrated and balanced. In this study, we have proposed a performance-oriented approach. KPIs are used for assisting organizations to clarify their training objectives, helping individuals make sense of work context and performance requests, and accordingly helping individuals set up rational learning objectives and enhance their learning process. With this approach, a prototype system has been developed by using ontology technology to implement the KPI framework and make it drive the system functions. A set of experiments has been conducted to demonstrate the effectiveness of the approach. Based on the evaluation results, we will make relevant modification and improvements to the system for further experiment and evaluation. This work has focused on e-learning development in view of short-term needs to improve job performance. In the workplace setting, learning needs should be addressed to enhance personal and career development in the long term. Ongoing learning is now a necessity for most employees and essential for those engaged in transitions across work and occupational boundaries. Our future work will look into long-term needs of workplace learning by integrating economic, social, and personal dimensions, and adopting human resource management and organizational learning perspectives. Further investigations will focus on developing and sustaining employees occupational competence to support individuals personal and professional advancement throughout their working life. Acknowledgements This research is supported by a UGC GRF Grant (No ) from the Hong Kong SAR Government and two Seeding Funds for Basic Research (No and No ) from The University of Hong Kong. References Baker, T. (1995). Key performance indicators manual: a practical guide for the best practice development, implementation and use of KPIs, South Melbourne, Vic.: Pitman Publishing. Bertolino, A. (2001). Chapter 5 - Software Testing. In Abran, A. and Moore, J.W. (Eds), Swebok: Guide to the Software Engineering Body of Knowledge: Trial Version 1.00, IEEE. Brink, B., Munro, J., & Osborne, M. (2002). Using online learning technology in an SME work-based setting. Educational Technology & Society, 5 (2), Chiu, D.K.W., Choi, S.P.W., Wang, M., Kafeza, E. (2008). Towards Ubiquitous Communication Support for Distance Education with Alert Management. Educational Technology & Society, 11(2), Collin, K. (2006). Connecting work and learning: design engineers learning at work. Journal of Workplace Learning, 18(7/8), Easterby-Smith, M., Araujo, L. & Burgoyne, J. (1999). Organizational learning and the learning organization: developments in theory and practice, London: Sage Publications. Ghidini, C., Rospocher, M., Serafini, L., Faatz, A., Kump, B., Ley, T., Pammer, V., & Lindstaedt, S. (2008). Collaborative enterprise integrated modeling. Proceedings of the 5th Workshop on Semantic Web Applications and Perspectives (SWAP). Horrocks, I., Patel-Schneider, P.F., Bechhofer, S., & Tsarkov, D. (2005). OWL rules: A proposal and prototype implementation. Web Semantics: Science, Services and Agents on the World Wide Web, 3(1), Illeris, K. (2003). Workplace learning and learning theory. The Journal of Workplace Learning, 15(4), Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating Training Programs (Third Edition ed.), San Francisco, CA: Berrett- Koehler Publishers, Inc. Knight, C., Gašević, D. & Richards, G. (2006). An ontology-based framework for bridging learning design and learning content. Educational Technology & Society, 9(1), Knowles, M.S., Holton, E.F. & Swanson, R.A. (1998). The Adult Learner: the definitive classic in adult education and human resource development, Houston: Gulf Publishing. 178

13 Korossy, K. (1997). Extending the theory of knowledge spaces: A competence-performance approach. Zeitschrift für Psychologie, 205, Ley, T., & Albert, D. (2003). Identifying employee competencies in dynamic work domains: methodological considerations and a case study. Journal of Universal Computer Science, 9(12), Mahmood, A.K. & Ferneley, E. (2006). Embodied agents in e-learning environments: an exploratory case study. Journal of Interactive Learning Research, 17(2), Merriam, S. (2001). Androgeny and self-directed learning: Pillars of adult learning theory. In Merriam, S. (Ed.), The new update on adult learning theory, San Francisco: Jossey-Bass, Nonaka, I. & Takeuchi, H. (1995). The Knowledge-Creating Company: how Japanese companies create the dynamics of innovation, New York: Oxford University Press. Noy, N.F., Fergerson, R.W. & Musen, M.A. (2000). The knowledge model of Protege-2000: Combining interoperability and flexibility. 2 nd International Conference on Knowledge Engineering and Knowledge Management (EKAW'2000), France. Paquette, G., Teja, I., Léonard, M., Lundgren-Cayrol, K., & Marino, O. (2006). An Instructional Engineering Method and Tool for the Design of Units of Learning. In Koper, R. & Tattersall, C. (Eds), Learning Design - A Handbook on Modelling and Delivering Networked Education and Training, Springer Berlin Heidelberg, Parmenter, D. (2007). Key performance indicators (KPI): developing, implementing, and using winning KPIs, Hoboken, N.J.: J. Wiley. Raelin, J.A. (1998). Work-based learning in practice. Journal of Workplace Learning, 10(6/7), Ran, W., Wang, M., & Law, N. (2008). Develop a workplace e-learning environment by using key performance Indicator. Proceedings of International Conference on e-learning in the Workplace (ICELW), New York. Ran, W. & Wang, M. (2008). Develop adaptive workplace e-learning environments by using performance measurement systems. Proceedings of International Conference on Enterprise Information Systems (ICEIS), Barcelona. Rosenberg, M.J. (2006). Beyond e-learning: approaches and technologies to enhance organizational knowledge, learning, and performance, San Francisco: Pfeiffer. Sambrook, S. (2003). E-learning in small organizations. Education + Training, 45(8/9), Seirafi, K. (2008). Learning performance management portal, retrieved August 10, 2010, from Servage, L. (2005). Strategizing for workplace e-learning: some critical considerations. The journal of workplace learning, 17(5/6), Sirin, E., Parsia, B. & Grau., B.C. (2007). Pellet: A practical OWL-DL reasoner. Web Semantics, 5(2), Slizyte, A. & Bakanauskiene, I. (2007). Designing performance measurement system in organization. Organizacij Vadyba: Sisteminiai Tyrimai, 43, Smith, P.J. & Sadler-Smith, E. (2006). Learning in organizations: complexities and diversities, London; New York: Routledge. Stojanovic, L., Staab, S. & Studer, R. (2001). elearning based on the SemanticWeb. Paper presented at the World Conference on the WWW and the Internet, Orland, Florida, USA Sun, P.C., Tsai, R.J., Finger, G., Chen, Y.Y. & Yeh, D., (2008). What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), Tynjälä, P. & Häkkinen, P. (2005). E-learning at work: theoretical underpinnings and pedagogical challenges. The Journal of Workplace Learning, 17(5/6), Vassileva, J. (1995). Dynamic Courseware Generation: at the Cross Point of CAL, ITS and Authoring, Proceedings of ICCE'95 International Conference on Computers, Singapore, Wang, M. & Yang, S.J.H. (2009). Editorial: Knowledge Management and E-Learning. Knowledge Management & E-Learning: An International Journal (KM&EL), 1(1), 1-5. Wang, M. (in press). Integrating organizational, social, and individual perspectives in Web 2.0-based workplace e-learning. Information Systems Frontiers, in press, DOI /s y Xu, D., Wang, H. & Wang, M. (2005). A conceptual model of personalized virtual learning environments. Expert Systems with Applications, 29(3), Zimmermann, V. & Fredrich, H. (2007). PROLEARN framework for process-oriented learning and knowledge work, retrieved August 10, 2010, from 179