A MODELING PROPOSAL FOR PERFORMANCE MEASUREMENT IN E- COMMERCE

Transcription

1 in the World Economic Scenario. Belo Horizonte, Brazil, 04 to 07 October 2011 A MODELING PROPOSAL FOR PERFORMANCE MEASUREMENT IN E- COMMERCE Rafael Tezza (UFSC) rafaeltezza@yahoo.com.br Antonio Cezar Bornia (UFSC) cezar@inf.ufsc.br Andrea Cristina Trierweiller (UFSC) andreatri@gmail.com Blenio Cesar Severo Peixe (UFPR) blenio@ufpr.br Andreas Dittmar Weise (UFSM) mail@adweise.de The aim of this paper is to propose a systematic to evaluate results in e- commerce B2C, analyzing tools, techniques and procedures that help auditing of the modeling, focusing on the client s satisfaction. In this work, the quality of inforrmation dimension is focused, adopting an evaluation methodology from the area of quality of service. First, a review of literature on the evaluation of performance in e-commerce service as well as in information systems (IS) is carried out, together with some analysis tools. Such a review of literature is used as the theoretical framework for a modeling proposal in which the elements of evaluation in e-commerce B2C are identified. Next, such elements are deeply studied in order to be better understood. Such a study is carried out using the development of a modeling process followed by a structuring and a control proposal.the investigation of such a structure makes the factors have direct action over the client s satisfaction in e- commerce B2C, generating a proposal based on the evaluation of performance. The structure helps the B2C companies to understand the involved elements in the interaction with the consumer and may identify improvement opportunities. The main contribution of this paper is in the articulation of a state-of-the-art methodology based on procedures that put the construction of knowledge over the matter first. A useful theoretical structure for the identification of important elements that directly or indirectly influence the client s satisfaction was created. Palavras-chaves: performance measurement, e-commerce, information systems, quality in service

2 1. Introduction The performance evaluation of a system has the objective of facilitating the comprehension of structures and relationship among the elements involved as one of its main functions. Following this line of thought, performance evaluation aims to construct mechanisms to control and manage conflicts and to promote constant improvement. So, in order to be efficient, a system of performance evaluation must take into account the interactions of the global context. Such an aspect fits into several systems, including e-commerce B2C, which is commerce using electronic means between an organization and the client. To become successful and to keep competitiveness in the B2C market, an organization needs both good products and marketing; however it also needs excellence in its processes, bearing the culture of constant improvement with excellence in mind (DUFFY; DALE, 2002). To reinforce such a fact, a more recent review of literature on e-commerce demonstrates the growing interest in the quality of online sales service approach (CAI; JUN, 2003). Such an interest is justified due to the fact that the quality of the online service is related to the offer for better services (YANG; JUN, 2002) and has a positive influence over the client s satisfaction, attraction, retention and gains (CAI; JUN, 2003; COX; DALE, 2001; GURAU, 2003). The Information systems (IS) literature presents a broad quantity of studies aiming to measure the final user s satisfaction (DOLL; TORKZADEH, 1988; DeLONE; McLEAN, 1992; TORKZADEH; LEE, 2003); however little attention has been given to web systems (HUANG et al., 2004; ABDINNOUR-HELM et al., 2005). The main contribution of this paper is in the articulation of a state-of-the-art methodology based on procedures that put the construction of knowledge over the matter first. A useful theoretical structure for the identification of important elements that directly or indirectly influence the client s satisfaction was created. 2. Performance Measurement in e-commerce According to Cullen and Webster (2007), the Office for National Statistics (ONS) has been adopted in the business area, using the term e-commerce for the use of electronic nets to make requests (using all kinds of nets measured by computers) and not only as a channel for payments and deliveries. Therefore, the bases for e-commerce are said to be the information systems; consequently, the evaluation of performance in e-commerce is closely-related to the evaluation of information systems. Based on the mathematical theory of communication Shannon e Weaver (1949), the system of communication was classified in three hierarchical levels: technical level, semantic level and efficiency level or influence. The technical level regards the acuteness to transmit the communication symbols. The semantic dimension regards the preciseness to transmit such symbols concerning the expected meaning. Finally, the efficiency level is related to the impact of such a communication regarding the behavior of the individuals involved in the process. 2

3 Using the works of Shannon and Weaver (1949), Mason (1978), DeLone and McLean (1992) and Myers, Kappelman and Prybutok (1997) as models, an enlargement of such dimensions is proposed, focusing on the evaluation of the information systems. The authors propose the following dimensions: - Quality of the system, representing the technical level; - Quality of the information and quality of the service, representing the semantic level; - Use of the system, user s satisfaction, individual impact, impact at the group work as well as organizational impact, representing the efficiency level or influence. According to the authors, such dimensions, either individually or grouped together, influence the use of the system as well as the client s satisfaction. Yet, according to Myers, Kappelman and Prybutok (1997), the use frequency may affect the user s degree of satisfaction and viceversa, either positively or negatively. The use of system together with the user s satisfaction are primordial factors of the individual impact. Such an impact on the individual performance may influence the performance of the group work as well as the organizational environment. The impact on the group work may also cause changes in the organizational environment. Such dimensions and their interactions can be visualized in Figure 1. Service Quality Use Workgroup Impact System Quality Individual Impact Information Quality User Satisfaction Figure 1 Dimensions of the information systems and its relationship. Source: Myers, Kappelman and Prybutok (1997) Organizational Impact In the quality dimension of the system, Mason (1997) describes the system of information production as a group of operations that receives and processes data according to some precise and systematic rules and procedures. DeLone and McLean (1992) has presented the following characteristics in this dimension: accessibility, flexibility, integration, timeresponse, reliability, ease of use and usefulness. The user s satisfaction dimension is the most used measurement to evaluate the information systems. In this dimension, the following factors are identified, among others: satisfaction regarding specific characteristics, total satisfaction, satisfaction regarding the information, satisfaction regarding the software, satisfaction regarding decision making (DeLONE; McLEAN, 1992). One of the first studies carried out focusing on the evaluation of the user s satisfaction regarding computer services was Bailey and Pearson (1983), which has identified 3

4 a series of factors that affect the users satisfaction of information systems and, via the eight empirical studies, 39 significant factors have been found regarding the final user s satisfaction. Ives et al (1983) has simplified and improved the Bailey and Pearson s instrument. It has been applied to a sample of production managers from manufacture organizations of the United States. Six factors with low correlation were taken out of the instrument, leaving only 13 original factors, being 5 factors related to the product information systems, 5 factors related to quality and services and 3 factors related to user s involvement. The most used and validated instrument over the last years was designed by Doll and Torkzadeh (1988), which has started with 38 items taken from the literature and reduced to 12, being distributed in 5 constructs, postulated as: content, preciseness, form, use facility and punctuality of the system. Chin and Lee (2000) presents an instrument for the client s satisfaction measurement made of three dimensions: satisfaction, expectation and desire. Although the authors use the instrument from Doll and Torkzadeh (1988) as basis for research, they criticize it, saying that such an instrument cannot measure the clients satisfaction of IT properly, and also that some constructs are conceptually incompatible. Therefore, the theorists propose a model, pinpointing that the feelings of the final users satisfaction is increased when it matches their perceptions of evaluation of the discrepancy level regarding desires and expectations. Parasuraman et al. (1985) and Grönroos (1994) deals with the evaluation of the quality of service. At first, the SERVQUAL scale, composed of five dimensions tangible elements, reliability, response capacity, safety capacity and empathy is proposed, besides 22 attributes. According to Grönroos (1994), the quality noticed by the client has three dimensions: - One technical dimension, or result dimension; - one functional dimension, or process dimension; and - one image dimension, which represents how the client sees the organization, and which is influenced by the functional and the technical dimensions. The technical dimension refers to the result of a good service offered to the client. Such a dimension can normally be evaluated objectively by the client. On the other hand, the functional dimension of the process is focused on the way the client receives the service and his/her experiences with either the process or consume. The client s evaluation in this dimension is more subjective than in the technical dimension. Alzola and Robaina (2007) proposes a theoretical model to measure results in e-commerce B2C based on the studies in Parasuraman et al. (1985) and Grönroos (1994) as well as in interviews with specialists and with e-commerce clients. Such a model is comprised of three dimensions: reliability (the affective getting of result), value (the cost-benefit relation) and guarantee (post-sale), besides ten attributes. This paper presents a new model to evaluate results in e-commerce based on the evaluation of information systems and on the evaluation of quality of service. 4

5 3. Modeling proposal The attempt was to integrate the whole process seen from the e-commerce B2C final client s point of view as well as seen from the evaluations of result in information systems and quality in service. According to Bititci (1995), the analysis of information systems and the project of techniques and models for the information systems are guided by three basic points of view: - Process point of view modeling of the information processing; - Data point of view modeling of the information of the structure; and - Control point of view modeling of the information dynamics. Such points of view, when properly explored in a system, be it either exclusively from information systems, organizational or from the service scope which is the objective in this paper, make it possible to have a broad view of the process. Based on the exploration of these points of view as well as on the adaptation to e-commerce B2C, the following steps to model a system of performance measurement in e-commerce B2C are proposed: - Identification of evaluation elements in e-commerce B2C; - Development of the modeling process; - Development of the modeling structure; - Development of the modeling control. 3.1 Identification of the evaluation elements in e-commerce B2C The identification of the evaluation elements in e-commerce B2C aims to promote the client s satisfaction, taking into account the supplier-client relationship and the basic concepts of quality in service. In order to identify such elements, it is important to know the client, to understand their needs and to satisfy such needs. After defining the requisites for each client, the measurement that can possibly offer either objective or subjective information over the capacity of e-commerce organizations to satisfy the clients needs is made in order to construct the model proposed here. The needs are translated here as being clients requisites based on the literature as well as on interviews with users and specialists. Examples of some of the requisites together with their indicators can be seen in Chart 1. Requisite Cost Accessibility Reliability Indicator % Quantity of products on sale % of extra taxes for products Price variation in relation to other organizations Site loading time in slow access % of accessible content for vision-impaired users % of accessible content for users with hearing disabilities % of alternative content to users that do not have access to multimedia % of product delivery failures % of clients complaints regarding either delivery or the product 5

6 Usability Customization Lead-time Average time of product guarantee % of unavailable products in stock Response time to complaints Success rate for the use of the search tool % of clients complaints regarding failures on the site Non-conformity rate in a heuristic usability quiz Quantity of personalized products Quantity of clients who have a profile form on the site. Figure 2 Clients requisites and measurements of such requisites. 3.2 Development of the modeling process The quality of a model, just like its contribution to the point in question, depends on the tools applied and on the clear understanding among the elements and actors involved in the whole system. So, this phase of development of the process consists of the construction of understanding of the elements involved in the client s satisfaction in e-commerce B2C. The first phase was related to the identification of the concepts involved in the matter as well as to their hierarchization, as can be seen in Figure 3. Client s satisfaction in e-commerce B2C Perceptions of a financial Reliability Accessibility Usability Customization Figure 3 Hierarchization of the concepts involved in the client s satisfaction in e-commerce B2C. From such an identification of the concepts involved in the process of the client s satisfaction in e-commerce B2C, a deeper investigation of each concept was started in order to know the relationship of the concepts with the client s satisfaction and, consequently, what its importance to modeling is. To do so, conceptual maps were designed for each one of the concepts. The conceptual map aims to present the subject s concepts considered to be relevant to the understanding of the main concept and the relationship between them in a graphic way. Moreira (1997) states that there is not an accurate map for a determined concept, but there must be coherence and correction in the understanding of the relationships. Thus, the map seems to be an important instrument for the investigation of meanings, differences, individual or collective links established and relevance of relationships. Figure 4 presents the example of the conceptual map for the usability concept. 6

7 Mental Models Usability Visual Systems Physical Systems affect generate affect related Human-Computer Interaction affect Learning Memorization Efficiency Quality Aims to Through Simplification Reduction of errors Through As a consequence generate Prioritization of useful information It facilitates the use generate Best Sites affect Facilitates Find the desired product affect fidelization buying decision Figure 4 Conceptual map - Usability The conceptual map in Figure 4 presents the major part of the subconcepts and relationships involved in the usability concept, just like its ends, represented here by the buying decision and/or by fidelization. In the development of such a structure, it is possible to identify subconcepts, such as the Human-Computer Interaction and the mental models that have a strong influence on the usability concept. From the performance measurement point of view, such a phase of the proposal represents a broader view of each requisite, in a strategic way; in other words, taking into account its final objectives. 7

8 3.3 Development of the structure of the model The development of the structure of the model follows the same sequence of the structure of the process discussed in the previous section, where there was an understanding over the elements that constitute the process of the client s satisfaction. In this item, each element, now, appropriately contextualized in the process and understood as its integrating part, is analyzed individually. The objective is to determine the factors that may contribute to the measurement of the result in a general view of the client s satisfaction. In order to achieve such an objective, the use of the cause-effect tool is proposed, as shown in Figure 5. The cause-effect diagram shows how each element cause have an impact on the result of the process of the client s satisfaction, clearly showing the direct and indirect causes. The usability requisite may interfere directly on the result of the client s satisfaction, and such interference may be analyzed, for example, as shown in Table 1, by the success rate in the use of the search tool. The search is an important user s experience on the Web. Users have great expectations on how the searcher might work. For Nielsen and Loranger (2006), a search system on a specific site must work better that on a search system on the web. Figure 4 shows the cause-effect diagram for the success rate on the use of the searcher, where it is possible to see a structure for the measurement of result in a more tactical and operational level. Therefore, the application of the cause-effect diagram offers a useful approach for the analysis from the tactile to the operational level of the relationship that interfere in the strategic result (client s satisfaction), helping the structuring of the model. 8

9 Orthograph y Quantity of Products Software Size and shape of letters Meaning of words typed Summary writing Expected patterns User s profile Synonyms Idiom Orthograph s error Updating Items updating Search type Quantity of products Link size Search Engine Optimization (SEO) Basic Advanced Parametric Settings Type of product Number of users Links size Type of product Search type Capacity Performance Reliability of the system Expected patterns Success rate for the use of the search tool Mental models Relevance classification Size Research box Identificatio n of search button Shape User s Experience Rearch Result Interface Figure 5 Cause-effect diagram: success rate on the use of the searcher. In the cause-effect diagram illustrated in Figure 5, it is possible to see that the graphic interface used by an e-commerce organization may affect its performance, influencing the success rate in the use of the search tool. Consequently, the factors listed in the arrows above as research box, identification of search button, type of product expected patterns, link size may change the graphic interface directly and, so, influence the success rate in the use of the searcher. So, while the graphic interface, search result, user s experience, orthography, quantity of products and software may be classified as tactile measurements, the causes listed in each of them (size and shape of letters, meaning of words typed, summary writing, items updating, search type, expected patterns, user s profile, mental models, updating, relevance classification, reliability of the system, number of users) constitute the basis for the operational measurements. 3.4 Development of the modeling control According to Bititci (1995), control is a model of interaction between states and events of a system in real time. The system responds to external events and passes through a series of modes or states. Such a definition matches practically all types of systems, including the system proposed in this paper. However, the modeling of these interactions and events in a system is a quite complex issue, involving several events. The suggested proposed here is a 9

10 general view of some possible relationships over the process of client s satisfaction in e- commerce B2C, following the requisites and metrics presented in the previous items. The analysis of such relationships may help the identification of properties and flaws in the functional level as well as how they can influence the strategic results. Figure 6 presents part of such relationships. Potential client Not satisfied client No Client with some needs? Yes Good accessibility Client has difficulties either cultural or physical Customization Good usability Client did not find what being looked for Value evaluation Reliability Client notices the low costbenefit relation Problem with the security system Satisfied client Figure 6 Flow chart of event transition Figure 6 simulates a sequence of events that start from a client with certain needs, going through several interactions with the e-commerce system until s/he arrives at the condition of satisfied or not satisfied client. As can be observed there can be problem in the system directly affecting the interaction with the client and, consequently, affecting the client s satisfaction. For example, the first possibility of failure is in the lack of access, in case the client considers this requisite totally necessary. It is also possible to observe that there are two key-points in this process: usability and reliability of the system. The first, as shown previously, is directly related to online operations and opens a range of possibilities for interaction with the user (both positive and negative), representing one delicate and important stage of the client s satisfaction process. Regarding reliability, it is more linked to the offline part of the process, but it is not less fundamental. The issue of value judgment is related, in this case, to the product costs as a whole (product cost + commission + taxes). According to Alzola and Robaina (2007), an e-commerce site is occasionally rejected by the client, either because the delivery expenses exceed the product cost, or because there are significant differences in terms of commission regarding other sites. The price and commission are the negative part, being necessary to offer a balance represented by the positive part of the offer. 10

11 Clients requisites Importance for the client % Quantity of products on sale % of extra taxes for products Price variation in relation to other organizations Site loading time in slow access % of accessible content for vision-impaired users % of accessible content for users with hearing disabilities % alternatives content Average time of product guarantee % of unavailable products in stock % of clients complaints regarding either delivery or the product Response time to complaints Success rate for the use of the search tool % of clients complaints regarding failures on the site Non-conformity rate in a heuristic usability quiz Lead-time % of unavailable products in stock Quantity of personalized products Quantity of clients who have a profile form on the site Relative Importance Total XVII INTERNATIONAL CONFERENCE ON INDUSTRIAL Customization comes as a positive differential to conquer a client. According to Page and Lepkowska-White (2002), the customization of a site offers a positive experience to the client, positively interfering in the image of the site. Obviously, the flow chart represents the process from a macro point of view, admitting, however, more details for each of the requisites and interactions. Features Cost Accessibility Usability Reliability Customization final price time payment 4 12 access for users with disabilities 2 58 access alternatives 2 50 ease of use foreseeing failure Logística post-sale 3 57 in safe transactions 5 45 customized environment 2 38 personalized product 2 22 Strong (6) Middle (3) Weak (1) Figure 7 Example of control through a QFD matrix In order to create a sequence of more specified and precise relationships, it is necessary to have good knowledge and comprehension of such interactions between the internal measurements in each requisite, and in between requisites. To help the specifying and comprehension, the use of the tool Quality Function Deployment (QFD) is proposed, making the creation of a view of the process of control possible, detailing the way how the measurements influence each other. According to Bititci (1995), the QFD tool offers an approach oriented to the process and describes the dynamic perspective of a system of performance measurement, being easy as well as simple to understand. The technique of QFD is a systematic procedure to define the clients needs and to interpret the product functionally, as well as the characteristics of the process. Figure 7 illustrates how an oriented QFD matrix 11

12 can be developed. In the upper part, the quality characteristics are presented and, in the inferior column, the clients requisites are presented. In the matrix, the relationship level between such characteristics is classified as not existing, weak, medium and strong, with its respective weights: zero, one, three and six. When the weights are multiplied by the given importance for the client to the respective requisites, they generate the degree of relative importance seen in the last column. This tool allows the identification of properties between several levels of measurement in an e-commerce system, also allowing the priority of performance measurement strategies in the environment, according to the proposed objectives. 4. Conclusion The e-commerce B2C has significantly grown, representing an important area for study and improvement, as for example, the success factors in this time of operation directly related to the quality and satisfaction from the client s perspective. In this context, this paper has dealt with the matter of performance measurement, aiming at the final client s satisfaction. To do so, basic concepts concerning performance measurement in service and in information systems were discussed, making a parallel with the e-commerce area. From this point on, a modeling proposal was developed in order to measure performance in e-commerce B2C. The main contribution of this paper is in the articulation of a state-of-the-art methodology based on procedures that put the construction of knowledge over the matter first. A useful theoretical structure for the identification of important elements that directly or indirectly influence the client s satisfaction was created. The construction of such a structure made use not only of concepts, but also of tools and analysis techniques that could indicate the following points: - The main phase of a model is the identification and the perfect understanding of the elements involved in the process, just like the type of client involved, allowing the definition of requisites for each client and the elaboration of performance indicators; - Some tools, like conceptual maps, cause-effect diagrams and QFD may contribute to the understanding of the interactions in a system of performance measurement, identifying critical phases of the process as a whole and/or separately; - Having a clear knowledge of the elements and interactions in the process of the client s satisfaction B2C, it is possible to identify important elements and possible flaws in the process through a flow chart of transition of events, avoiding the generation of client loss for lack of access, bad usability, low reliability, among others. Finally, it is possible to conclude that modeling system of performance measurement in e- commerce B2C consists mainly of a deep knowledge of the elements involved and their interaction in the process in the operational, tactile and strategic levels. The use of some management tools may significantly contribute to the identification and design of a solid and integrate performance measurement. 12

13 References ABDINNOUR-HELM, S. F.; CHAPARRO, B. S. & FARMER, S. M. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction with a Web Site. Decision Sciences. Vol. 36, n.2, 2005 p ALZOLA, L. M. & ROBAINA, V. P. Measuring the results in B2C e-commerce. International Journal of Quality & Reliability Management Vol. 24 No. 3, 2007 pp BAILEY, J. E. & PEARSON, S. W. Development of a Tool for Measuring and Analyzing Computer User Satisfaction. Management Science. Vol.. 29, n. 5, 1983 p BITITCI, U. S. Modelling of performance measurement systems in manufacturing enterprises. International Journal of Production Economics, v.42, 1995, pp CAI, S. & JUN, J. Internet users perceptions of online service quality: a comparison of online buyers and information searchers, Managing Service Quality, Vol.13 No. 6, 2003, pp CHIN, W. W. & LEE, M. K. O. On The Formation of End-User Computing Satisfaction: A Proposed Model and Measurement Instrument. International Conference on Information Systems. Proceedings of the twenty first international conference on Information systems. Brisbane, p COX, J. & DALE, B.G. Service quality and e-commerce: an exploratory analysis, Managing Service Quality, Vol. 11 No. 2, 2001, pp CULLEN, A. J. & WEBSTER, M.. A model of B2B e-commerce, based on connectivity and purpose. International Journal of Operation & Production Management, Vol. 27 No. 2, 2007, pp DELONE, W.H. & MCLEAN,E.R. Information System Success: The Quest for the dependent variable, Information Research, Vol 3 No 1, 1992, p DOLL, W. J. & TORKZADEH, G. The measurement of end-user computing satisfaction. MIS Quarterly. Vol. 12, n. 2, 1988, p DUFFY, G. & DALE, B.G. E-commerce processes: a study of criticality, Industrial Management & Data Systems, Vol. 102 No. 8, 2002, pp GRÖNROOS, C. Marketing y gestión de servicios (Services Marketing and Management), Díaz de Santos, Madrid, GURAU, C. Tailoring e-service quality through CRM, Managing Service Quality, Vol. 13 No. 6, 2003, pp HUANG, J. H.; YANG, C.; JIN, B. H. & CHIU, H. Measuring satisfaction with business-toemployee systems. Computers in Human Behavior. Vol. 20, 2004, p IVES, B.; OLSON, M. H. & BAROULDI, J. J. The measurement of user information satisfaction. Communications of the ACM. Vol. 26, n. 10, 1983, p MASON, R. O. Measuring information output: a communication systems approach. Information & Management, v.1, 1978, p MYERS, B. L., KAPPELMAN, L. A. & PRYBUTOK, V. R. A comprehensive model for assessing the quality and productivity of the information systems function: toward a theory for information systems assessment. Information Resources Management Journal, vol.10, n , p MOREIRA, M. A. Aprendizagem Significativa : um conceito subjacente. In: Encuentro Internacional sobre el Aprendizaje Significativo, Burgos, Espanha, 15 a 19 de set. de NIELSEN, J. & LORANGER, H. Prioritizing Web Usability. California:New Riders,California, PAGE, C. & LEPKOWSKA-WHITE, E. Web equity: a framework for building consumer value in online 13

14 companies, Journal of Consumer Marketing, Vol. 19 No. 3, 2002, pp PARASURAMAN, A., ZEITHAML, V. A. & BERRY, L. L. A conceptual model of service quality and its implications for future research. Journal of Marketing, v.49, 1985 p SHANNON, C. E. & WEAVER, W.. The mathematical theory of communication. Urbana: University of Illinois Press, TORKZADEH, G. & LEE, J. Measures of perceived end-user computing skills. Information & Management. Vol. 40, 2003, p YANG, Z. & JUN, M. Consumer perception of e-service quality: from internet purchaser and non-purchaser perspectives, Journal of Business Strategies, Vol. 19 No. 1, 2002, pp