Assessing the Success of ICT s from a User Perspective: Case Study of Coffee Research Foundation, Kenya Michael W. Makokha 1 & Daniel O. Ochieng 1 1 School of Computing and Informatics, University of Nairobi, Nairobi, Kenya Correspondence: Michael W. Makokha, School of Computing and Informatics, University of Nairobi, Nairobi, Kenya. E-mail: wambwere@gmail.com Received: August 26, 2014 Accepted: September 27, 2014 Online Published: October 16, 2014 doi:10.5430/jms.v5n4p46 URL: http://dx.doi.org/10.5430/jms.v5n4p46 Abstract The study was conducted to validate the application of DeLone & McLean s Information System (IS) success model (2003) in a local setup in Kenya and to evaluate the success of an enterprise resource planning (ERP) system from a user perspective. It was carried out at Coffee Research Foundation (CRF) in Kenya. A number of past studies to measure the success of information systems in different settings were reviewed leading to the choice and use of the updated DeLone & McLean IS success model in this study. The research involved the use of questionnaires as well as interviews and focus group discussion (FGD). A number of hypotheses were formulated and tested and the results indicated that the updated DeLone & McLean s IS success model was valid as a useful model for this particular study. The study also indicated that the variables system quality, information quality, and service quality stood out as critical determinants of how information and communication technology systems can be used to improve organizational performance. Keywords: ERP system, DeLone & McLean IS success model, system quality, information quality, service quality, use, user satisfaction, net benefits 1. Introduction Before 1990, organizations in most countries across the world used information technology (IT) in a very restricted manner (Bryson & Daniels, 2007). At the time, the use of information and communication technologies (ICT s) for business operations was considered to be costly rather than a way of presenting organizations with a new approach to business. As a result of increased technological advancements, however, things took a different turn and the use of ICT s became an important consideration for most organizations (Al-adaileh, 2009). A number of factors including increased global activities and the evolution of a digital environment have greatly contributed to the increased use of ICT s by organizations. As a result of the above changes, many organizations are compelled to spend heavily on ICT s in order to improve performance and increase efficiency (Rao, 2006; May et al., 2007). However, the initial excitement quickly disappears when it becomes almost impossible to determine the role and exact benefits associated with the use of ICT s. Consequently, it is critical for any organization to assess the contribution of ICT systems to performance. Considering that the use of ICT s is aimed at enhancing the users ability to work better and produce more, it is imperative to measure and assess the success of ICT s from the users perspective. Arguably, the success of ICT s depends more on users than on technology (Akman et al., 2005). 2. Literature Review 2.1 Evaluating the Success of ICT s In general, assessing the success of ICT s makes it possible to align usage with business objectives and increase the success rate. Drawing from a study by Alshawi and Alalwany (2009), traditional approaches to the evaluation of ICT systems such as return on investments, cost-benefit analysis, payback period, and present worth have been criticized for targeting only direct tangible costs and benefits. Effective assessment presents organizations with an opportunity to learn, to reduce the level of uncertainty about ICT investments, and to gain investor confidence. Published by Sciedu Press 46 ISSN 1923-3965 E-ISSN 1923-3973
2.2 IS Success Models By and large, measuring the success of ICT systems is very important if their value and contribution to users and organizational performance is to be understood. Although various IS success models have been used in the past, DeLone & McLean s IS success model was chosen given that it has been tested and used in various scenarios over the years. 2.3 DeLone & McLean IS Success Model Following the publication of DeLone and McLean s work, their original IS success model received several criticisms. DeLone and McLean later reviewed studies done from 1992 and revised their original model to produce an updated model as illustrated by Figure 1. Figure 1. DeLone & McLean IS success model (2003) Source: DeLone & McLean, 2003 In the updated DeLone and McLean IS success model, service quality was included while individual and organizational impacts were replaced with net benefits to account for benefits at multiple levels of analysis and to allow researchers to apply the model to any intended level of analysis (Petter et al., 2008). The updated IS success model also indicates that information quality, system quality, and service quality can positively or negatively affect intention to use/use as well as user satisfaction. In turn, intention to use/use and user satisfaction can positively or negatively affect net benefits. 2.4 Adoption of DeLone & McLean s IS Success Model As pointed out earlier, an important objective of this study was to test and validate the updated DeLone & McLean s IS success model in a local setup in Kenya. Consequently, the study was based on the updated DeLone & McLean IS success model. The research model is illustrated by Figure 2. Figure 2. Research model Source: Adopted from DeLone & McLean (2003) Published by Sciedu Press 47 ISSN 1923-3965 E-ISSN 1923-3973
The choice of the construct use rather than intention to use was informed by the fact that system usage in this study was voluntary. 3. Methodology The study used both quantitative and qualitative techniques and included the use of numerical methods as well as statistical tools for data collection and analysis. Primary data was collected based on relevant criteria and measurements developed from literature review while secondary data was accessed from different sources including text books, journals, and company documents. In addition, online ICT evaluation studies were used as a source of secondary data. Primary data was gathered using questionnaires, interviews, and focus group discussion. A non-probabilistic sampling approach was used to determine the sample size for the study. In order to establish the relationship between the various model variables and to validate the research model, Pearson s correlation analysis was used. Ethical considerations were observed prior to, during, and after data collection. 3.1 Sampling A suitable sample was chosen from CRF employees based at the organization s headquarters. Although respondents were deliberately chosen by the researchers, reliability was upheld by remaining impartial and exercising due diligence throughout the sampling process. Out of 200 questionnaires distributed, 174 were received back, representing a response rate of 87%. The returned questionnaires were later subjected to careful scrutiny and the various responses were then coded based on the different items on the questionnaires. 3.2 Reliability of Instruments To ensure the reliability of questionnaires, experts were consulted to determine their suitability. The questionnaires were also pre-tested by distributing them to some respondents and making revisions based on the feedback received. Finally, Cronbach s alpha was used to measure the internal consistency of the model constructs. Since acceptable Cronbach s alpha values range from 0.7 to 0.95 (Tavakol, 2011), all constructs exhibited an acceptable level of reliability as shown in Table 1. Table 1. Reliability analysis of constructs Source: Research 3.3 Hypotheses Formulation The following hypotheses were formulated to test and validate the research model. System Quality (SysQ) Concerning this construct, we hypothesized as follows: H1a: System Quality positively affects Use H1b: System Quality positively affects User Satisfaction Information Quality (IQ) The following hypotheses were formulated for this variable: H2a: Information Quality is positively associated with Use H2b: Information Quality is positively associated with User Satisfaction Published by Sciedu Press 48 ISSN 1923-3965 E-ISSN 1923-3973
Service Quality (SrvQ) Regarding this construct, we hypothesized as follows: H3a: Service Quality is positively associated with Use H3b: Service Quality is positively associated with User Satisfaction Use (U) The following hypotheses were formulated for this construct: H4a: Use is positively associated with User Satisfaction H4b: Use is positively associated with Net Benefits User Satisfaction (UsrS) Concerning user satisfaction, the following hypotheses were formulated: H5a: User satisfaction is positively associated with Use H5b: User satisfaction is positively associated with Net Benefits The integration of the above hypotheses into the research model is shown by Figure 3. Figure 3. Research model and hypotheses Source: Adopted from DeLone & McLean IS Success Model (2003) 4. Analysis and Results Data was analyzed using statistical tools that included the use of tables and analysis techniques. Pearson s correlation was used to determine the presence of correlation among the various constructs. The overall results of the analysis are presented in Table 2. Published by Sciedu Press 49 ISSN 1923-3965 E-ISSN 1923-3973
Table 2. Correlation analysis of variables Source: Research In general, the results showed that there were significant correlations among the model constructs. However, the greatest correlations were between system quality & user satisfaction, service quality & user satisfaction, and between use & user satisfaction. Based on the results, service quality has a considerable influence on the success of an information system followed by system quality and use in that order. Seemingly, service quality had a greater significance since users were more concerned about the quality of the services delivered by the ERP system. Similarly, system quality and ability to effectively use the ERP system had a significant influence given that users expressed confidence in having a reliable system that is at the same time easy to work with. 4.1 Hypotheses and Model Validation Although some of the hypotheses were more significant than others, they all were significant nevertheless. A hypothesis was accepted as long as the p-value was less than 0.05. The validation results are discussed as follows: H1a: System Quality is positively associated with Use The Pearson s correlation coefficient value between system quality and use which supports our hypothesis was 0.427. There was thus a strong positive association between system quality and the use of the ERP system. A p-value of 0.001 further indicated the presence of a very significant correlation. This is in agreement with the findings of DeLone & McLean (1992), Seddon (1997), and Rai et al. (2002). The first hypothesis concerning the construct system quality was therefore supported. H1b: System Quality is positively associated with User Satisfaction Pearson s correlation coefficient between system quality and user satisfaction was 0.709 with a p-value of 0.000 indicating that the correlation between these two variables was a very significant one. The second hypothesis concerning the construct system quality was thus supported. H2a: Information Quality is positively associated with Use Pearson s correlation coefficient between information quality and use was 0.279. Although the association was a weaker one, a p-value of 0.041 indicated that the said correlation was significant. This hypothesis was thus supported. Published by Sciedu Press 50 ISSN 1923-3965 E-ISSN 1923-3973
H2b: Information Quality is positively associated with User Satisfaction Pearson s correlation coefficient between information quality and user satisfaction was 0.308 implying a positive association between information quality and user satisfaction. The association was, however, a weak one but nonetheless significant as the p-value obtained was 0.023. This hypothesis was equally supported. H3a: Service Quality is positively associated with Use Pearson s correlation coefficient between service quality and use was 0.291 with a p-value of 0.033 which indicated the presence of a weak but significant correlation between the two constructs. This hypothesis was thus supported. H3b: Service Quality is positively associated with User Satisfaction A strong positive association existed between service quality and user satisfaction. Pearson s correlation coefficient of 0.761 and a p-value of 0.000 indicated that there was a positive and very significant correlation between service quality and user satisfaction. Consequently, this hypothesis was supported. H4a: Use is positively associated with User Satisfaction Pearson s correlation coefficient between use and user satisfaction was 0.548 affirming that there was a strong positive association between use and user satisfaction. A p-value of 0.000 also indicated that the correlation was very significant. Accordingly, this hypothesis was supported. H4b: Use is positively associated with Net Benefits Pearson s correlation coefficient between use and net benefits was 0.398 and confirmed that there was a positive correlation between use and net benefits. A p-value of 0.003 indicated that the correlation was a very significant one. This hypothesis was therefore supported. H5a: User Satisfaction is positively associated with Use Pearson s correlation coefficient between user satisfaction and use was found to be 0.548 indicating that there was a positive association between user satisfaction and use. A p-value of 0.000 also indicated that the correlation was a very significant one. This hypothesis was thus supported. H5b: User Satisfaction is positively associated with Net Benefits Pearson s correlation coefficient between user satisfaction and net benefits was 0.406 indicating that there was a positive correlation between user satisfaction and net benefits. A p-value of 0.002 indicated that the correlation was a very significant one. Consequently, the hypothesis was supported. A summary of the hypotheses test results is shown by Table 3. Table 3. Summary of hypotheses test results Source: Research Published by Sciedu Press 51 ISSN 1923-3965 E-ISSN 1923-3973
All the stated hypotheses were supported indicating that the updated DeLone and McLean IS success model was valid in the given local setup in Kenya. 4.2 Triangulation of Quantitative and Qualitative Analysis As earlier noted, the study used both quantitative and qualitative analysis. Focus group discussion (FGD) was used for qualitative analysis. The design of the FGD guide was based on the structure of the questionnaire and borrowed heavily from the items used to measure the different constructs. The results of the FGD offered a useful confirmation of the outcome of the quantitative analysis. A part from information quality, service quality, and user satisfaction where the results differed slightly, the outcomes on system quality and use were similar. FGD results also indicated that the updated DeLone and McLean IS success model was valid as a useful model for the study. The research model together with correlation coefficients is represented by Figure 4. Figure 4. Research model with hypotheses and correlation coefficients Source: Adopted from the updated DeLone & McLean IS Success Model 5. Conclusion and Recommendations Drawing from the discussion presented in this paper, all the hypotheses tested were found to be significant. Accordingly, the updated DeLone and McLean IS success model proved to be a valid model for this study. However, seven out of the ten hypotheses tested appeared to be more significant. Based on the findings of the study, users at CRF expressed satisfaction with the use of the ERP system in their work operations. By and large, users agreed with the various statements used to measure the different constructs. Despite the fact that the study was carried out in one organization in Kenya, it represented to a very wide extent, attributes that made the model worth using for the study. Since 2009, for example, CRF has been using an ERP system with financial, payroll, human resource, transport, and procurement modules. This is in addition to other applications used for specific purposes. The decision by the organization to continue investing in information and communication technologies is mainly motivated by the desire to improve the level of employee productivity throughout the organization. Given similar characteristics therefore, the updated DeLone and McLean IS success model can be used to assess how successful information systems in an organization are. However, since DeLone and McLean did not factor in the effects of age, gender, and experience, it may be necessary to determine how the final results may be affected by the inclusion of these variables. Published by Sciedu Press 52 ISSN 1923-3965 E-ISSN 1923-3973
References Al-adaileh, R. M. (2009). An Evaluation of Information Systems Success: A User Perspective - the Case of Jordan Telecom Group. European Journal of Scientific Research, 37(2), 226 239. Alshawi, S., & Alalwany, H. (2009). E-Government Evaluation: Citizen s Perspective in Developing Countries. Information Technology for Development, 15(3), 193-208. http://dx.doi.org/10.1002/itdj.20125 Bryson, J. R., & Daniels, P. W. (2007). The Handbook of Service Industries. Northampton, Massachusetts: Edward Elgar Publishing Limited. DeLone, W. H., & McLean, E. R. (1992). Information Systems Success: The Quest for the Dependent Variable. Information Systems Research, 3(1), 60 95. http://dx.doi.org/10.1287/isre.3.1.60 DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, 19(4), 9 30. May, J., Karugia, J., & Ndokweni, M. (2007). Information and Communication Technologies and Agricultural Development in Sub-Saharan Africa: Transformation and Employment Generation. Final Framework Paper prepared for the African Economic Research Consortium (AERC). Petter, S., DeLone, W., & McLean, E. (2008). Measuring Information Systems Success: Models, Dimensions, Measures, and Interrelationships. European Journal of Information Systems, 17, 236 263. http://dx.doi.org/10.1057/ejis.2008.15 Rai, A., Lang, S. S., & Welker, R. B. (2002). Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis. Information Systems Research, 13(1), 50-69. http://dx.doi.org/10.1287/isre.13.1.50.96 Rao, M. (2006). Measuring the Success of Information Systems: Case Study of a University Student Services System in Southwest Decision Sciences Institute. Proceedings of a Conference, Southwest US, pp. 408 418. Seddon, P. B. (1997). A Respecification and Extension of the Delone and Mclean Model of IS Success. Information Systems Research, 8(3), 240 253. http://dx.doi.org/10.1287/isre.8.3.240 Tavakol, M. (2011). Making Sense of Cronbach s Alpha. International Journal of Medical Education, 2, 53 55. http://dx.doi.org/10.5116/ijme.4dfb.8dfd Published by Sciedu Press 53 ISSN 1923-3965 E-ISSN 1923-3973