System Usability Scale for Usability Testing of Agriculture E-Commerce Website

Transcription

1 DOI : /rwjasc/v10i4/06 DOI URL : System Usability Scale for Usability Testing of Agriculture E-Commerce Website Joko Purwadi, Department of Informatics, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia. Rosa Delima, Department of Informatics, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia. Argo Wibowo, Department of Information System, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia. Novia Indrayono Rante Toding, Department of Informatics, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia. Halim Budi Santoso, Department of Information System, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia. (Received September 21, 2019; Accepted October 25, 2019) ABSTRACT A software development project should pass many stages starting from defining needs to implementation. Before the process of system implementation, the development team will conduct a usability test and a user acceptance test for the application. This stage is very important to ensure the implementation process can be done well by minimizing problems when users interact with the system. In this study, usability and user acceptance tests will be carried out on the sale and purchase application of agricultural products (agricultural e- commerce). The method used in the study consists of 6 main stages, such as preliminary studies, participant studies, tool development, test implementation, data analysis, and result formulation. The tools to be used are such as a questionnaire for participants data collection, a System Usability Scale (SUS) questionnaire to measure the application acceptance and task scenarios to measure system usability. This article specifically discusses the design of task scenarios and the data taking process on the usability test on an agricultural e-commerce system. Keywords: Agriculture E-Commerce; Usability Test, User Acceptance, System Usability Scale; SUS. INTRODUCTION: The field of agriculture in Indonesia continues to experience growth both in terms of agricultural technology and information technology that supports agriculture. Various systems have been developed to support increased productivity and expansion of agricultural product markets both in Indonesia and in the world. Some applications that have been developed in Asian and African countries are e-choupal (India), Agricultural Market Information System (Bangladesh), and Infotrade (Uganda) (Delima, Santoso, & Purwadi, Kajian Aplikasi Pertanian yang Dikembangkan di Beberapa Negara Asia dan Afrika, 2016). Meanwhile, in Indonesia several information applications have been developed in agriculture (Delima & Purwadi, Analisis Situs Web Pertanian Berbahasa Indonesia, 2015). One of the E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [43]

2 agricultural applications that has been developed is a system for sales and puschasing agricultural product (Delima, Santoso, Andriyanto, & Wibowo, 2018). The Agricultural Sales and Purchasing System is a system developed to support economic development and market expansion, particularly for farmers and people in rural areas. The system was completed in Before the system is introduced to the users, one step is needed, such as testing the user acceptance of the application. A test aimed at determining the level of user acceptance e-commerce application. This test is very important to minimize failures in the system implementation process. The test is carried out through the SUS questionnaire. The paper consists of five parts. Part I is an introduction, literature review can be seen in Part II, while the methodology and design of test equipment can be seen in Parts III and IV, and Part V is a conclusion and future work. LITERATURE REVIEW System Usability Scale (SUS): System Usability Scale (SUS) is a questionnaire developed by John Brooke in 1986 that was used to evaluate the usability level of Blackboard (Brooke, 1996). The SUS questionnaire consists of 10 questions, with odd sequences for positive items and even sequences for negative items. Each question is graded on a 5 scales from Strongly Disagree to Strongly Agree. The ten list of questions on SUS can be seen in Table 1. No Table 1: Question in SUS (Brooke, 1996) Questions 1. I Think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. To measure the application s level of acceptance using the SUS questionnaire, we can use a measurement developed by (Bangor, Kortum, & Miller, 2009) as shown in Table 2. Through the value of SUS, the level of application s acceptance is evaluated from the reusability aspect. Table 2: Application Acceptance Rates on the SUS Questionnaire (Bangor, Kortum, & Miller, 2009) Nilai SUS Interpretation < 50 Not Acceptable Marginal > 70 Acceptable SUS questionnaire has been widely used in the process of system reusability testing. Some studies using the SUS questionnaire are (Puspita, Sibarani, Muslim, & Moch., 2016) (Baharrudin, Wardani, & Herlambang, 2018) (Devy, Wibirama, & Santosa, 2017) (Devy, Wibirama, & Santosa, 2017), and (Al- Omar, 2018). E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [44]

3 E-Commerce: In today s global era, E-commerce is a promising new business model. This business model utilizes information and communication technology (ICT) support. According to Turban et al (Turban, King, Lee, Liang, & Turban, 2015), electronic commerce (abbreviated as EC, or E-commerce) is a trading model that includes the process of buying, selling, transferring, and exchanging products, services or information through computer networks, including the internet. E-commerce business is one of the most important parts of the economy in Indonesia. According to (Gentur, 2014) The Ministry of Trade of the Republic of Indonesia estimated that the sale and purchase of goods transactions via the internet (E-commerce) from Indonesia will penetrate the number of US $ 10,08 billion. The average value of online shopping transactions is growing 40 percent every year. Further, according to Agus Tjandra in the News (Berita, 2014) as vice chairman & foreign relations of the Indonesian E-commerce Association (idea), on the signing of E-commerce cooperation between the Indonesian Information Technology Federation (FTII) and the Taipei Computer Association (TCA) in Jakarta stated that Indonesia s total E-commerce transaction value in 2013 reached US $ 8 billion or around Rp 94 trillion and is predicted to continue to increase to reach US$ 24 billion or around Rp 283 trillion in The number of people who shop online in the country is recorded at 4,6 million people during 2013 and is expected to increase to 8,7 million in E-commerce also provides many benefits for both entrepreneurs and customers. Some of the benefits of e-commerce according to (Khan, 2016) are such as increasing entrepreneur income, reducing company s operating costs, saving time to do shopping for consumers, and having a certain level of convenience in shopping. Dodolantani: An Agriculture E-Commerce: Agricultural Sales and Purchasing System is a system developed to support economic development and market expansion for farmers in particular and people in rural areas (Delima, Santoso, Andriyanto, & Wibowo, 2018). Sales and Purchases are two processes of trading goods and services business. At present the business model of trading in Information and Communication Technology (ICT) is known as e-commerce. E- Commerce is a trading model including the sale, purchase and exchange of goods and services through computer networks and the internet. E-Commerce has several advantages for both sellers and buyers, such as e-commerce can reduce operational costs and buyers can save time in shopping. Dodolantani agricultural e-commerce has 2 main modules namely the purchase module and sales module. The purchase module functions to manage the product s purchasing process, this module consists of the user s login page (Figure 1). There are 2 types of logins, such as login as a buy er and login as an admin.to be able to login as buyer, can be done by selecting the register button on the right side of the login button. After log in, the buyer will see the product page (Figure 2). This page shows the latest and most popular products sold by the seller. Through the product page, buyers can see, search and discuss the products they want to buy. After being sure to make a purchase transaction, the user can continue to make purchases through the buy button on the product details (Figure 3.). Then the buyer can make a purchase transaction (Figure 4). Purchase transactions include determining the purchase amount, purchase address, and shipping expedition. After making a product purchase, the buyer will then be on the shopping cart page (Figure 5) and the purchase can be proceeded to the payment process. After making payment the buyer must upload the proof of payment. After the buyer makes a payment, the system admin must verify the payment made by the user. Verification can be done via the validation page (Figure 6). If the payment does not match, the system admin can send a notification to the buyer regarding the payment incompatibility. If the payment is correct and validated, the process continues with sending goods via the specified courier. The purchase module sitemap can be seen in Figure 7 (Delima, Santoso, Andriyanto, & Wibowo, 2018). E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [45]

4 Figure 1: User Login Page Figure 2: Page E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [46]

5 Figure 3: Detail Page Figure 4: Shopping Page E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [47]

6 Figure 5: Shopping Cart Page Figure 6: Payment Validation Page E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [48]

7 Figure 7: Site Map Purchase Module in Dodolantani Purchase Module Home Registration List Checking Purchasing Payment Report Profile Equipment Substance Purchase Data Entry Discussion Upload Payment Receipt Sales Transection Sales Report Agriculture Comment Entry The sales module in dodolantani e-commerce functions to support the seller in conducting goods sales transactions. The sales module of agriculture is divided into 2 menu features such as the seller display which consists of the login page, the seller main page, product listing page, product added form page, product offer form page, product listing page, product price check page, sales page, inbox page, seller report page, profile page. Next, there is admin display which consists of admin main page, product category list page, product list page, product offer list page, seller list page, seller validation page, seller data page, product validation page, transaction data page, payment validation page and admin report page. While on the product listing page, the details of product offerings, sales transactions, product discussions, and each sale reports can be seen in Figure 8 until Figure 12 and Site map of the sales module can be seen in Figure 13 (Delima, Santoso, Aditya, Purwadi, & Wibowo, 2018). Figure 8: List Page E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [49]

8 Figure 9: Detail Sales Page Figure 10: List of Sales Transactions Page E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [50]

9 Figure 11: Discussion Page Figure 12: Sales Report Page. E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [51]

10 Figure 13: Site Map Sales Module. Sales Module Home Registration List Checking Selling Inbox Entry Report Profit Equipment Sales List Information Discussion Equipment Sales Transection Substance Agriculture Order Checking Status Review Substance Agriculture Sales Report RESEARCH METHODOLOGY: This research will apply a testing method which consists of six stages, namely a preliminary study, participant/respondent study, test tool development, implementation, data analysis, and result formulation. The order of the stages of the research can be seen in Figure 14. Figure 14: Research Methodology Scheme. Preliminary Study Perticipants Study Develop Tools Conduct Testing Data Analysis Results The Preliminary Study is carried out to make plan, identify needs, and the availability of research resources. The test would be conducted in May to July Data collection will be carried out directly with agricultural stakeholders including farmers, sellers of agricultural products, and farmer groups. The data are collected in the Central Java and Yogyakarta area during May to July Researchers have cooperation with Farmer Group in these region. The targeted respondents consist of two groups of respondents such as respondents who are end users such as buyers and sellers and respondents who are administrators of the system. The total number of respondents is targeted at around 35 respondents. Criteria for buyer and seller respondents are a minimum age of 20 years, a minimum of junior high school graduate, and able to operate a computer. The questionnaire used to test user s acceptance is the SUS questionnaire and to support the data retrieval process, the Google form is used. To measure the application s level of acceptance using the SUS questionnaire, we use acceptance rates as shown in table II. ANALYSIS AND DISCUSSION: SUS Questionnaire: Questionnaire is one of the data collection tools in form of a series of questions or statements that would be submitted to the respondents. The questionnaire is based on and refers to the SUS (System Usability Scale) Questionnaire, which is a questionnaire consisting of 10 statements (Brooke, 1996) (Lewis & Sauro, 2017). Google Form for Support Usability Testing: To support the process of data retrieval during usability test, Google form is used for data collection. There are 2 forms developed such as a form to get the respondent s characteristics and a form for SUS questionnaire. A form to get the respondent s characteristics consists of questions that include personal identity (name, address, gender, age, education), farming experience, experience in using computer or mobile phone, experience related to social media and internet, and experience in conducting online transactions. Meanwhile the SUS questionnaire was made using a Likert scale from 1 to 5, with 1 for the category of strongly disagree and 5 for strongly agree. E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [52]

11 Data Collection: The test was conducted from May to July 2019, before conducting the test a brief interview was conducted to explore the respondents experience in the use of laptops and smartphones. This process is carried out to ensure that the respondents are qualified as required. If the respondent meets the next requirements, then the researchers gives an explanation of the purpose of data retrieval and provide a brief explanation related to the system to be tested. Respondents were asked to fill out the agreement form. Then the respondents are given time to try to run the system independently. If the time is considered as enough, then the test data would be taken to complete the task in accordance with the scenario that had been prepared. After finishing it, the process continued with filling SUS questionnaire to find out the level of users acceptance. Participants Characteristics: Respondents in this study are 35 persons, with a distribution of 30 respondents who were buyers or sellers and five respondents were admin users. The respondents came from Bantul and Salatiga. The characteristics of the respondents can be seen in Table 3. Table 3: Participants Characteristics. Variable Class Total Percentage years 9 26% Age years 12 34% years 12 34% > 50 years 2 6% Gender Male 16 46% Female 19 54% Junior High School 5 14% Senior High School 1 3% Education Level Diploma % Under Graduate/ Bachelor 13 37% Post Graduate 1 3% Based on the data in Table 3, it is known that the majority of respondents aged between 31 to 50 years old reached 78%. Based on gender, respondents of the research were quite balanced even though the percentage of women is more than men, but the difference is not too significant. Based on the level of education, the largest percentage was undergraduate education at 37%. The graph of the respondents characteristics can be seen in Figure 15 to Figure 17. Figure 15: Graph based on Participants Age Respondent Distribution Based on Age thn thn thn > 50 thn E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [53]

12 Figure 16: Graph based on Participants Gender. Women 54% Men 46% Men Women Figure 17: Graph based on Participants Education Respondents Education Level Junior High School Senior High School Non Degree Diploma Qualification Undergraduate Graduate Result: System Usability Scale Questionnaire consists of 10 questions as stated on Table I. There are two groups of respondents. The first group is buyer and seller. The second group is System Administrator. Questionnaire uses 5 Likert Scale which has following criteria as mentioned on Table 4. Table 4: Likert Scale for SUS Questionnaire. Skala Categories 1 Very disagree 2 Disagree 3 Neutral 4 Agree 5 Very Agree Table 4 describe likert scale for System Usability Scale Questionnaire (SUS Questionnaire). Respondent should answer the questionnaire using this scale. After respondent answer the questionnaire, the next step is to calculate the value of SUS using normalization formula (1) and (2), where SUSValue i,j is defined as a value of question i from respondent j and RVji is respondent j s value for question i. SUSvalue ij = RV ji 1 for i is an odd questions (1) SUSvalue ij = 5 RV ji for i is an even question (2) Acceptance value is got by using formula (3), where j is a respondent and n is the number of respondents. 10 i 1 x 2.5) Acceptance Value = j=1 ( SUSvalue ij n E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [54] n (3)

13 User acceptance level is interpreted as acceptance value and value interpretation as shown in Table 3. Based on SUS Questionnaire, Table 5 describes the response from buyer and seller. While Table 6 describes the response from system administrator. Table 5: Result for Buyers and Seller Respondents. R SUS Questions Total Total x Table 6: Result for Admin System Respondents. R SUS Questions Total Total x E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [55]

14 As mentioned before, Table 5 shows the result for buyer and seller respondents and Table 6 shows the result for System Administrator respondents. After calculating using formula (1) and (2), we got acceptance value for buyer and seller is 57.8 and acceptance value for system administrator is Based on those acceptance rate from Table 2, this acceptance value can be categorized as marginal acceptable. Furthermore, there are some improvement should be done to the system in order to increase the acceptance value. We also conducted another analysis to test average value for each question in SUS Questionnaire. During the design phase, we divided 10 questions in SUS Questionnaire into two groups: Questions which give positive contributions and questions which give negative contributions. Positive contribution question exists for the odd number questions and negative contribution question exists for the even number questions. After have some normalization using formula (1) and formula (2), we gain value range from 0 to 4 with interpretation as shown on Table 7. Table 7: Data Range and Interpretation SUS Quistionnaire. Data Value Interpretation Impact Neutral Table 7 shows data range and interpretation of SUS Questionnaire. Based on the value range, we conducted calculation analysis to calculate average response for each questions. The result is shown on Table 8. SUS Questions Table 8: Average Value for SUS Value. Average Value Buyer and Saller Respondents Admin System Respondents Table 8 shows the average value for SUS Value. From this analysis, it is found that 90% of each average value above neutral but this value has not reached positive impact value to the acceptance value of the application. This result supports the first finding that there are some improvements should be conducted in order to satisfy user and increase user acceptance value. CONCLUSION: From the analysis, it can be concluded that user acceptance level for agriculture e-commerce dodolantani on middle acceptable level. This result is based on SUS Questionnaires that are distributed to the users. It means that some system improvement should be done to satisfy user acceptance level. For future work, it needs some user task test in order to get the detail feature that should be improved. Through usability testing, researchers are expected to find some specific feature that needs to be improved and revised. E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [56]

15 ACKNOWLEDGMENT: Special thanks to Research and Community Services Institution and Faculty of Information Technology Universitas Kristen Duta Wacana for providing facilities and funding for publishing this article. REFERENCES: Al-Omar, K. (2018). Evaluating the Usability and Learnability of the "Blackboard" LMS Using SUS and Data Mining, 2nd Conference on Computing Methodologies and Communication (ICCMC) (pp ). Erode, India: IEEE. Baharrudin, M., Wardani, N. H., & Herlambang, A. D. (2018). Analisis Usability Pada Sistem Informasi FILKOM Apps Fakultas Ilmu Komputer Universitas Brawijaya, Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, Vol 2 No. 10, Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean : Adding an adjective rating scale, Usability Stud, Vol. 4, No. 3, pp Berita, S. (2014, September Rabu). 2016, Potensi Bisnis E-Commerce Indonesia Mencapai Rp 283 Triliun. Indonesia: Berita Satu. Retrieved from Berita Satu: Brooke, J. (1996). SUS-A quick and dirty usability scale Industrial usability evaluation. 189(194): p Delima, R., & Purwadi, J. (2015). Analisis Situs Web Pertanian Berbahasa Indonesia, Seminar Nasional Komputer dan Informatika Terapan (pp. 1-5). Jember: Politeknik Negeri Jember. Delima, R., Santoso, H. B., & Purwadi, J. (2016). Kajian Aplikasi Pertanian yang Dikembangkan di Beberapa Negara Asia dan Afrika, Seminar Nasional Aplikasi Teknologi Informasi. 2016, pp. pp. B19-B26. Yogyakarta: Universitas Islam Indonesia. Delima, R., Santoso, H. B., Aditya, G. H., Purwadi, J., & Wibowo, A. (2018). Development of Sales Modules for Agricultural E-Commerce Using Dynamic Syatem Development Method, International Journal of New Media Technology (IJNMT), Delima, R., Santoso, H. B., Andriyanto, N., & Wibowo, A. (2018). Development of Purchasing Module for Agriculture E-Commerce using Dynamic System Development, International Journal of Advanced Computer Science and Application, Devy, N. I., Wibirama, S., & Santosa, P. I. (2017). Evaluating Eser Experience of English Learning Interace using User Experience Questionnaire and System Usability Scale, 1st International Conference on Informatics and Computational Science (ICICoS) (pp ). Semarang, Indonesia: IEEE. Devy, N. R., Wibirama, S., & Santosa, P. I. (2017). Evaluating user experience of english learning interface using User Experience Questionnaire and System Usability Scale, 1st International Conference on Informatics and Computational Sciences (ICICoS). Semarang, Indonesia. Gentur, P. J. (2014, September Selasa). Transaksi e-commerce Indonesia Capai US$ 10 Miliar, Retrieved from CNN INDONESIA: /transaksi-e-commerce-indonesia-capai-us--10-miliar/ Khan, A. G. (2016). Electronic Commerce: A Study on Benefits and Challenges in an Emerging Economy, Global Journal of Management and Business Research: Beconomics and Commerce, 3-4. Lewis, J. R., & Sauro, J. (2017). Can I Leave This One Out? The Effect of Dropping an Item From the SUS, Journal of Usability Studies, 13(1), Puspita, M. A., Sibarani, T. M., Muslim, E., & Moch., B. N. (2016). Perancangan Ulang User Interface Mesin Parkir Meter Jakarta, Jurnal Ergonomi dan K3, Vol. 1 No. 1, Turban, E., King, D., Lee, J., Liang, T., & Turban, D. (2015). Electronic Commerce : A Managerial and Social Network Perspective, Cham, Switzerland: Springer International Publishing AG. Retrieved from E-commerce: E-ISSN: ISSN: Vol. X, Issue 4, October 2019 [57]