Software Quality Characteristics Tested For Mobile Application Development

Transcription

1 Thesis no: MGSE Software Quality Characteristics Tested For Mobile Application Developent Literature Review and Epirical Survey WALEED ANWAR Faculty of Coputing Blekinge Institute of Technology SE Karlskrona Sweden

2 This thesis is subitted to the Faculty of Coputing at Blekinge Institute of Technology in partial fulfillent of the requireents for the degree of Master of Science in Software Engineering. The thesis is equivalent to 10 weeks of full tie studies. Contact Inforation: Author(s): WALEED ANWAR E-ail: University advisor: Dr. Sion Poulding Departent of Software Engineering Faculty of Coputing Internet : Blekinge Institute of Technology Phone : SE Karlskrona, Sweden Fax :

3 ABSTRACT Context. Sart phones use is increasing day by day as there is large nuber of app users. Due to ore use of apps, the testing of obile application should be done correctly and flawlessly to ensure the effectiveness of obile applications. Objectives. The objective of this research is to find out the iportant obile application quality characteristics fro developer s perspective and how developers actually test for the. Apart fro that how the developers test their obile applications are also addressed. Methods. Two ethodologies were used: the literature survey and the epirical survey. The literature survey was used to get failiar with the ost coonly known obile application quality characteristics for which obile applications are tested for. The epirical survey was used to get data fro developers by sending an online questionnaire link to the Google Play store developers and their response was recorded and further evaluated to present results. I categorized apps as top rated apps and low rated apps. Results. I got 43 responses as a whole fro 1000 developers. I got 20 responses out of 500 fro the top rated apps developers and 23 responses out of 500 fro the low rated apps developers. The results were used to copare the quality characteristics iportance to testing. According to the responses fro both top rated and low rated apps developer, functional testing and usability testing is considered to be the ost iportant and are ostly tested for Android obile applications. Load testing and energy testing is considered to be not that iportant fro developer s point of view but energy testing and eory testing is not so often tested for Android obile applications by developers. When it coes to testing process, top rated and low rated apps developer perfor testing by creating test cases that are predefined in scripts or docuents. They prefer to execute their tests on eulator rather than on obile and run their tests autoatically via tool or service as a cloud. Conclusion. My study reports the quality characteristics iportance for which the obile applications are tested for fro developer s perspective, actual testing done by developers and the testing process used by the. Functional testing and usability testing are considered to be ost iportant and ostly tested quality characteristics according to developer s perspective. Testing is done by running their tests on eulator autoatically through soe tool. They prefer to create their test cases that are predefined in scripts. Keywords: Mobile Application Testing, Quality Characteristics 3

4 ACKNOWLEDGEMENT I would like to extend y gratitude to y supervisor Sion Poulding for his vital support and feedback till end. This thesis would not have been possible without Sion Poulding guidance throughout the research. I really feel honored working with hi. I would like to thank y o, siblings and friends who helped e and supported e. Finally, I sincerely appreciate all the participants of the web questionnaire fro all over the world because without their response y thesis would not be accoplished. 4

5 TABLE OF CONTENTS ABSTRACT... 3 ACKNOWLEDGEMENT... 4 TABLE OF CONTENTS... 5 LIST OF FIGURES... 7 LIST OF TABLES INTRODUCTION AIMS AND OBJECTIVES RELATED WORK METHODOLOGY: RESEARCH QUESTIONS LITERATURE REVIEW EXTENDED DESIGN AND DERIVING THE START SET OF PAPERS: EMPIRICAL SURVEY PREPARATION OF QUESTIONNAIRE RESEARCH DESIGN AND DATA COLLECTION: RESULTS TOP RATED APPS DEVELOPER S FEEDBACK IMPORTANCE OF QUALITY CHARACTERISTICS TESTING OF QUALITY CHARACTERISTICS TESTING PROCESS OF TOP RATED APPLICATIONS LOW RATED APPS DEVELOPER S FEEDBACK IMPORTANCE OF QUALITY CHARACTERISTICS TESTING OF QUALITY CHARACTERISTICS TESTING PROCESS OF LOW RATED APPLICATIONS ANALYSIS DATA ANALYSIS TOP RATED APPS IMPORTANCE BY TOP RATED DEVELOPERS TESTING BY TOP RATED DEVELOPERS TESTING PROCESS OF TOP RATED APPS LOW RATED APPS

6 4.3.1 IMPORTANCE BY LOW RATED DEVELOPERS TESTING LEVEL BY LOW RATED DEVELOPERS TESTING PROCESS BY LOW RATED DEVELOPERS COMPARISON OF QUALITY CHARACTERISTICS IMPORTANCE, TESTING AND THEIR TESTING PROCESS DISCUSSION CONCLUSION VALIDITY FUTURE WORK REFERENCES APPENDIX A: QUESTIONNAIRE APPENDIX B: TOP RATED DEVELOPER S INFORMATION: APPENDIX C: LOW RATED DEVELOPER S INFORMATION:

7 LIST OF FIGURES Figure 1: Coparison of iportant quality characteristics fro top rated developer s perspective..26 Figure 2: Coparison of testing level of quality characteristics fro top rated developer s perspective Figure 3: Coparison of iportant quality characteristics fro low rated developer s perspective 30 Figure 4: Coparison of testing level of quality characteristics fro low rated developer s perspective

8 LIST OF TABLES Table 1: Research Questions Table 2: Quality characteristics iportance by top rated apps developers Table 3: Quality characteristics testing by top rated apps developers Table 4: Testing perfored by top rated developers Table 5: Quality characteristics iportance by low rated apps developers Table 6: Quality characteristics testing by low rated apps developers Table 7: Testing perfored by low rated apps developers Table 8: Analysis for the iportance of quality characteristics fro top Rated apps developer s perspective Table 9: Analysis for the testing level of quality characteristics fro top rated apps developer s perspective Table 10: Analysis for the iportance of quality characteristics fro low Rated apps developer s perspective Table 11: Analysis for the testing level of quality characteristics fro low Rated apps developer s perspective

9 1 INTRODUCTION A obile application (or obile app) is a software application that is designed to run on sartphones, tablet coputers and on other obile devices [1]. Mobile applications are becoing popular and also very coplex. As obile application users expect the applications to be reliable and fault free, so it akes the duty for application testers to ensure the quality of obile applications before it is released in arket [2]. Testing is a technique used to identify the faults in applications and to satisfy their custoers by providing the highly reliable and error free applications [27]. Testing of obile application is extreely difficult because there is a lot of diversity in obile devices, their runtie environent and the resources needed to test a obile application differs fro obile phone to tablet [28]. According to Mobile Application Testing Tutorial published by the IEEE Coputer Society [7], obile application testing is different fro the conventional software testing because of the following several unique requireents. 1. Mobile applications ust be able to work correctly anytie and anywhere. 2. As obile applications are often developed for a set of targeted devices, these applications ust work properly on different platfors having different operating systes, screen sizes, processing power, network bandwidth and battery life. 3. Mobile applications ust also include ultiple input channels (e.g. keyboard, voice, and gestures), ultiedia support, and other enhanced usability features in order to provide the rich experience as expected by the obile device users. 4. As obile devices support a range of wireless networks (e.g. 2G, 3G, 4G, Wi-Fi, WiMAX), obile applications ust also able to work in different network connectivity environent. Considering these unique requireents for obile applications and to be fully aware of iportant obile application quality characteristics, I was interested in using the research ethod literature survey and epirical survey. The literature survey was carried out to know about the well-known quality characteristics tested for obile applications. The quality characteristics is defined as what characteristics should an application have for exaple application should function properly, application should be easy to use for users, application should be copatible with different devices etc. The epirical survey was perfored to find out which quality characteristics are iportant fro developer s perspective and how often those quality characteristics are tested for Android obile applications by developers. Apart fro that I was also interested in perforing epirical survey on the testing process used by developers for testing of Android obile applications. In epirical surveys, inforation is gathered by asking people questions [8]. Afterwards, those quality characteristics were copared according to respondent s answers which helped e in finding out which quality characteristics were iportant fro their point of view and how they actually use the. Moreover, how the testing is perfored was also figured out. The epirical survey is the ethod used to collect data fro developers. An online questionnaire was ade through Google For and sent to 1000 developers and their response was recorded and further evaluated to present results. The apps whose rating was 4 or above was categorized as top rated apps and the apps whose rating was below 4 cae into the category of low rated apps. The developer s inforation is given in Appendix B and Appendix C. I contacted 500 developers of top rated apps and 500 developers of low rated apps to balance y findings and it was also tie 9

10 consuing to find developer s inforation anually. The apps were chosen randoly fro different categories available on play store i.e. top grossing apps, top selling apps, new and updated gaes, new and updated apps, apps, recoended for you apps, ovies apps, books apps, tools apps, counication apps, photography apps, offline apps etc. The developers contact inforation is given in Appendix which can be reused when soeone wants to survey again on Android obile applications. This inforation will help the to just copy the eail addresses and sent the their questionnaire as it will save tie and effort. It took e two weeks to get this inforation fro rando apps by checking its rating and storing it. This thesis contributes to find out which quality characteristics are iportant fro developer s perspective and how often these quality characteristics are tested for Android obile applications. Apart fro that it also contributes to know the current testing state of Android obile applications by addressing how the developers test their Android obile applications. The results can be used by developers to know which quality characteristics should be given high priority in testing of Android obile applications and how to actually test obile applications. 1.1 Ais and Objectives This research will be organized in two surveys (i.e. a literature survey and an epirical survey). Main ai of the literature survey is to know about well-known quality characteristics tested for obile application developent. Main objective of conducting the epirical survey is to find out which well-known quality characteristics for which obile applications are tested for is iportant fro developer s point of view and how they actually test the. Moreover, the testing process used by developers to test obile applications is also surveyed. 1.2 Related Work Mobile application testing is an activity which is aied at evaluating quality of a progra and also for iproving it, by identifying defects and probles [2]. In 2013, Android s Google Play crossed one illion available applications and fifty billion downloads [3]. Therefore due to large nuber of app users and their expectations fro the app to be error free, it is very iportant to eet the quality criteria of an application. Mobile applications growth is increasing intensively today. According to Gartner Inc. [4], in 2013, illion obile devices with Android operating syste were shipped, and in 2014 the expectation of this growth ight increase to over one billion. "Every onth, around 20,000 new applications are released, and the current nuber of apps in the Android arket is over 1,200,000 [5]". These nubers reflect that this platfor is intensively increasing day by day and that the focus of developers is to develop ore and ore obile applications. Furtherore, apps are easily available through Google Play Store so it has attracted a large nuber of developers and copanies to develop their apps and put it on Google Play Store [9]. Due to this ease there are illions of apps on this platfor but it doesn t ensure that apps are error free. The apps containing errors can significantly affect the developer s iage and ay har the users. So it is the duty of developers to test the apps before releasing to the arket to ensure the quality of apps [2]. The apps with alost no errors are ost downloaded and used by users and trust is built between user and developers [2]. 10

11 Mobile application testing is an activity which is aied at evaluating quality of a progra and also for iproving it, by identifying defects and probles [2]. It is uch ore difficult than the conventional software testing because the ipact of obile applications are saller than the desktop applications on a virtual achine. Mobile application testing has becoe a great challenge that needs iediate action to be tackled and there is no doubt that obile applications need specific quality characteristics [6]. The quality characteristics are used by obile application developers to ensure the quality of obile applications [2]. Google Play fraework akes it very easy for users to search and install the app. Therefore, soe developers think it is not necessary to test the app before releasing as it requires a lot of tie, so they wait for the users feedback if they report any proble they try to fix it and release the new version of app.[9] In y research ethodology course, I contacted 200 developers to perfor the epirical survey on quality characteristics tested for Android obile applications by developers. The developers were contacted to know about the ost quality characteristic that the obile applications are tested for. Those 200 developers are the part of these 1000 developers that are contacted. The saple size was sall and few quality characteristics were asked. But the iportance, use and testing process of Android apps still needs to be figured out. There have been any epirical studies conducted on Android obile application testing. Takala applied odel based user interface testing on Android applications and he has reported his experience [17]. Bhattacharya et al. perfored an investigation on bug reports and bug fixing ethod of Android applications [18]. Kropp and Morales exained strengths and weaknesses of instruentation fraework and positron fraework for testing obile graphical user interface applications [19]. Syer et al. studied the 15 ost coon Android applications and then copared it with 3 desktop applications [20]. McDonnell et al. studied the constancy and acceptance of APIs in Android network [21]. Maji et al. studies and characterized the failures in both Android and Sybian obile OSes [22]. Recently there was a research done that used survey ethodology, the tools were identified that are used in testing of Android obile applications by calculating their code coverage, the presence of test case and the challenges faced by developers in testing of Android obile applications [9]. Another research was done to understand the challenges faced by Android obile application developer s [23]. The quality characteristics tested for windows phone has been researched recently [9]. Different testing techniques are being introduced to overcoe the challenges faced by developers but still quality characteristics that Android developers think are iportant and are tested for Android obile applications is unknown and are being ignored. So, to overcoe this gap I was interested in perforing the epirical survey on the quality characteristics iportance and how often these quality characteristics are tested for Android obile applications. Apart fro that I was also interested in perforing epirical survey on the testing process used by developers for testing of Android obile applications. I surveyed Android obile application developers to know about current testing state of Android obile applications. The rest of the sections are organized as follows: Section 2 presents the research questions and the ethodologies used. Section 3 presents the results of an epirical survey. Section 4 presents the analysis of the results. Section 5 presents discussion section in which answers to the research questions are explained briefly and connected with RQS. Section 6 describes validity threats, conclusion and the future work needed. 11

12 2 METHODOLOGY: A ixed approach: literature review and epirical survey was used in this research to carry out the answers of research questions. The literature review was conducted to know about the wellknown quality characteristics for which obile applications are tested for fro the published work and the epirical survey was used to know the current testing state of Android obile applications. 2.1 Research questions The research questions considered in this work are as follows: Table 1: Research Questions Research Questions RQ1: What are the quality characteristics that obile applications are tested for? RQ2: Which are the ost and least iportant quality characteristics for which obile applications is tested for according to developer s perspective? RQ3: Which are the frequently and barely quality characteristics for which the obile applications is actually tested for by developers? RQ4: Does the iportance and testing of quality characteristics for which the obile application is tested for differs between top rated and low rated apps developer? RQ5: How are Android obile applications tested? Description This research question ais at finding the set of quality characteristics used to test obile applications fro the published work. This research question ais to find out the ost iportant and least iportant quality characteristics for which the obile applications is tested for according to developer s perspective. This research question ais to find out the frequently and barely quality characteristics for which the obile applications is actually tested for fro the developers perspective. This research question ais to find out whether the iportance and testing of quality characteristics for which the obile applications are tested differs between top rated and low rated apps developer. This research question ais to find out the current testing process used by developers in testing of Android obile applications. 2.2 Literature review A ixed approach: database search and snowballing approach was used to extract the papers. Soe papers were extracted fro IEEE and ACM databases by using the keywords that were derived fro research questions. The snowballing approach is then used which involves two steps: (1) deriving the start set of the papers and (2) perforing forward and backward snowballing. The research papers extracted fro IEEE and ACM databases were used as a start set to perfor snowballing. 12

13 2.2.1 Extended Design and deriving the Start set of papers: According to Wohlen, a good start set should have the following characteristics [29]. Diversity of the papers should be targeted which should cover different publishers, authors, years. To obtain this, it is recoended to cover this in start set. Size of start set depends on the size of area being studied for exaple if you are focusing on a specific area it requires lesser papers than focusing on a broader area. If search string is not forulated well too any papers can be found, To itigate it papers which are highly cited and have ore relevant references can be used by to obtain perfect start set. There can be a risk to iss relevant papers by using different terinology. To itigate it, synonys can be used with keywords that are derived fro research questions. The following search string was used to identify the start set was the following: (Testing types OR practices OR quality characteristics) And (Mobile applications) I used backward snowballing which is looking at the reference of the papers by applying inclusion and exclusion criteria. The ain advantage of using snowballing is that it just focuses on the referenced and cited papers [26]. The snowballing approach is as good as database search and easy to perfor as copared to database. I did the literature review not a systeatic literature review due to tie constraint. Systeatic review is basically in depth study of the relevant topic that identifies, selects all high quality research evidence which is relevant to that topic and to do it successfully it requires a lot of tie. Literature review is basically qualitatively reviews on a topic using general ethods to collect data. The literature review was conducted to find different quality characteristics for which the obile applications is tested. The literature review covers the answer of RQ1. Following are the different testing characteristics that the obile applications are tested for is found fro the published work. I. Usability Testing: Usability testing is used for activities that assess UI content and alerts, user operation flows and scenarios, edia richness, and gesture interaction [7] [9]. II. Perforance Testing: Perforance testing is used for the activities to deterine the speed or effectiveness of coputer, network, software progra or device [6] [7] [9] [24]. III. Reliability Testing: Reliability Testing relates to testing software s ability to function in given environental conditions for a particular aount of tie [6] [7] [24]. IV. Security Testing: Security testing includes encryption/decryption techniques to ensure the data counication, checks for ulti user support, and checks within app for the access of files [6] [7] [23] [24]. V. Meory Testing: 13

14 Meory testing is used to check how uch eory the application is taking. It is also used to check for the eory leakage, if the applications are already killed and their active processes are still taking eory [6] [24]. VI. Energy Testing: Energy testing is used to ensure that the applications don t use ore energy than required one. If ore energy is taken by single application then it will reduce the battery life of device [6] [24]. VII. Functional Testing: Functional testing is a quality assurance (QA) process and a type of black box testing that is used to check each and every function works perfectly or not. Functional testing usually describes what the syste actually does. Functional testing tests the portion of the functionality of the whole syste [6] [7] [9]. VIII. Interoperability Testing: Interoperability testing involves testing whether a given software progra or technology is copatible with others and prootes cross-use functionality [7]. IX. Copatibility Testing: Copatibility testing is part of software non-functional tests, is testing conducted on the application to evaluate the application's copatibility with the coputing environent [23] [6] [9]. X. Connectivity Testing: Connectivity testing is a testing activity designed to validate the continuity of network counications [6] [7] [23] [24]. XI. Load Testing: Load testing is the process of putting deand on a syste or device and easuring its response to check its efficiency [9]. 2.3 Epirical Survey There are any different ways to perfor a survey like street surveys, telephone survey and electronic surveys. I used the epirical survey (online survey) to gather required data because it is a suitable strategy to collect data fro ore nuber of people (ethods, tools, developers and copanies) to gain the understanding of larger population [10]. With an increasing use of internet users it is easy to perfor surveys because you can contact the developers or copanies online (i.e. online surveys) and with the availability of this technology the interest to perfor surveys is increasing day by day this. Kitchenha and Pfleeger s [16] show that the interest in doing a survey is increasing in the field of software Engineering. There is another study which is done within the Geran ViSEK project depicts the growing interest of epirical survey. My survey was based on an online questionnaire with three close ended questions and one open ended question (optional) to collect the required inforation. All the three close ended questions [11] have a fixed nuber of options that the respondent has to choose while in open ended question, the respondent is required to write soething as answer. The online questionnaire was 14

15 created on Google Fors [12] and the questionnaire link was provided to respondents via eail. There are two challenges arise in collecting the online survey: (1) keeping track of response sources and (2) preventing ultiple responses fro the sae person [25]. I filtered responses received based on subission tie [15].Since there can be ultiple subissions fro one syste due to browser or internet connectivity proble so screening for tie of subission (within a 2- inute interval) eliinated such duplicate responses. Abiguous responses were not received as our survey was liited to the Android developers of obile applications in play store. Survey responses have to be saved in the database such as Oracle or MySQL [25]. My survey was done through Google Fors so responses were saved online in Google database. 2.4 Preparation of Questionnaire Based on the results of the literature review, the questionnaire was developed.the questionnaire was then discussed with the supervisor and soe changes were ade according to the research questions. The questionnaire was then finalized and sent it to the developers for their feedback. The questionnaire was divided into three parts: The first part was related to the quality characteristics iportance fro their perspective. The second part was related to the actual testing of quality characteristics by Android obile application developers. The third part was related to what testing process is used by developers in testing of Android obile application. The questions were close ended and every question has checkboxes provided for the predefined answer because nowadays ostly people prefer to answer close ended questions [10].There was also one open ended question but that was optional in case if a developer wants to add any other inforation. The questionnaire is provided in the Appendix (A). There was a sequence in the questions so that respondents get interest in filling the survey. The questionnaire did not have a lot of questions as it is said that the ore questions you ask, the ore tie it will take to answer it and respondent will lose their interest in filling the questionnaire [10]. 2.5 Research Design and Data Collection: I divided the quality characteristic iportance into three levels (ost iportant, least iportant and not iportant) and testing of quality characteristic also in three levels (i.e. ostly used, rarely used and never used). By doing the literature review in section 2.2, I found eleven unique quality characteristics for which obile applications are tested for. I included the in the online questionnaire as close ended questions. In addition, I added an open ended question in order to get a response about any other quality characteristic that is also uniquely used by obile application developers. I decided to liit y research to only one online obile application store. So I perfored the survey on Google Play App. store developers, because according to the Sartphone OS Market Share analysis by IDC [13], Android OS has the largest OS arket share as copared to other obile operating systes. Google Play App. Store website [14] enabled us to easily find contact inforation of developers. The questionnaire was sent on 20 th May to 400 developers but I didn t get enough responses. The developers nuber was then increased fro 400 to Again the eail was sent to reaining 600 developer s (300 top rated and 300 low rated) on June15. Three reinders were given to the. First one was in the onth of May, second was in the onth of July and last reinder was given on the onth of August. The questionnaire was open fro 20 th May to 5 th Septeber. 15

16 Total 43 responses were obtained out of 1000 developers contacted of the Google Play App. Store.20 responses out of 500 were obtained fro the top rated apps developers and 23 responses out of 500 were obtained fro the low rated apps developers. All the developers contact inforation was gathered fro Google Play store. 16

17 3 RESULTS The results are categorized according to the top rated developer s response and low rated developer s response separately. The suary of each quality characteristics iportance, testing and testing process is given in the tables with the help of a pie chart. There is one box on which it is entioned any other quality characteristics iportance and its testing level. Blue color indicates the Iportant and Mostly used quality characteristic fro developer s perspective, Red color indicates the Less iportant and Rarely used quality characteristic and yellow color indicates Not iportant and Never used quality characteristics. This color schee applies to all the graphs given in the results section. The analysis on this raw results are perfored in section Top rated apps developer s feedback The results shown below are the feedback I got fro the top rated Apps developers in ters of iportance, testing of quality characteristics and the testing process used to test Android obile applications Iportance of quality characteristics The suary of each quality characteristics iportance according to the top rated developer s perspective are given below in Table 2 with the help of pie chart. Table 2: Quality characteristics iportance by top rated apps developers. 1.1 Usability Testing 1.2 Perforance Testing 1.3 Reliability Testing Iportant 16 80% Iportant 16 80% Iportant 12 60% Less Iportant 4 20% Less Iportant 3 15% Less Iportant 7 35% Not Iportant 0 0% Not Iportant 1 5% Not Iportant 1 5% 1.4 Security Testing 1.5 Meory Testing 1.6 Energy Testing Iportant 14 70% Iportant 13 65% Iportant 12 60% Less Iportant 4 20% Less Iportant 4 20% Less Iportant 3 15% Not Iportant 2 10% Not Iportant 3 15% Not Iportant 5 25% 17

18 1.7 Functional Testing 1.8 Interoperability Test 1.9 Copatibility Testing Iportant 19 95% Iportant 11 55% Iportant 12 60% Less Iportant 1 5% Less Iportant 7 35% Less Iportant 6 30% Not Iportant 0 0% Not Iportant 2 10% Not Iportant 2 10% 1.10 Connectivity Testing 1.11 Load Testing Any other quality characteristic and its iportance? Iportant 12 60% Iportant 10 50% Less Iportant 8 40% Less Iportant 6 30% Not Iportant 0 0% Not Iportant 4 20% None Testing of quality characteristics The Suary of each quality characteristics testing according to the top rated developer s perspective are given below in Table 3 with the help of pie chart. Table 3: Quality characteristics testing by top rated apps developers. 2.1 Usability Testing 2.2 Perforance Testing 2.3 Reliability Testing Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75% Rarely used 6 30% Rarely used 7 35% Rarely used 4 20% Never Used 0 0% Never Used 1 5% Never Used 1 5% 18

19 2.4 Security Testing 2.5 Meory Testing 2.6 Energy Testing Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40% Rarely used 5 25% Rarely used 7 35% Rarely used 4 20% Never Used 3 15% Never Used 4 20% Never Used 8 40% 2.7 Functional Testing 2.8 Interoperability Test 2.9 Copatibility Testing Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65% Rarely used 3 15% Rarely used 7 35% Rarely used 5 25% Never Used 0 0% Never Used 3 15% Never Used 2 10% 2.10 Connectivity Testing 2.10 Load Testing Any other quality characteristic and its testing level? Mostly Used 9 45% Mostly Used 9 45% Rarely used 7 35% Rarely used 5 25% Never Used 4 20% Never Used 6 30% None 19

20 3.1.3 Testing process of top rated applications Below in Table 4 it s been suarized how do the top rated apps developer perfor testing on Android obile applications. Table 4: Testing perfored by top rated developers. 3.2 Low rated apps developer s feedback The results shown below are the feedback I got fro the low rated Apps developers in ters of iportance, testing of quality characteristics and the testing process used to test Android obile applications. 20

21 3.2.1 Iportance of quality characteristics The Suary of each quality characteristics iportance according to the low rated developer s perspective are given below in Table 5 with the help of pie chart. Table 5: Quality characteristics iportance by low rated apps developers. 1.1 Usability Testing 1.2 Perforance Testing 1.3 Reliability Testing Iportant % Iportant 20 87% Iportant % Less Iportant 2 8.7% Less Iportant 3 13% Less Iportant % Not Iportant 0 0% Not Iportant 0 5% Not Iportant 0 0 % 1.4 Security Testing 1.5 Meory Testing 1.6 Energy Testing Iportant % Iportant % Iportant % Less Iportant 3 13% Less Iportant % Less Iportant % Not Iportant 1 4.3% Not Iportant % Not Iportant % 1.7 Functional Testing 1.8 Interoperability Test 1.9 Copatibility Testing Iportant % Iportant % Iportant % Less Iportant 1 4.3% Less Iportant % Less Iportant % Not Iportant 0 0% Not Iportant 1 4.3% Not Iportant 1 4.3% 21

22 1.10 Connectivity 1.11 Load Testing Testing Any other quality characteristic and its iportance? None Iportant % Iportant % Less Iportant % Less Iportant % Not Iportant 2 8.7% Not Iportant 1 4.3% Testing of quality characteristics The suary of each quality characteristics testing according to the low rated developer s perspective are given below in Table 6 with the help of pie chart. Table 6: Quality characteristics testing by low rated apps developers. 2.1 Usability Testing 2.2 Perforance Testing 2.3 Reliability Testing Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75% Rarely used 6 30% Rarely used 7 35% Rarely used 4 20% Never Used 0 0% Never Used 1 5% Never Used 1 5% 2.4 Security Testing 2.5 Meory Testing 2.6 Energy Testing Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40% Rarely used 5 25% Rarely used 7 35% Rarely used 4 20% Never Used 3 15% Never Used 4 20% Never Used 8 40% 22

23 2.7 Functional Testing 2.8 Interoperability Test 2.9 Copatibility Testing Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65% Rarely used 3 15% Rarely used 7 35% Rarely used 5 25% Never Used 0 0% Never Used 3 15% Never Used 2 10% 2.10 Connectivity Testing 2.10 Load Testing Any other testing characteristic and its testing level? Mostly Used 9 45% Mostly Used 9 45% Rarely used 7 35% Rarely used 5 25% Never Used 4 20% Never Used 6 30% None Testing process of low rated applications Below in Table 7 it s been suarized how do the low rated apps developer perfor testing on Android obile applications. Table 7: Testing perfored by low rated apps developers. 23

24 24

25 4 ANALYSIS 4.1 Data Analysis The purpose of data analysis was to copare the quality characteristics with respect to their iportance and how they are actually used by obile application developers. To obtain this, I assigned the points to every type of answer. I.e. I assigned 1 point to Most Iportant and Mostly used characteristic, 0.5 point to Less Iportant and Rarely used and (0) point to Not iportant and Never used. I added these points for each quality characteristic to obtain the total points for that characteristic. Moreover, to represent how the testing is perfored and the coparison of quality characteristics I decided to use Bar Chart to represent y analysis result. 4.2 Top Rated Apps Iportance by top rated developers Here the analysis is being done to understand which quality characteristics are iportant, less iportant and not iportant fro top rated developer s perspective. Table 8 represents the analysis done for the iportance of quality characteristics fro top rated apps developer s perspective. Table 8: Analysis for the iportance of quality characteristics fro top Rated apps developer s perspective. S. Quality Iportant Less Not No Characteristic (1p) Iportant iportant Total Points (0.5p) (0p) 1- Usability Testing 16*(1) = 16 4*(0.5) = 2 0*(0) = = Perforance Testing 16*(1) = 16 3*(0.5) = 1.5 1*(0) = = Reliability Testing 12*(1) = 12 7*(0.5) = 3.5 1*(0) = = Security Testing 14*(1) = 14 4*(0.5) = 2 2*(0) = = Meory Testing 13*(1) = 13 4*(0.5) = 2 3*(0) = = Energy Testing 12*(1) = 12 3*(0.5) = 1.5 5*(0) = = Functional Testing 19*(1) = 19 1*(0.5) = 0.5 0*(0) = = Interoperability 11*(1) = 11 7*(0.5) = 3.5 2*(0) = = 14.5 Testing 9- Copatibility Testing 12*(1) = 12 6*(0.5) = 3 2*(0) = = Connectivity Testing 12*(1) = 12 8*(0.5) = 4 0*(0) = = Load Testing 10*(1) = 10 6*(0.5) = 3 4*(0) = = 13 25

26 The calculation results for each quality characteristic iportance according to top rated developer s perspective can also be illustrated by using bar chart as shown in Figure 1 below: Figure 1: Coparison of iportant quality characteristics fro top rated developer s perspective. According to the survey responses received, the ost iportant quality characteristic for which obile applications are tested for aong obile application developers is functional testing and the least iportant quality characteristic is load testing as illustrated in the Figure 1. We can order these quality characteristics with respect to their iportance as following: 1. Functional Testing 2. Usability Testing 3. Perforance Testing 4. Security Testing 4. Connectivity Testing 6. Reliability Testing 7. Meory Testing 7. Copatibility Testing 9. Interoperability Testing 10. Energy Testing 11. Load Testing 26

27 4.2.2 Testing by top rated developers Here the analysis is being done to understand which quality characteristics are ostly used, rarely used and never used fro top rated developer s perspective. Table 9 represents the analysis done for the testing level of quality characteristics fro top rated apps developer s perspective. Table 9: Analysis for the testing level of quality characteristics fro top rated apps developer s perspective. S. Quality Mostly Rarely Used Never Used No Characteristic Used (0.5p) (0p) Total Points (1p) 1- Usability Testing 14*(1) = 14 6*(0.5) = 3 0*(0) = = Perforance Testing 12*(1 ) = 12 7*(0.5) = 3 1*(0) = = Reliability Testing 15*(1) = 15 4*(0.5) = 2 1*(0) = = Security Testing 12*(1 ) = 12 5*(0.5) = 2.5 3*(0) = = Meory Testing 9*(1) = 9 7*(0.5) = 3.5 4*(0) = = Energy Testing 8*(1) = 8 4*(0.5) = 2 8*(0) = = Functional Testing 17*(1) = 17 3*(0.5) = 1.5 0*(0) = = Interoperability 10*(1) = 10 7*(0.5) = 3.5 3*(0) = =13.5 Testing 9- Copatibility Testing 13*(1) = 13 5*(0.5) = 2.5 2*(0) = = Connectivity Testing 9*(1) = 9 7*(0.5) = 3.5 4*(0) = = Load Testing 9*(1) = 9 5*(0.5) = 2.5 6*(0) = = 11.5 The calculation results for each quality characteristics testing level according to top rated developers can also be illustrated by using Bar Chart as shown in Figure 2 below: 27

28 Figure 2: Coparison of testing level of quality characteristics fro top rated developer s perspective. According to the survey responses received, the ost used quality characteristic aong obile application developers is functional testing and the least used quality characteristic is Interoperability Testing as illustrated in the Figure2. We can order these quality characteristics with respect to their testing as following: 1. Functional Testing 2. Usability Testing 2. Reliability Testing 4. Copatibility Testing 5. Perforance Testing 6. Security Testing 7. Interoperability Testing 8. Connectivity Testing 9. Meory Testing 10. Load Testing 11. Energy Testing Testing process of top rated apps In this section, it is explained how the top rated developers actually test their applications. According to the survey responses received ost of the top rated apps developer while 28

29 creating test cases they prefer to use the test cases that are predefined in scripts or docuents instead of doing testing by theselves (for exaple rando tests, exploratory test etc.). To run their tests, top rated app developers prefer to run their tests on eulator rather than running on phone and 55% of the developers prefer to run their tests autoatically via a tool or service as a cloud while reaining 45% run their tests anually. 4.3 Low Rated Apps Iportance by low rated developers Here the analysis is being done to understand which quality characteristics are iportant, less iportant and not iportant fro low rated developer s perspective. Table 10 represents the analysis done for the iportance of quality characteristics fro low rated apps developer s perspective. Table 10: Analysis for the iportance of quality characteristics fro low Rated apps developer s perspective. S. Quality Iportant Less Not No Characteristic (1p) Iportant iportant Total Points (0.5p) (0p) 1- Usability Testing 21*(1) = 21 2*(0.5) = 1 0*(0) = = Perforance Testing 20*(1 ) = 20 3*(0.5) = 1.5 0*(0) = = Reliability Testing 11*(1) = 11 12*(0.5) = 6 0*(0) = = Security Testing 19*(1 ) = 19 3*(0.5) = 1.5 1*(0) = = Meory Testing 12*(1) = 12 6*(0.5) = 3 5*(0) = = Energy Testing 11*(1) = 11 7*(0.5) = 3.5 5*(0) = = Functional Testing 22*(1) = 22 1*(0.5) = 0.5 0*(0) = = Interoperability 13*(1) = 13 9*(0.5) = 4.5 1*(0) = = 17.5 Testing 9- Copatibility 14*(1) = 14 8*(0.5) = 4 1*(0) = = 18 Testing 10- Connectivity Testing 12*(1) = 12 10*(0.5) = 5 1*(0) = = Load Testing 10*(1) = 10 11*(0.5) = 5.5 2*(0) = =

30 The calculation results for each quality characteristic iportance can also be illustrated by using Bar Chart as shown in figure (3) below: Figure 3: Coparison of iportant quality characteristics fro low rated developer s perspective. According to the survey responses received, the ost iportant quality characteristic aong obile application developers is functional testing and the least iportant quality characteristic is eory testing as illustrated in the Figure 3. We can order these quality characteristics with respect to their iportance as following: 1. Functional Testing 2. Usability Testing 3. Perforance Testing 4. Security Testing 5. Copatibility Testing 6. Interoperability Testing 7. Reliability Testing 7. Connectivity Testing 9. Load Testing 10. Energy Testing 11. Meory Testing Testing level by low rated developers Here the analysis is being done to understand which quality characteristics are ostly used, rarely used and never used fro low rated developer s perspective. Table 11 represents the 30

31 analysis done for the testing level of quality characteristics fro low rated apps developer s perspective. Table 11: Analysis for the testing level of quality characteristics fro low Rated apps developer s perspective. S. Quality Mostly Rarely Used Never Used Total Points No Characteristic Used (0.5p) (0p) (1p) 1- Usability Testing 18*(1) = 18 5*(0.5) = 2.5 0*(0) = = Perforance Testing 13*(1 ) = 13 10*(0.5) = 5 0*(0) = = Reliability Testing 10*(1) = 10 10*(0.5) = 5 3*(0) = = Security Testing 11*(1 ) = 11 8*(0.5) = 4 4*(0) = = Meory Testing 3*(1) = 3 14*(0.5) = 7 6*(0) = = Energy Testing 5*(1) = 5 8*(0.5) = 4 10*(0) = = 9 7- Functional Testing 19*(1) = 19 4*(0.5) = 2 0*(0) = = Interoperability 6*(1) = 6 15*(0.5) = 2*(0) = = 13.5 Testing Copatibility 8*(1) = 8 14*(0.5) = 7 1*(0) = = 15 Testing 10- Connectivity Testing 7*(1) = 7 15*(0.5) = 1*(0) = = Load Testing 4*(1) = 4 15*(0.5) = 4*(0) = = The calculation results for each quality characteristic can also be illustrated by using Bar Chart as shown in figure (4) below: 31

32 Figure 4: Coparison of testing level of quality characteristics fro low rated developer s perspective. According to the survey responses received, the ost used quality characteristic aong obile application developers is functional testing and the least used quality characteristic is energy testing as illustrated in the Figure 4. We can order these quality characteristics with respect to their testing as following: 1. Functional Testing 2. Usability Testing 3. Perforance Testing 4. Security Testing 4. Copatibility Testing 4. Reliability Testing 7. Connectivity Testing 8. Interoperability Testing 9. Load Testing 10. Meory Testing 11. Energy Testing Testing process by low rated developers In this section, it is explained how the low rated developers actually test their applications. According to the survey responses received ost of the low rated apps developer while creating test cases they prefer to use the test cases that are predefined in scripts or docuents instead of doing testing by theselves (for exaple rando tests, exploratory test etc.). To 32

33 run their tests, low rated app developers also prefer to run their tests on eulator rather than running on phone and 70% of the developers prefer to run their tests autoatically via a tool or service as a cloud while reaining 30% run their tests anually. 4.4 Coparison of quality characteristics iportance, testing and their testing process In this section the analysis which was done earlier for the top rated and low rated is copared. The ost iportant quality characteristics aong top rated and low rated apps developer for which obile applications are tested is functional testing and usability testing and these quality characteristics are ostly used by both of the when testing obile applications. Quality characteristics which are not that iportant fro top rated developer s perspective for which obile applications are tested is load testing and energy testing and they are not used often by top rated apps developer. Quality characteristics which are not that iportant fro low rated developer s perspective is energy testing and eory testing and these quality characteristics are not used often by low rated apps developers. When it coes to testing, top rated and low rated developer s both create test cases that are predefined in scripts or docuents. Both developer s prefer to execute their tests on eulator rather than on obile and they prefers to run their tests autoatically via a tool or service as a cloud. The iportance, testing and testing process of top rated apps and low rated apps developer doesn t differ uch but when it coes to uniportance and not testing of quality characteristics for which obile applications are tested for it differs between top rated and low rated apps. 33

34 5 DISCUSSION In this research study I explored various quality characteristics that are used in testing of Android obile applications. Then an epirical study was perfored to verify the literature findings by conducting an epirical survey. The iportance and testing of quality characteristics of both the top rated and low rated were figured out. Moreover, the testing process used by top rated apps developer and low rated apps developer was also figured out. Answer to Research questions: RQ1: What are the quality characteristics that obile applications are tested for? Answer: 11 different quality characteristics that the obile applications are tested for is found fro the literature I-e Usability Testing, Perforance Testing, Reliability Testing, Security Testing, Meory Testing, Energy Testing, Functional Testing, Copatibility Testing, Interoperability Testing, Connectivity Testing and Load Testing. RQ2: Which are the ost and least iportant quality characteristics for which obile applications is tested for according to developer s perspective? Answer: According to the survey responses top rated apps developers and low rated apps developer both responded functional testing and usability testing are the ost iportant quality characteristics for which the obile application is tested for. Top rated apps developer responded energy testing, load testing are not iportant while low rated apps developer responded eory testing and energy testing are not iportant when testing Android obile applications. So the results differ between top rated and low rated apps developer when asked about the not iportant testing characteristics for which the obile applications are tested for. RQ3: Which are the frequently and barely quality characteristics for which the obile applications is actually tested for by developers? Answer: According to the survey responses top rated apps developers and low rated apps developers both responded functional testing, usability testing are the ost tested quality characteristics for which the obile applications is tested for. Top rated apps developer responded energy testing, load testing are not tested often while low rated apps developer responded eory testing and energy testing are not tested often when testing Android obile applications. So the results differ between top rated and low rated apps developer when asked about the not tested quality characteristics for which the obile applications are tested for. RQ4: Does the iportance and testing of quality characteristics for which the obile application is tested for differs between top rated and low rated apps developer? Answer: According to the results, the iportance and testing of quality characteristics that the obile applications are tested for doesn t differ between top rated and low rated apps developer doesn t differ. They both responded functional testing and usability testing is ost iportant and ostly tested quality characteristics for which obile applications are tested for. There is a difference between top rated apps developer and low rated apps developer regarding uniportance and not tested quality characteristic for which the obile applications are tested for. Top rated apps developer responded energy testing and load testing are not that iportant fro their perspective and is not tested often. Low rated apps developer responded eory testing and energy testing is not iportant so they don t test it often. 34