DATA SYNCHRONIZATION ON MOBILE DEVICES

Transcription

1 JUNE 16, 2015 FINAL BACHELOR PROJECT AT UNIVERSITY COLLEGE OF NORTHERN DENMARK SOFTWARE DEVELOPMENT DATA SYNCHRONIZATION ON MOBILE DEVICES UNDER THE SUPERVISION OF MOGENS HOLM IVERSEN ZLATIMIR ZAHARIEV AND GEORGI DAMYANOV

2 Abstract This report describes techniques and strategies for data synchronization on ios and Android devices. Hence, the research is based on problems that may occur in future stages of the Skonto project. Currently, the Android and ios apps download and store data from the server and present it to the final user. With the increase of data and a demand for an upload from client to server, the current implementation may not be sufficient. Thus, the solution presented in this report provides a set of relevant techniques for optimal data transfer, data storage and synchronization.

3 Table of Contents Abstract... 0 Introduction... 4 Architecture... 4 Problem statement... 5 Research method... 5 Data synchronization... 6 Synchronization patterns... 6 Synchronization Mechanisms... 6 Data Storage and Availability Data Transfer Data Synchronization in Android Skonto Android Architecture Synchronization mechanism in Android Data storage and availability in Android Data transfer in Android Android implementation challenges Data Synchronization in ios Apple s Guidelines Skonto ios Architecture ios Synchronization mechanism ios Data Storage and Availability Data Transfer in ios ios Implementation Challenges Skonto Synchronization problems Solved Problems - Deletion Scalability and future problems Solution Problem Analysis Design Data Upload Local data reduction... 40

4 Conclusion Bibliography... 47

5 Introduction During our internship at Webs&Apps we were trusted with the development of the Android and ios versions of the Skonto mobile app. Its purpose is to enlist shops, which provide exclusive discounts to registered users. The app users have a digital membership card and get notified when they are close to a shop with a discount. While approaching the first release of the product, a concern was raised regarding how the system would scale with significant data growth and additional requirements. Architecture The current architecture of the system, Figure 1, was developed before the start of this project. Influenced by the initial requirements and budget, the design considerations were not backed by a solid research. The system consists of a backend, Android, ios mobile clients and a small user-friendly web client for the administrator/owner. The backend is storing all the data in a database and exposes it for the mobile clients through a web service. When the mobile apps are open, they check the time of the last update and download any new or updated data from the server. Considering the initial expectation that the data in the system will not change often, a local database was implemented on the mobile clients. As a result the data is available offline and the user can quickly browse through it. Figure 1 - General Architecture

6 Problem statement With the increase of data and the current implementation of the architecture, a negative impact on performance may be experienced. Additionally, it does not support upload from mobile device to server and offline changes, which may be required for future releases. Considering those problems a new solution has to be designed. This requires a research regarding data synchronization and storage. Therefore, the following final project topic was found as most suitable: How to optimally synchronize data between mobile applications and server? Research method For the purpose of this project a secondary research also known as desk research method will be used. It consists of finding data from previous researches and reaching a conclusion. This method is the most viable for the project, since data synchronization is not something we have previously worked on, but it as a researched topic. The approach consists of exploring the available strategies for data synchronization and storage on mobile devices, while elaborating on their advantages and disadvantages. Additionally, the current Skonto data synchronization strategy will be analyzed and possibly improved.

7 Data synchronization With the recent development and improvement of network communications, data synchronization has become a fundamental part of almost every mobile application. As a result, nowadays an instant access to real-time data is available; information is accessible and editable offline, while providing the user only with the most relevant data. However this comes at a price. The implementation of the synchronization adds complexity to every project and variety of challenges like: data consistency, bandwidth limitation and data-freshness. Synchronization patterns In order to handle the challenges, involved with data synchronization, there are a number of strategies, which are adaptable in different context, inspired by the experience of other people, personal knowledge and research. In order to describe every pattern, format similar to the one used in Design Patterns-Elements of Reusable Object-Oriented Software will be adopted. The patterns are divided into three categories [1]: Synchronization mechanisms Data storage and availability Data transfer Synchronization Mechanisms The data synchronization mechanisms can be considered from two perspectives: uploading and downloading. The downloading is the transfer of data from the remote server to the mobile application, while uploading is the transfer from the mobile application to the remote system. Depending on the context and factors such as application requirements, fault isolation and user experience, different mechanisms can be implemented. Asynchronous Synchronization Synchronize data asynchronously without making the user interface irresponsive. Problem Blocking the interface while the data is being synchronized will result in a very poor user experience. The user may not be aware whether the app is doing something or it just blocked, since there is not visual indication about it. In fact Google went as far as not allowing networking calls, on the main thread in Android, thus preventing the UI from freezing on events like data synchronization [2]. Apple also recommends avoiding synchronous networking calls on the main thread [3].

8 Applicability The applicability considerations for asynchronous data synchronization can be viewed from downloading and uploading perspectives: Downloading new data asynchronously has the advantage of not preventing the user interaction with the app. Applications, which are functional with stale data, can benefit from this. However, data freshness is a crucial factor for other applications like stock trading or betting, where asynchronous synchronization is not applicable. Uploading can be done without the user waiting for the transfer to complete. Once the synchronization is triggered he/she can continue using the app and when the process is completed an appropriate alert can be shown. Solution As shown on Figure 2, the synchronization is triggered by an event in the app and it is done without blocking the main thread. Once the task is done the result is returned to the code on the main thread and the app handles it. Visual Representation Figure 2 - Asynchronous Synchronization

9 Consequences Advantages Not blocking the main thread and the UI o By using Asynchronous Data Synchronization, interface freezing is avoided while the information is being synchronized with the remote server. Disadvantages Increased implementation complexity o While it provides a much better user experience, implementation of Asynchronous Data Synchronization requires handling multiple threads and different execution paths, which results in more development time and more potential for bugs to hide. Examples A possible example is the post feature in social medias like Facebook. When the user uploads a post or a photo, after pressing the post button, the app continues to function properly and allows normal interaction. When the upload is finished the user gets visual indication whether it was successful or not. Synchronous Data Synchronization Manage a data synchronization event synchronously. A typical synchronous mechanism will return the result to the final user/client as soon as the execution is finished. While it occurs, the user interface is blocked. Problem This is an era of real time systems. "Customers want to see what the status of their order in e- shop is; the status of a parcel delivery - a real time parcel tracking; what the current balance on their account is." This shows the need of systems, which are constantly up to date and enable smooth manufacturing process in real-time. Additionally, many mobile applications are connected to artificial intelligence systems and business analytics systems, which are dependent on the newest information. A different scenario may be that the mobile application system relies on a response for an action from a remote system before performing another action. To prevent an application from entering unknown, non-functional states, a developer must ensure an application blocks until data synchronization is completed (successfully or unsuccessfully) [1].

10 Applicability When synchronous synchronization is applied, the user has available the most recent information, required for the functionality of the system. If asynchronous mechanism is adapted, the user runs the risk of working with stale data. Additionally, the synchronous data synchronization may be crucial in mobile applications offering online services. If the user has the option of placing orders, It would be reasonable to confirm the success or failure of the purchase synchronously and make the user wait, rather than letting him/her do orders asynchronously, allowing the user to submit more than one order without acknowledgment of the previous ones [1]. Solution Initially, the data transfer can be triggered vie event (such as user action). The process has to be performed synchronously. The application blocks and proceeds after the result is received. This process is visualized on Figure 3. Visual Representation Figure 3 - Synchronous Synchronization

11 Consequences Advantages The state of the mobile application is easily managed o The synchronous mechanism adds simplicity, because of the instructions being executed in order. Compared to the synchronous approach, the asynchronous implementations significantly increase the complexity. It requires multi-threading and proper handling of many paths of executions and states. By using synchronous data synchronization, those extra paths and states can be eliminated. Disadvantages Blocking user interface thread o The use of synchronous data synchronization results in user frustration. When such an event occurs, the application may become unresponsive. Usually the data transfers are performed asynchronously on the background, while showing a loading dialog. This does not block the user interface; it provides feedback regarding the synchronization and gives the opportunity to be canceled. Examples A user opens an application. A loading dialog is shown while the newest data is downloaded. The application continues only after the result is received. User open banking applications and enters username and password. The credentials are validated, before letting the user further. A user purchases desired item from a mobile application. The order is processed synchronously. While such an event takes a place, loading dialog is shown. Finally, result for successful or unsuccessful order is presented to the user.

12 Data Storage and Availability In many cased the data set is too large to be stored locally or has to be accessed offline. Data storage and availability patterns present a solution to these issues by managing how much data should be stored on the mobile device. Partial Storage Download and store data on demand in order to reduce unnecessary storage use. Problem The storage capacity on mobile devices is scarce; users have a lot of their own media and resources stored on the phone. As result they are often unwilling to use applications, which take too much of their valuable storage. Additionally, downloading all the data for the app at once while connected to a mobile network can be expensive. Applicability The app does not need the entire data set at the same time; therefore it can load most of the data on demand. For example an app may pre-download only the recent s and not store the whole communication archive on the device, since it will rarely need it. Solution Instead pre-fetching the whole data, the app synchronizes the information by using triggers only when it s needed. A popular approach is to use a Virtual Proxy, which creates expensive objects on demand [4]. The proxy objects provides interface for accessing and manipulating the business object, however it only loads the data if it is needed. This is done with a Data Access Object, which determines if the data is available locally and if not it loads it from the data source [5].Figure 4, visualizes the solution.

13 Visual Representation Figure 4- Partial storage Consequences Advantages Less storage is required o One of the main benefits of using Partial Storage is the reduction of space usage. Instead of storing the whole dataset on the phone, it is loaded only when needed. As a result the app can use amounts of data much larger than what could be stored on the phone. Disadvantages Examples Dependent on network connection o A drawback of this approach is the dependence on network connectivity. The app relies on loading the data on demand; therefore some of the functionality may fail in the lack of internet connection. Moreover, a slow bandwidth may significantly increase the required time for loading the data and it can cause a poor user experience. Many of the applications use partial storage, for example opening the Gmail app without internet connection shows the latest s on the screen, however trying to search in the archive or load more s brings a connectivity error. Therefore, it is safe to assume that Google is using the pattern for pre-downloading the recent conversations, but if the user wants to search in his inbox they are fetching the results on demand from their server.

14 COMPLETE STORAGE All the data is synchronized and stored on the mobile device. If the user needs it, the applications response or loading time is much faster. Problem The consumer electronics industry has spent the past 20 years making everything connect wirelessly to the Internet. An average smartphone connects wirelessly in three ways: via mobile broadband, Wi-Fi and Bluetooth. Despite the fact that wireless connectivity is widespread and available on many places, there are times when a network connection is not available, but the application must still function. In cases where the user has limited connectivity and mobile data, loading data on request may be an issue. Additionally, performing often synchronizations can reflect on the application performance. Applicability The whole data must be synchronized between the mobile device and the remote system. The complete storage pattern can be used in situations where the data set is not too large and it is not frequently changed. It can be used in cases where the information is required when wireless connectivity is not available. Solution Store the entire data set on the mobile device. It must be synchronized in cases the wireless connectivity is optimal. Visual Representation Figure 5-Complete storage

15 Consequences Advantages Not entirely dependent on network connectivity o If wireless connectivity not available, the data can t be synchronized, by the application provides certain functionality. Performance o Storing the entire data set on the mobile device makes the applications more responsive and improves the user experience, since the time for requesting information from the remote system is eliminated. Disadvantages Increased data storage o The storage of the mobile device is highly valuable for the user; therefore occupying a significant part of it may be frustrating. Increased bandwidth usage o The synchronization action transfers all the data, which may include redundant or duplicated information. This reflects on the mobile bandwidth usage. Examples Most of the weather apps are a good example of applications, which are usable offline. The apps store the latest weather information locally and present it to the user even if there is no internet connection.

16 Data Transfer Minimizing the resource usage for data synchronization on mobile application is crucial since network bandwidth is often a concern. The data transfer patterns provide transfer strategies for optimal resource utilization depending on the context. Full Transfer All the data is transferred from the server to the mobile device. Problem In some cases downloading the entire data set is the most straightforward implementation and it is not practical to spend time and resources in developing a more complex solution. Applicability Full transfer is usable when the data set is small and easily transferable from the server to the app. It is useful for apps that primarily download data from the server, which if corrupted can easily be re-downloaded. Solution Download or upload all the data when a synchronization event is triggered, Figure 6. Visual Representation Figure 6-Full transfer

17 Consequences Advantages Simplicity o This solution is easy to implement, hence it saves on development resources and it is less prone to failure. Handling corrupted data o In cases of data inconsistency it can be easily fixed by transferring the information again. Disadvantages Redundant data transfer o Downloading/Uploading all the data on synchronization can be wasteful if it has only partial changes, thus increasing the bandwidth usage Examples Simple apps like RSS feed readers or Word of the Day apps can benefit from full data transfer, since the data size is not too big and in cases of data corruption it can be easily re-downloaded. Timestamp transfer During synchronization event between the mobile device and remote system, only new and changed elements of the data set are transferred. This is achieved by using a timestamp, which shows when a record has been created or modified, compared with the time of the last synchronization event. Problem With the limitation of the wireless connection speed and bandwidth, sending the entire data set, which contains redundant information, may not be efficient. The waste of resources has to be eliminated, and the load of information has to be minimized. Applicability Using this approach the data set of an application can be synchronized in specific pieces based on timestamp. The app can download or upload those parts based on comparison between the timestamps on the mobile device and on the remote system. For this purpose, every data element must have a timestamp field, showing the last time this particular record has been modified.

18 Solution For a download, visualized on Figure 7, a timestamp from the last successful transfer is sent to the remote system with the request for new or changed data. The remote system compares the timestamps and returns only the data that has been added or modified after the timestamp provided by the mobile client application. For an upload of data from the mobile device to the remote system, only the data that has been added or changed since the last successful upload is submitted. Visual Representation Figure 7-Timestamp Transfer Consequences Advantages Minimized data set while transferring o By using Timestamp Transfer, only the new and changed data is synchronized. Decreasing the size of information has positive impact on the bandwidth usage.

19 Disadvantages Timestamp consistency o The correct timestamps have to be used in order to keep the synchronization and data consistent. It is a common practice to use the timestamp from the remote server for downloaded data, and use the device timestamp for uploaded data. Status o Timestamp synchronization works well to keep track of changes in general. A disadvantage is that the types of changes are not recorded. That makes the deletion operation complex. Examples Well-known examples are cross platform applications for keeping a list of notes. Once a new note is added, it is synchronized through all devices based on timestamp without re-downloading older data. Mathematical Transfer Since the last synchronization, only changed parts of the data are transferred. This process is based on mathematical algorithm. Problem Considering the disadvantages of using full and timestamp transfer, a more optimal approach has to be used. The mathematical synchronization significantly reduces the use of resources and determines changes more efficiently. Applicability The dataset of an application can be mathematically synchronized with remote system. In this case the mathematical properties of the data are used for determining what part of the data has to be synchronized. This strategy is used in situations where the use of bandwidth has to be significantly decreased or is restricted. However, this comes at a price and even though the mathematical approach minimizes the usage of resources, it is complex and it can be computationally intensive.

20 Solution Mathematical algorithm is used to determine the difference of data between the mobile application and remote system, Figure 8. Visual Representation Consequences Advantages Figure 8-Mathematical transfer Optimal bandwidth usage o This method uses the least bandwidth compared to full and timestamp transfers.

21 Disadvantages Time consuming o Depending on the data and the mathematical algorithm used for determining a change, the process can be very computational intensive and time consuming. Based on mathematical properties of data o Since this approach is based on unique mathematical properties, the algorithm has to be adapted to the specific data type. Hence there is no one-size-fit-all solution. Complex o The implementation of this pattern is expensive from development perspective, therefore it may not be cost effective in particular cases

22 Data Synchronization in Android This chapter will focus on how data synchronization is currently implemented between the Android mobile app and the server. Skonto Android Architecture Currently every time the application is opened, the Sync Adapter tigers a synchronization, which downloads the newest data through the REST Service Adapter and saves it into a SQLite database by the use of the Content Provider. More detailed information for every component is provided after Figure 9. Figure 9-Skonto Android architecture

23 Sync Adapter In general, this component encapsulates the code that calls tasks, which transfer data between mobile device and server. The execution of this code can be based on various options [6]. o Server data changes-the sync adapter runs based on response indicating server data change. This approach refreshes the data from the server without draining the battery with unnecessary network requests. o Device data changes In this case, the sync adapter runs when the information on the mobile device is modified. This ensures that the server has the latest data. o Network connection is available- Scheduling the sync adapter to run every time when the network connection is available, ensured that the data is regularly updated. o Interval In this case the sync adapter can run either after a certain period of time or a particular time of the day. o On demand This option is triggered by user action. Running the sync adapter on demand and requesting an immediate sync is a rather inefficient, because it bypasses all the network and battery usage optimizations. This approach must be used in situations where the demand for immediate sync is necessary [7]. The sync adapter, implemented into the Skonto application, runs on demand every time the user opens the app, but it does not automatically connect and transfer data to the server. REST Service Adapter This service adapter, part of the Skonto architecture, encapsulate the code for processing the data between the mobile device and a server. It is a component which: o Prepares the HTTP URL and HTTP request body o Executes the HTTP transaction o Parses the HTTP response o Inserts parsed data into the Content Provider Content Provider The content provider manages the access to a structured set of data. It presents the information as tables similar to the one used in relational databases, consisting of rows and columns. Although data can be directly accessed from the database, the content provider is used because of its mechanisms. Its main purpose is to provide interface that connects data in one process with code running in another, but it also provides an abstraction from the underlying data source (SQLite Database). The information from the content provider is accessed by using Content Resolver. It consists of basic methods as insert, update, query, and delete, typical for persistence storage. When the data is modified, the content provider notifies the Content Resolver. On the other side, the Cursor Loader handles the notification by implementing a Content Observer.

24 The Cursor Loader is used for asynchronous communication with the content provider. It can query in the background, which avoids the UI thread being un-responsive. When the result is retrieved, the loader populates different views of the activities by using cursor adapters. Activities and Cursor Adapter The activity is component of the application, which provides difference user interface elements [8]. The essential views in the Skonto applications are lists, which displays shops stored into the local database. When the activity and its views are created, a cursor loader is called. It runs asynchronously and query against the content provider in the background. When the result is retrieved, a cursor adapter displays the data from the query into a list view. Synchronization mechanism in Android The Sync Adapter responsible for triggering a synchronization event runs on demand asynchronously. It runs every time the user opens the application, forcing it to execute methods for downloading the newest shop data. Advantages Responsive user interface o The sync adapter runs the synchronization tasks on a background thread. This does not block the interface letting the Skonto users enjoy a fully functional app, even when data is synchronizing. Disadvantages The on demand synchronization is not completely practical in this case o This is not a particular disadvantage related to the asynchronous mechanism, but it is a downside of the Sync Adapter, as it is specifically designed to conserve battery power when it runs sync according to a schedule. Synchronizing on demand force it to run, which is inefficient use of network and power resources [7]. o At this stage of the Skonto project, the data on the server is rarely changed. Providing synchronization on demand leads to unneeded requests without the data being changed, which is degrading performance and wasting battery. Does not support upload operations o The current implementation of Android Skonto application does not support uploading data from a mobile device to a server. It only downloads information and stores it into a local database without letting the user modify it. With the growth of the size of the project and the expectation of new requirements, the upload of data from the mobile device to the server may become essential to the Skonto application functionality.

25 Data storage and availability in Android As a requirement of the Skonto application, the information about the shops should be accessed offline. For the purpose of this, the data on a mobile device must be synchronized with the Skonto remote system, whenever the application is started and network connectivity is available. When the Sync Adapter runs and the data transfer is performed, the information stored via Content Provider into a SQLite database, should be identical with the data on the Skonto server. This follows and implements the complete storage pattern described in the previous chapters. Therefore, some of the pros and cons of the pattern overlap with the implementation of the Skonto app. Advantages Performance o All the data is stored on the Android mobile device. By taking advantage of Cursor Loader, which asynchronously loads information from the Content Provider and presents it to the user by using Cursor Adapter, the Skonto application responsiveness and user experience are improved. Efficiency o The information is accessed from the local SQLite database. Thus unnecessary network calls are eliminated, which prevents decreased performance and waste of battery life. Offline functionality o Storing all the data on the Android mobile device provides the full functionality of the Skonto application, even if there is not any network connectivity. It gives the user ability to search information about the shop and get notified whenever he or she is close to it. Disadvantages Scalability o Because of the limited storage that the Android mobile devices have and the probability of increasing the data, storing all of the information may not be practical.

26 Data transfer in Android The strategy used for data transfer, when synchronizing between the Skonto Android application and the remote system, is based on timestamp. The Skonto Android app stores the time of the last modification in the Shared Preferences. The Shared Preferences provides a general framework that allows storing and fetching key-value pairs of primitive data type. This data persists even if the application is killed by the system or the user [9]. When a synchronization event occurs, the time of the last successful update is sent as a request to the Skonto server. The Skonto remote system receives the time and compares it with all records of the data set containing a field, which holds a timestamp showing the last time this element has been modified. If changes are identified, a response containing the newly modified records is build and sent to the mobile client. This implementation follows the timestamp transfer pattern, since it was the most reasonable and practical solution at the time. Advantages Resource efficiency o By taking an advantage of using timestamp transfer, which drastically decrease the amount of data transferred between the Android mobile device and Skonto server, it is also a reasonable strategy considering time and cost, essential constrains of the project. Disadvantages Timestamp storage o When implementing a timestamp transfer for Skonto, an extra care had to be taken on keeping the correct timestamp and its format. A minor difference could lead to a data inconsistency. Deletion o When an item gets deleted, it cannot be simply removed from the server. There might be still mobile clients, whose devices are offline and have to be synchronized. When a data transfer is performed, the mobile client must be notified of the deletion of this particular item.

27 Android implementation challenges The components responsible for synchronization in the Skonto Android applications are Sync Adapter and Rest Service Adapter. By following the guides, provided by Google developer website, the Sync Adapter was implemented in a straightforward manner. While implementing Rest Service Adapter, responsible for sending request to the Skonto server and handling properly the given response, an extra time and effort had to be taken on keeping the correct time stamp. TIMESTAMP PRESERVATION Every time a data transfer is executed, the Android mobile client sends a timestamp of the last synchronization. But what happens if it was not successful? This may be caused by problems with the network connectivity, issues while parsing the response from the server or SQLite database error, refusing an insertion or update of a shop. Those situations had to be handled properly and the timestamp must be updated only in case where no problems are encountered. Data Synchronization in ios The focus of this part of the project is how data synchronization and storage were implemented in ios, including a discussion of the architecture specific for the ios app and which patterns are currently followed. Apple s Guidelines Apple s software and hardware ecosystem is often referred to as a walled garden. [10] Developing for ios requires careful examination of what Apple considers best practices, since the review process for the App Store is fairly strict and it is common for apps to be rejected on ground of not following the preset guidelines. Therefore, when implementing the synchronization mechanism for Skonto, it was crucial to consider all the recommendations and requirements enforced by Apple, in order to ensure the successful release of the app.

28 Skonto ios Architecture The architecture of the app follows the well-integrated in ios development MVC model. Elaborating on the MVC related implementation details is out of the scope of this project; hence the focus will be on the data synchronization and storage. The following, Figure 10, is a diagram of the architecture from that perspective. Figure 10-Skonto i0s architecture UIViewController is a fundamental class of the controller layer in every ios application. The controller is responsible for updating the view according to the underlying data. Usually this class is subclassed and adapted according to the specific needs. The view controller coordinates with other controller objects, thus the app presents a consistent and coherent user experience. [11]. In the Skonto case, every time a controller is loaded a pre-configured fetch is executed and the data is loaded from the Core Data persistence framework.

29 Core Data is a framework that provides solutions to tasks related to object life cycle and object graph management, including persistence. Even though Core Data with underlying SQLite database is widely used for persistent storage on ios, the framework differs from traditional databases in variety of ways. To begin with it focuses on loading the object graph in the main memory from where the objects can be accessed. It is in fact possible to use Core Data without any persistence, but in most cases it is reasonable to store the data. Additionally all Core Data objects are proper Objective C/Swift objects with their own methods, attributes and relationships. [12] Therefore passing them from one controller to another is as easy as passing a reference and all the related objects can be accessed on demand. Sync Engine is a custom class implemented with the Singleton pattern, therefore only one instance of the class is created through the app. Unfortunately ios does not provide a Sync Adapter like the one available for Android. The Sync Engine triggers an event every time the app is brought to the foreground. A slightly different approach could be to start the synchronization engine only when the app launches. However, ios has the nature of keeping apps on the background as long as a user is often accessing them and only kill them if they are not used for a significant amount of time. This may lead to serious update delay for the most frequent users. Hence, the current synchronization timing was applied. The Networking Classes are building the communication layer of the app. They are primarily responsible for making request to the web service and parsing the results. Having them encapsulated in a different layer ensures that in future the app can easily adapt different method of synchronization with the server without having to change any other layers. Currently the networking classes are making HTTTP POST requests to the server, which return JSONS responses to be parsed from the app. ios Synchronization mechanism Synchronization can be time consuming, since it is network dependent, therefore the engine starts on a background thread provided by the Grand Central Dispatch ios mechanism and does not disturb the user interaction with the app. Grand Central Dispatch (GCD) is a set of language features and runtime libraries that provide easy access to concurrent code execution on ios and OSX devices, hence it was a perfect way of implementing the asynchronous aspect of the data update. Similarly to the Android implementation, the ios app shares some of the general pros and cons of the asynchronous synchronization mechanism.

30 Advantages Instant access to data and responsive interface o Starting the synchronization when the app is launched ensures that the user will have the most recent data, however it may not happen instantly. As a result it is better to have old data then no data at all. Disadvantages No data when the app opens for the first time o Once the Skonto application has been downloaded for the first time from the app store, it may take considerable amount of time to get all the data from the server, depending on the amount of data and internet speed. As a result the user will not have any data to see, which can be quite unpleasant and if the synchronization takes too long he or she may even loose interest in the app Concurrency handling o Even though Grand Central Dispatch significantly reduces the complication of concurrent programming, it is still harder to think and debug code parts that run simultaneously ios Data Storage and Availability In the Skonto ios implementation a need for persistent local storage was identified in the beginning of the project, therefore after reflecting on the options Core Data was chosen to handle it. One of the considerations was to use a mature and reliable technology, since Apple recommends using Core Data when persistence is needed and also considering the continuous improvements of the framework, it was safe to assume that Core Data will be alive and well maintained in the near future. Moreover, as previously mentioned Apple is quite keen with it s application review process, hence minimizing the usage of third party frameworks is often welcomed. Advantages Storage flexibility o In addition to offline functionality and improved performance, positive effects of adapting the complete storage strategy, it also allows storing specific user settings for every shop in the app. For example, when a user marks a shop as favorite it is straightforward to just flag it in the local copy of the database. Presenting favorite shops happens by just querying the Core Data implementation for all the marked entries. Similarly all kind of other user dependent settings can be added to the shops, as a result of using local database.

31 Disadvantages Data migration o As the app grows the increase of attributes and requirements is inevitable. In order to follow the complete storage strategy, new fields will have to be introduced to the local database. As a result the migration complexity will increase exponentially. Data Transfer in ios The current data synchronization is adapting the timestamp data transfer mechanism. The ios app stores the time of the last update as NSDate object in the NSUserDefaults. The NSUserDefaults is simply a dictionary accessible throughout the whole app, which is storing key-value pairs used for representing user settings. Similarly to the Android implementation, when building the HTTP POST request in the networking layer of the app, the timestamp is added as a parameter to the request, which is send to the server. Advantages Cost effectiveness o Since the data synchronization is triggered every time the ios app is brought to the foreground and not only when it is launched, it is crucial to optimize the data transfer as much as possible. As in Android, timestamp transfer is the most suitable solution considering the available resources. Disadvantages Extra coupling between client and server o In order for the timestamp data synchronization to work properly, both mobile client and server have to agree on a date time format. This adds additional degree of coupling and if a new format has to be adapted for some reason it may require changes through the whole system.

32 ios Implementation Challenges While working on the implementation of the synchronization a few problems were encountered. Most of them were related to handling concurrency with the data insertion and data visualization, as well as data migration when the database structure has to be changed. Core Data Concurrency In order to not block the main thread while all the shops are inserted in the local database it was crucial to run the update and insertion of records on a separate thread. However, simultaneously the user has to be able to view the information, which was already inserted, hence some concurrency issues were encountered. In order to understand the concurrency issue with core data, a brief introduction to how the framework works is necessary. The following, Figure 11, shows an overview of it. Everything is stored as a database file and exposed to the framework as a Persistent store object, which is essentially the connection to the file. In order to access and query the data a Managed object context has to be created, Apple describes the Managed object context as a scratch pad for manipulating objects in the database [13]; it is usually shared through the app. Once a context is available it can be used to load or insert managed objects (records) in the database. The main pitfall when working with concurrency and core data is to continue using the same shared managed object context from different threads. The context object is not thread safe, therefore using one object for the main and background thread can lead to unpredictable results. In the Skonto project, after encountering this challenge it was solved by introducing another context object, connected the same persistent store. This way it was ensured that data could be inserted in the local database via one managed context and viewed from the main thread through another managed context Figure 11-Core Data concurrency Source:

33 Data Migration Another challenge in the data synchronization was changing and improving the database scheme between different versions of the app. Once the data model has changed the app has to know how to map the already existing data in the scheme to the new one. There are three major migration techniques available with Core Data: lightweight (automatic), manual and custom. Lightweight migration relies on Core Data inferring mapping model from the differences between the two data models. For this technique to work the changes in the model must fit an obvious migration pattern like [14]: Simple addition of a new attribute Removal of an attribute A non-optional attribute becoming optional An optional attribute becoming non-optional, and defining a default value Renaming an entity or property Fortunately, through the Skonto project all of the data model changes fit into one of this categories, therefore the data migration was not too complex. However, with every new version of the app the data model is likely to grow, thus significantly increasing the migration complexity.

34 Skonto Synchronization problems Solved Problems - Deletion One of the disadvantages of using Timestamp Transfer approach is the inability of keeping track of the status of a specific record. As described, the challenge for implementing the Timestamp Transfer strategy on both platforms, ios and Android, is to manage properly the delete operation. To determine that an item has been deleted, a Boolean flag is used. For this purpose a new field is added, as shown on Figure 12, which indicates whether the item has been deleted or not. Figure 12-Deletion When synchronization occurs, the mobile client receives a response from the server with the newly modified records. The deleted field is used to mark a record so if a deletion is indicated, the mobile client would remove it from the local repository. Scalability and future problems With the growth of the system, and an analysis of the current Skonto data synchronization a few problems are possible to occur in the near future. First and foremost, the system currently does not support upload from a mobile device to server. This was not an issue in the early stages of the project, but as the apps mature it may become unavoidable. Furthermore, a large part of the data that is synchronized from the server to the app is not actively used. This combined with the expected growth of shop s attributes to be stored may lead to a high level of redundancy. Additionally, the present synchronization scheduling is not optimal, which generates unnecessary requests to the server.

35 Solution Considering the problems described in the previous chapter, only the issues most valuable from stakeholder perspective will be analyzed. In this part of the project the focus will be on optimizing the locally stored data and introducing data upload from mobile client to server. The solution will be designed, but not implemented. The main reason is the fact that the system is in very early stages and it is not known how it will advance in future. The risk is too high and possible implementation will not be practical, considering time and resources. Additionally, if the project evolves, the requirements and priorities may drastically change, which could eliminate the need for this particular implementation. Problem Analysis Initially, Skonto started primarily with discounts at fast food joints only. However, the owner quickly diversified and introduced other businesses. As a result, many new shops and categories were added to the app, which highly increased the amount of inessential information locally stored on the mobile device. Moreover, extra attributes such as comments and photo albums are expected to appear in future versions. Hence, it will not be practical to store them on the phone. Additionally, if the system proves to be successful extended functionality such as comments and ratings may have to be included. After reflecting on those problems, the following requirements were identified as critical for Skonto s near future: Data upload - include support for upload from mobile client to server in order to allow ratings and comments Reduce locally stored data eliminate unnecessary information stored in the local storage of the mobile device

36 Design This part focuses on building solutions for the analyzed problems. It is logically divided into two parts, each one with emphasis on a particular requirement from the previous chapter. Data Upload Sending data from the apps to the server has to be implemented in order to support a rating system and user comments for every shop. The user selected rating will be uploaded to the server through a web service. In cases of no internet connection the chosen rating will have to be stored in the local database until synchronization with the server is possible. When the user wants to leave a comment a similar approach will be adapted. In order to implement this new functionality a few things have to be changed in the system at general. To begin with the backend has to be extended to store the user ratings and comments. Afterwards, a service has to be exposed in order to allow submitting ratings and comments from the mobile clients. The rating calculation and storage on the server side are outside the scope of this project. However, the solutions for ios and Android will be discussed in detail in the following two parts.

37 DATA UPLOAD IN ANDROID In order to support upload functionality, the Android architecture has to be modified, as shown on Figure 13. After a field, which indicates that a record is updated, has been added to the local SQLite database, the Content Provider must expose a method, which queries the dataset, selects and returns all modified records since the last successful synchronization. Additionally, the web service that connects the Skonto Android application with the remote system has to be modified so it requires a timestamp of the last successful data transfer, along with a list of updated shops. The rest of the components are unaffected. Figure 13-Android Data Upload architecture The process of synchronization, consisting of upload and download operation, between a Skonto Android application and Skonto server can be visualized as a sequence diagram as shown on Figure 14. It is initialized every time the user opens the application. When the activity is started, an event, which runs the Sync Adapter, is triggered. Before sending a network request to the Skonto server, the timestamp of the last successful synchronization, stored in the Shared Preferences, is retrieved and used for comparison for getting all newly modified records. Then the body of the POST Request is constructed, and the fetched timestamp and items are sent to the Skonto server. The server processes the information and returns a response, which contains new or modified shops for the mobile device. The shops are parsed, stored into the database via the Content Provider and visualized. Finally, the timestamp in the Shared Preferences is updated.

38 Figure 14-Android Data Upload SD Data Upload in ios Similarly to the Android implementation the ios architecture will have to be changed, as visualized on Figure 15, in order to support uploading. A modified attribute will have to be included in the local database in order to indicate when the last time the particular record has changed. When the Sync Engine is triggered it will get all the records that have changed since the last synchronization and it will send them as parameters to the Networking Layer. The changed data is included in the request to the web service and the response is handled without any difference.

39 Figure 15-i0S Data Upload architecture The following sequence diagram, Figure 16, represents a detailed visualization of how the logic will flow when synchronization is triggered. The SyncEngine is still responsible for controlling the synchronization process; the first thing it does is taking the timestamp of the last synchronization from NSUserDefaults and loading all the shops that have local changes waiting for upload. Once the shops are loaded from the database, through the Core Data framework, the SyncEngine calls the networking class responsible for making POST Requests and parsing the server response. The networking class sends the updated data to the web service and in return receives new or updated records from the server. The records are being sent back to the SyncEngine, which updates the local database and stores a new timestamp in the NSUserDefaults.

40 Figure 16- i0s Data Upload SD

41 Local data reduction The following diagram, Figure 17, presents all the shops attributes currently stored on the mobile devices. Figure 17 Most of the attributes are self-explanatory, however there are a few which should be described in more detail. Isfavourite is an integer representation of Boolean, which shows whether the shop is marked as favorite on this particular device or not.

42 Notificationsareenabled is an integer representation of Boolean, which indicates if notifications related to this shop are allowed on the particular phone. The user can activate and deactivate the notifications for every shop. Modified is a timestamp, which represents the time of last data change. This is used for uploading only the data that has changed since the last synchronization. Notification_sent is a timestamp, which indicates when the last time a notification related to that shop was sent. A requirement for the Skonto app was to not send more than one notification per shop in 24 hours, as a result this field was introduced to keep track on the notification timing. Rating and selected_rating are used to store the general rating of the shop and the one selected by the user. The selected _rating field is important because if the user picks a rating while he or she is offline it will be locally stored until synchronization occurs. With the inclusion of new attributes, such as comments and photo albums, it will not be practical to store everything locally since it may result in a high usage of storage and data transfer resources. Downloading and accessing the required information from the server on demand could avoid this. Similarly, the current set of attributes could be decreased. After careful consideration and discussion with the owner it was decided, that if data has to be excluded and not available offline the local database could look the following way, Figure 18: Figure 18

43 One of the main changes compared to the previous diagram is the exclusion of opening and closing hours, which were represented in many attributes since a part of the requirements was to show in the view of every shop if it is open or closed at the moment. Additionally, also the shop description was removed, as it usually consists of a long text, which occupies larger space than any other piece of information. A disadvantage of this reduction is the inability of providing working hours information and shop description when the phone is offline. In extreme cases an even more drastic data reduction could be needed, hence another exclusion of locally stored attributes was considered. The following diagram, Figure 19, is the shop information that is regarded as absolutely essential. Figure 19 Each one of these attributes is necessary for the proper functionality of the app. The flags for notifications, favorites and rating selection are used for storing user preferences; therefore they have to be stored locally. The title, category, city and region are essential for sorting the shops in the app. The coordinates are used for delivering notifications based on geofencing, even when no internet connection is available. With the attributes reduction, a part of the information will be accessed on demand. This leads to many network request to the Skonto server which can be reduced with the help of caching. Nevertheless, caching is a complex subject and it is outside the scope of this project. Therefore it will not be elaborated upon. Another data reduction strategy is to reduce the amount of shops synchronized on the mobile device. Since the Skonto application has discount at shops through whole Denmark, it is unreasonable for a user located in Aalborg or other city to have the data for the whole country saved on his/her device. Accordingly, the data set of synchronized shops could be based on user location, region or city. This way the locally stored information will be reduced significantly. However, handling frequent travelers and users living in between two major areas will be a challenge. Considering the desire for simplicity and the currently limited budget of the owner, such a solution will not be rational, hence the focus will be on attribute exclusion strategy.

44 LOCAL DATA REDUCTION IN ANDROID After a careful consideration, the set of attributes, stored locally, has to be defined, so that some of them will be kept on the mobile device and the rest received via network call to the remote system. This is visualized on Figure 20. The data stored on the Android device will be accessed through a Content Provider, where an AsyncTask, which will be added to the RestServiceAdapter, will have the responsibility of sending a request to the Skonto server and visualizing the response. It will perform the network call asynchronously and publish the result to the interface only after the task is finished. Figure 20-Android Local data reduction Architecture With this solution, there are two ways of accessing the information, both of which are visualized and discussed accordingly: Data stored locally When a information is saved on the mobile devices,the Cursor Loader communicate with the Content Provider,and desired data is returned.the process is visualized on Figure 21.

45 Figure 21-Data stored locally Data not stored locally When data is accessed via network call, a new background thread, which does not block the UI thread, is started. A request is sent to the Skonto service and the response is published as a result on the user interface. The technique can be seen on Figure 22. Figure 22-Data not stored locally LOCAL DATA REDUCTION IN IOS Similarly to the Android implementation, the ios architecture has to be modified in order to support loading a set of shop attributes on demand. The bold parts of Figure 23 visualize which components of the current implementation will have to be changed. The UIViewController will have to start an asynchronous synchronization event on a separate thread with the help of the Grand Central Dispatch (GCD). Afterwards the Networking Layer will load the additional attributes from the server and the process will return the data to the controller, which will update the view.