Database Recovery Mechanism For Android Devices

Transcription

1 Database Recovery Mechanism For Android Devices Seminar Report Submitted in partial fulfillment of the requirements for the degree of Master of Technology by Raj Agrawal Roll No: 11305R004 Under the guidance of Prof. Deepak B. Phatak Department of Computer Science and Engineering Indian Institute of Technology, Bombay

2 Contents 1 Introduction 1 2 Android Introduction The Android Architecture Need Of Database in Android Devices SQLite Introduction SQLite File Structure SQLite database file Schema table Table b-tree Page structure and unallocated area Recovery Mechanism Log Based Recovery Shadow Paging Recovery mechanism of SQLite Using Rollback Journal Using Write Ahead logging Problem Statement Solution Adaptive Logging Overview Problem statement Solution Conclusion 20 i

3 List of Figures 2.1 Android Architecture[1] SQLite file structure and header first 32 bytes SQLite b-tree structure SQLite page structure[2] Internal Page Header Contents Internal Page Header Contents Leaf cell structure Rollback Journal of SQLite[3] Two Update patterns[4] ii

4 Abstract Android devices are becoming popular due to their lower cost and integration with Google services. Open Source Andriod-SDK encourages the development of variety of applications. Considerable amount of data needs to be stored and organized for these applications. Flash Memory is well adapted to store any type of data and provides secure services in a mobile devices. But many cases of application failure, can corrupt data in flash Memory. This report compares two most popular recovery algorithms used in Database Management Systems, i.e. shadow paging and log based. The report also highlights the benefits of using shadow paging for flash memory devices. An overview of adaptive logging approch is also provided in the report.

5 Chapter 1 Introduction ANDROID is a Linux-based Light Weight Open-Source Operating System developed by Google for mobile devices such as smart phones and tablet computers. It is one of the most widely used mobile OS these days. Android owns 68% share of global Smartphone market by Q4, 2012 after being launched in 2007[5]. Developers can develop applications for free or on a propriety basis for Android using Android SDK. Google has also established Play store which allows developers to distribute their applications to Android users. Almost all of these applications require to store and update considerable amount of data. The internal memory of Android devices are very less, for example Aakash is having 512 MB, hence only small number of application can reside in the device and rest of the applications reside on the SD card(flash memory) which comes as an external storage. Flash memory is non-volatile, shock-resistant, uses little power and also it is hundreds of times faster than a hard disk in read operations. These attractive features make flash memory one of the best choices for portable information systems. Android comes with SQLite as its native database engine. SQLite stores the data of every application in a separate file. This file cannot be accessed by any other application. Recovery is one of the most important part of any database. It stores the information for each and every update in the database and uses this information to restore database to the previous consistent state, if something goes wrong. An important question arise here is that which recovery mechanism should be used? as there are many recovery mechanisms available. This is because selecting an optimal recovery mechanism can help in achieving the attractive features of the device or the memory in which database engine will work. It can save both space and time, as it helps in speeding the commit operation, as well as the recovery process. The rest of the report is organized as follows. Chapter 2 gives an overview of the architecture of android system and then highlights the need 1

6 of database in android system and the requirement that database should posses. Chapter 3 explains the file structure of SQLite (the native database engine of android). Chapter 4 talks about the various recovery mechanisms and about the recovery mechanism of SQLite (log based recovery mechanism). We also present the idea of implementing shadow paging in SQLite, instead of using log based recovery mechanism. Chapter 5 explain the concept of adaptive logging and its benefits. 2

7 Chapter 2 Android 2.1 Introduction Android is a software package comprising of an Operating System, series of JAVA applications for mobile devices or tablets, and a middleware which act as interface between these applications and the operating system. Android s operating environment is based upon Linux kernel V2.6. Android platform is produced to develop innovative mobile application program, to make full use of all functions connected to mobile devices. The Android SDK provides the tools and APIs necessary to begin developing applications on the Android platform using the Java programming language. 2.2 The Android Architecture Android is a multi-process system in which each application runs its own process. It consists of 4 layers as shown in the Figure 2.1.[6] Linux Kernel- Android relies on Linux V2.6 for core system services such as security, memory management, process management, network stack, and driver model. The kernel also acts as an abstraction layer between the hardware and rest of the software stack. Libraries- Android includes a set of C/C++ libraries used by various components of the Android system. These capabilities are exposed to developers through the Android application framework. Android Runtime- Android includes a set of core libraries that provides most of the functionalities available in the core libraries of Java programming language. Every Android application runs in its own process, with its own instance of the Dalvik virtual machine(vm). Dalvik has been written so that a device can run multiple VMs efficiently. The Dalvik VM executes files in the Dalvik Executable (.dex) format which is optimized for minimal memory footprint. 3

8 Figure 2.1: Android Architecture[1] Application Framework- By providing an open development platform, Android offers developers the ability to build extremely rich and innovative applications. Developers are free to take advantage of the device hardware, access location information, run background services, set alarms, add notifications to the status bar, and much more. Applications- Android ships with a set of core applications including an client, SMS program, calendar, maps, browser, contacts, etc. All applications are written using the Java programming language. 2.3 Need Of Database in Android Devices Recent advances in processors, memory, storage, and connectivity have paved the way for next generation applications that are data-driven. Memory sizes have gone up and prices have come down significantly, enabling us to extend amount of data and number of application that can reside in a device. With advances in flash memory technology, large flash drives are available at reasonable prices. Flash drives consume less power than conventional disks, making them ideal for portable devices. All this leads to mobile and specialized application architectures. Application components can run on mobile platform when mobile devices are provided with a database. 4

9 Chapter 3 SQLite 3.1 Introduction SQLite is an ANSI-C-based Open Source Database which is embedded into Android. SQLite supports standard relational database features like SQL syntax, transactions and prepared statements. In addition it requires only little memory at runtime (approx. 250 Kbyte). The increased use of portable devices, such as smart phones, has contributed to growing adaptation of SQLite. The data-centric applications on android devices use SQLite as a native database engine. Unlike other types of database systems, SQLite is usually used as local-only database that saves all the results of database usage in a single file. 3.2 SQLite File Structure SQLite database file The overall structure of a SQLite file is provided in Fig 3.1, and the file signature is 0x c f 72 6d (16 bytes). SQLite files do not have a specific filename extension, and therefore, identifying a SQLite file is determined by the signature of target file. The page that appears at the beginning of file is called header page. The upper part of a header page comprises the database file s header and its signature. The lower part is the schema table containing the table information of the database. The next pages consist of B-trees, again largely separated as index B-tree and table B-tree where the latter stores data contents. In SQLite, the elementary unit of store operation is a cell, while cell structure and stored data content vary to internal and leaf pages. An internal page is found in the middle section of tree, and the cells store the pointers 5

10 Figure 3.1: SQLite file structure and header first 32 bytes to locate lower page. A leaf page is located at the bottom of tree. The cells in this part contain database records Schema table The schema table in a header page contains information of table, index, and trigger comprised in the SQLite file in application software. The schema type indicates which of table, index, and trigger the type of a specific schema is. The schema name section contains generated schema names and then the copy of each name, followed by root page numbers (1-byte integer), followed by a string which is a SQL query statement that is sent to SQLite when the schema is created. There are four field types in SQLite, namely INTEGER, TEXT, BLOB, and NUMERIC. Datatypes including VARCHAR(125), BIG INT, and DATE that exist in the standard SQL query are converted to one of the four datatypes in SQLite, which ever the system determines as the most relevant Table b-tree The purpose of a table b-tree is to store actual data. Structure of a table b-tree is provided in Fig An internal page contains pointers to the child page, while actual data are stored in the leaf page Page structure and unallocated area All pages in a SQLite file, both internal and leaf pages, share the structure illustrated in Fig A page header is followed by list of big endian integers 6

11 Figure 3.2: SQLite b-tree structure at 2-byte offset values, which specifies the location of a cell that contains actual data. Cells are used in upward sequence from the bottom of the page. The area between a cell offset and a cell is referred to as free space, filled with zeros when initially created. Beside free space, there can be unallocated space called free block within a leaf page, a place to store a cell until deleted and maintained to be used later for newly generated and assigned cell Internal page header An internal page header shown in the Fig 3.4, consists of 12-byte, where the first byte is a page flag with value of 0Ö05. The first two offsets are 2-byte big endian integers indicating the first free block offset. The third and fourth offset represents the number of cells existing in the page, written in 2-byte big endian integers. The specific page is a free page, when the number of cells are described as zero. Offsets fifth and sixth are the offset of first appeared cell. The seventh offset describes the number of free blocks (3-bytes and smaller), written in 1-byte integer. The final offsets from 8 to 11 are the rightmost page number found within the lower section of the current page, present as 4-byte big endian. 7

12 Figure 3.3: SQLite page structure[2] Leaf page header A leaf page header has a structure similar to an internal page header. The leaf page header consists of 8-bytes, where the first byte is a flag with value of 0 Ö0D. The rest of the content are identical to that of an internal page, the only difference is that the last 4-byte information found in an internal page is now omitted as there is no child page Internal cell SQLite uses a cell as the elementary unit of stored information. The cells in an internal page, as described in Fig. 3.5, are for maintaining a b-tree and have pointers to the child page (child page number). A key, the identification value of a cell, is in variable length integer. The first four bytes in a cell is the child page number in big endian format Leaf cell Cells within a leaf page of a table b-tree include database records and are called Leaf Cell. In order to describe the records stored in a leaf cell, records are managed in three areas: cell header, record header, and record data area (see Fig 3.6). In a cell header, the length of the cell, excluding the 8

13 Figure 3.4: Internal Page Header Contents Figure 3.5: Internal Page Header Contents Figure 3.6: Leaf cell structure cell header, and rowid in variable length integers. A record header contains the value of record header length in variable length integer format, followed by the record data area, in which byte length information of each field of a record is stored. 9

14 Chapter 4 Recovery Mechanism Recovery is one of the most important part of any database. It stores the information for each and every update in the database and uses that information to restore database to the consistent state, if something goes wrong. There have been many recovery mechanism developed over time. 4.1 Log Based Recovery Log based recovery is one of the most widely used recovery mechanism. It stores the logs for each and every update in the database and uses those logs to restore database to the consistent state, if something goes wrong. Log is a history of actions executed by a database management system. Log is a sequence log records, recording all the update activities in the database. In short, transaction log is a journal, which contains history of all transactions performed. Log contains start of transaction, trans-id, recordid, type of operation (insert, update, delete), old value, new value, end of transaction i.e. commit or abort. Example: ˆ < T i start> for start of transaction. ˆ < T i abort> says transaction has aborted. ˆ < T i commit> says transaction has committed. ˆ <T i, X j, V 1, V 2 >. where V 1 and V 2 are old and new value of attribute X j. Whenever a transaction updates database, it is essential that, log records for the operations are generated and added to the log file, before the database is modified. Once log records are written to the stable storage, we can perform the modification to the database. By using log records we can undo or redo the effects of the aborted or committed transaction. 10

15 For log record to be useful for recovery from system and disk failures, the logs must reside in stable storage. 4.2 Shadow Paging This recovery scheme does not uses log records. In Shadow paging, an update transaction creates the new copy of the pages that are being updated and the page table itself. All the updates are done to the new copy of the pages, leaving the original page, as it is. Any page that is not updated is not copied, but instead the new page table simply stores the pointer to the original page. If at any point the transaction has to be aborted, the system deletes the new copy of the pages and the page table. The current copy of the database pages is identified by the pointer to the page table (page table pointer). When the transaction commits, the pointer to the page table is updated atomically to point to the new copy of the page table. The implementation actually depends on the write to the page table pointer being atomic; that is, either all its bytes are written or none of its bytes are written. Since recovery involves neither undoing nor redoing data items, this technique can be categorized as a NO-UNDO/NO-REDO technique for recovery. 4.3 Recovery mechanism of SQLite. There are two recovery mechanism in SQLite, Rollback journal and Write Ahead logging. Either of the two is used. Rollback journal is the default recovery mechanism Using Rollback Journal Updating disk content ˆ Initially all the data resides in Disk, the OS cache is empty, as well as the main memory is also empty. ˆ Acquiring A Read Lock: When an transaction wants to read a data from the disk, it first obtains a shared lock on the OS buffer. A shared lock allows two or more database connections to read from the database file at the same time, but it prevents another database connection from writing to the database file concurrently. 11

16 ˆ Reading Information Out Of the Database: The data is transferred from the disk to the OS cache and the user space (main memory) reads from OS cache. ˆ Obtaining A Reserved Lock: A reserved lock allow other processes to read from the database file concurrently. A single reserve lock can exist with multiple shared locks from other processes. However, there can only be a single reserved lock on the database file. Hence only a single process can be attempting to write to the database at one time. ˆ Creating A Rollback Journal File: The next step is to creates a separate rollback journal file and write into the rollback journal, the original content of the database pages that are to be modified. The rollback journal contains a small header that records the original size of the database file. If a change causes the database file to grow, we still know the original size of the database. ˆ Changing Database Pages In User Space: After the original page content has been saved in the rollback journal, the pages are modified in user memory. ˆ Flushing the Rollback Journal File To Mass Storage: In next step the content of the rollback journal file is flushed to disk. This is a two step process. In the first step, the body of rollback journal i.e the pages is flushed to disk. In next step the header of the rollback journal is modified to show the number of pages that is written in first step and then the header is flushed to disk. ˆ Obtaining an Exclusive Lock: In next step we obtain an exclusive lock on the database file. Obtaining an exclusive lock is a two-step process. First a pending lock is obtained and then it is escalated to an exclusive lock. A pending lock allows other processes that already have a shared lock to continue reading the database file. But it prevents new shared locks from being established. ˆ Writing Changes to The Database File: Once an exclusive lock is held, we actually modify the database file. Usually these changes are made only to the disk cache and mass storage is not updated. ˆ Flushing Changes To Mass Storage: Finally a flush is done to write all the database changes to the disk. ˆ Deleting the Rollback Journal: After flushing the database changes on the disk, the rollback journal file is deleted, as shown in Fig 4.1. This is the instant where the transaction commits. 12

17 Figure 4.1: Rollback Journal of SQLite[3] ˆ Releasing the Lock: The last step in the commit process is to release the exclusive lock so that other processes can once again start accessing the database file When something goes wrong: Rollback ˆ Suppose the power loss occurred while the database changes were being written to disk. After power is restored, the situation might be that partial data is written on the disk. Hence we require a rollback. ˆ Hot Rollback Journals: A hot journal only exists when an earlier process was in the middle of committing a transaction when it crashed or lost power. A rollback journal is a hot journal if all of the following are true: The rollback journal exist. The rollback journal is not an empty file. There is no reserved lock on the main database file. The header of the rollback journal is well-formed and in particular has not been zeroed out. ˆ Obtaining an Exclusive Lock on the Database: The first step towards dealing with a hot journal is to obtain an exclusive lock on the database 13

18 file. This prevents two or more processes from trying to rollback the same hot journal at the same time. ˆ Rolling Back Incomplete Changes: Once a process obtains an exclusive lock, it then proceeds to write the content of the rollback journal, back to the database file. ˆ Deleting the Hot Journal: After all information in the rollback journal has been played back into the database file (and flushed to disk in case we encounter yet another power failure), the hot rollback journal can be deleted Using Write Ahead logging The default method by which SQLite implements atomic commit is Rollback Journal. Alternately, a new Write-Ahead Log option (referred as WAL ) is available. There are advantages and disadvantages to using WAL instead of a rollback journal Advantages: ˆ WAL is significantly faster in most scenarios. ˆ WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently. ˆ Disk I/O operations tend to be more sequential using WAL Disadvantages: ˆ All processes using a database must be on the same host computer. WAL does not work over a network file system. ˆ WAL might be very slightly slower than the traditional rollback-journal approach in applications that do mostly reads and rare write. ˆ There is the extra operation of checkpointing. ˆ WAL works best with smaller transactions. WAL does not work well for very large transactions. For transactions larger than about 100 MB, traditional rollback journal modes will likely be faster. For transactions in excess of a GB, WAL mode may fail with an I/O or disk-full error. It is recommended that one of the rollback journal modes be used for transactions larger than a 40 MB. 14

19 How WAL Works The rollback journal works by writing a copy of the original unchanged database content into a separate rollback journal file and then writing changes directly into the database file. In the event of a crash or ROLLBACK, the original content contained in the rollback journal is played back into the database file to revert the database file to its original state. The COMMIT occurs when the rollback journal is deleted. The WAL works in opposite way. The original content is preserved in the database file and the changes are appended into a separate Log file. A COMMIT occurs when a record indicating a commit is appended to the Log file. Thus a COMMIT can happen without ever writing changes to the original database, which allows readers to continue operating from the original unaltered database while changes are simultaneously being committed into the Log file Checkpointing Moving the Log file transactions back into the database is called a checkpoint. By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.[8] 4.4 Problem Statement Recently, Flash memory is non-volatile, uses little power and also it is hundreds of times faster than hard disk in read operation. These attractive features make flash memory one of the best choices for portable information systems. However, flash memory has two major drawbacks. ˆ A segment, blocks of flash memory, need to be erased before they can be rewritten. This is because flash memory technology only allows individual bits to be toggled in one way for writes. The erase operation writes ones or zeros into all the bits in a segment. This erase operation takes much longer than a read or writes operation. ˆ The life of each memory block is limited to 10,00,000 writes[9]. SQLite uses Rollback journaling or write ahead logging, which are updatein-place approach. In update-in-place approaches, both undo and redo logs should be saved in a log file before the transaction commits in order to make failure recovery possible. As the name suggest, Update-in-place rewrite the same memory block, which doesn t overcome the above two mention problem of flash memory. 15

20 4.5 Solution In Shadow paging approach, a database maintains two images per page during the lifetime of a transaction: a current page and a shadow page. When a transaction starts, both pages are identical. The shadow page is never changed over the duration of the transaction. The current page will be changed when a transaction performs a write operation. To undo modification, it frees current page. To commit modification, it modifies all pointers to old (shadow) page to now point to new (current) page, and frees the shadow page. If the system fails, then shadow pages are used to recover and set the status of the system as it was before the failure.[9] Implementing shadow paging can benefit in following ways: ˆ Shadow paging help in overcoming the above mention constraints of flash memory. ˆ Log based mechanism will not be needed, which eliminates the overhead of creating and storing logs resulting in optimization of storage space. ˆ At the time of failure there is no need to first read from a log file and then update database, instead just change the next page and previous page pointers. ˆ Changing pointers rather than reading from file and then updating database, is much simpler and faster resulting in speedy recovery mechanism. 16

21 Chapter 5 Adaptive Logging Adaptive logging is a novel recovery method, in which according to the update state of each page, recovery mechanism can switch from log based (which is further referred as ARIES) to shadow paging at a page level dynamically on run time. This approach focuses on reducing the update log size, by applying different logging methods at page level dynamically on run time switching from ARIES to shadow paging according to the update state of each page. There are two patterns of update called A-pattern (ARIES-favourable update pattern); and S-pattern (Shadow-paging-favourable update pattern). A page is used as the basic unit of recovery, first. Fig 5.1 shows these two patterns. A-pattern updates a small area of a data page in a transaction, which generates a small amount of log that is less than the size of the data page. If a data page is updated by an A-pattern update operation, it benefits from no-force policy of ARIES, since ARIES writes log records without flushing the data page to disk on commit time. A transaction consisting of mostly A-pattern update operations are called as small update transaction. S-pattern updates a large area of a data page or repeatedly updates the same area of a data page in a transaction, which generates large amount of log that is larger than the data page size. If the update pattern of the transaction is S-pattern, it can t make benefit of the no-force policy, because the log size itself is larger than the data page size. In this case, shadow paging is better since it doesn t make any log record and its force policy needs less pages to be written: updated data pages are less than log pages by ARIES. We call this type of transaction as large update transaction. Updating large objects is one typical example of the large update transaction.[11] 17

22 5.1 Overview Figure 5.1: Two Update patterns[4] ARIES is applied as a default recovery mechanism. When a transaction starts and update occurrs, the log records are generated using ARIES way. Each data page records the the total size of log records generated from the page. If the size exceeds a predefined threshold value, the logging mechanism of the page is switched to shadow paging. However, if the page lock of the page cannot be acquired due to the other transaction s operations, the logging mechanism will stay with ARIES. When the switch occurs; a new copy of the page is created in the buffer as well as in the disk. Then all updates to that page is reflected into the new copy, without generating log records until the transaction commits. For the same transaction updates for the pages of which the threshold is not exceeded, log records are still generated. When the transaction commits, all new copies of the data pages from the transaction as well as log records are flushed to stable storage. When the next transaction starts, the recovery method for updates to all pages will be ARIES. Shadow paging is only applied adaptively to the page of which total size of log records exceeds the threshold value. 18

23 5.2 Problem statement When SQLite is used in Android, we know that the database creates a single database file and limited number of pages within that file. Whenever there is an update request for some attributes, one of the two following scenario can occur: ˆ The number of attributes is more than one and is present in the same page. Request will update the large area of a data page. ˆ The request is for single attribute, resulting in repeatedly update of same area of a data page. In both the scenarios, lots of logs are being generated, causing the size of the log file to grow even bigger than the size of data page itself. 5.3 Solution Once shadow paging is implemented, we can further enhance recovery mechanism by implementing adaptive logging, which dynamically switches from log-based recovery to shadow paging, once the number of logs for a particular page exceed the predefined threshold value. This predefined threshold value can be half of the size of the page and further, experiments can be performed to observe the effects of changing the threshold value to obtain an optimal threshold value. The benefits of using adaptive logging are: ˆ Shifting to shadow paging does not require further logging of records. ˆ Since the recovery method is selected at the page level, at the time of failure, rollback to the original state can be achieved much quicker. 19

24 Chapter 6 Conclusion Android Operating System provides a native database engine, SQLite. We have studied about file structure of SQLite. We also had an overview of different recovery techniques, including mechanisms which SQLite uses, i.e. rollback journal and write ahead logging. The study showed that shadow paging is better for flash memory. We also studied the adaptive logging technique. According to this technique, when the size of log reaches the threshold value, the recovery mechanism switches from log based (ARIES) to shadow paging. 20

25 Acknowledgements I would like to express my deepest gratitude to my guide Prof. D. B. Phatak, for his patience and guidance throughout the seminar. I would also like to thanks Nagesh Karmali for his support for my work. I would also like to thank everyone who supported me in my work. 21

26 Bibliography [1] Android Architecture. Architecture. [Online; accessed Apr-2013]. [2] S. Jeon, J. Bang, K. Byun, and S. Lee, A recovery method of deleted record for sqlite database, Personal and Ubiquitous Computing, vol. 16, pp , [3] Roolback Journal. [Online; accessed Apr-2013]. [4] Y.-S. Kim, H. Jin, and K.-G. Woo, Adaptive logging for mobile device, Proc. VLDB Endow., vol. 3, pp , Sept [5] J. Pepitone, Android races past Apple in smartphone market share. smartphone-market-share/index.html, [Online; accessed Apr-2012]. [6] S. Lee, Creating and using databases for android applications, Academic Journal, vol. Vol. 5, p. 99, June [7] Silberschatz, Korth, and Sudarshan, DataBase System Concepts, Sixth Edition. McGraw Hill, [8] Write-Ahead Logging. [Online; accessed Apr-2013]. [9] S. Byun, S. Cho, and M. Huh, Flash memory shadow paging scheme for portable computers: Design and performance evaluation, ACM Trans. Database Syst., vol. 39, Oct [10] C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, and P. Schwarz, Aries: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging, ACM Trans. Database Syst., vol. 17, pp , Mar

27 [11] R. A. Lorie, Physical integrity in a large segmented database, ACM Trans. Database Syst., vol. 2, pp , Mar