Faculty of Computer Science. Real-Time Group. Diploma Thesis. Analyzing Real-Time Behavior of Flash Memories

Transcription

1 Faculty of Computer Science Real-Time Group Diploma Thesis Analyzing Real-Time Behavior of Flash Memories Author: Date of Birth: Daniel Parthey March 10, 1982 in Karl-Marx-Stadt Date of Issue: October 2, 2006 Date of Submission: April 2, 2007 Supervisor: Dr. Robert Baumgartl

2 Task Description Flash Memories have an increasing importance for the construction of mechanically robust embedded computer systems and consumer electronics. For consumer applications, at least five different solutions exist: Compact Flash (CF), Sony Memory Stick (MS), Secure Digital (SD), Multimedia Card (MMC) and the xd-picture Card. Of every type, different generations with different technical parameters (access speed, capacity) exist. Usually, the embedded controller of the medium is responsible for wear-levelling and error correction. If no controller exists, the file system must take care of these aspects. Therefore, a number of specialized file systems have been developed, among them the Journalling Flash File System (JFFS2) and the Yet Another Flash File System (YAFFS2). The aim of this thesis is to develop methods to characterize the timing of access operations for flash memories of different types and with different file systems. Amongst other the work should present detailed analysis of the following aspects: basic write and read access operations to different media types performance with random and sequential access patterns achievable and sustainable throughput including worst cases mount times for media of different capacity and file systems evolution of certain parameters for media of different age worst case behavior of different file systems influence of file system on performance and timing predictability The term real-time signifies that not only an average value has to be obtained for every parameter, but the worst case timing is interesting as well. 2

3 Abstract Flash memories are used as the main storage in many portable consumer electronic devices because they are more robust than hard drives. This document gives an overview of existing consumer flash memory technologies which are mostly removable flash memory cards. It discusses to which degree consumer flash devices are suitable for real-time systems and provides a detailed timing analysis of some consumer flash devices. Further, it describes methods to analyze mount times, access performance and timing predictability of flash memories. Important factors which influence access timings of flash memories are pointed out and different flash file systems are evaluated with regard to their suitability for real-time systems. Some remaining problems of existing flash file system implementations concerning real-time use are discussed.

4 Contents 1 Introduction 1 2 Basics Flash Memory Technologies Media Types Interfaces Flash Management File Systems for Flash State Of The Art Storage Benchmark Tools Previous Benchmark Results Summary Flanatoo - A Flash Analysis Tool Basic Use Time Measurements Direct Access Supported Benchmarks Removable Flash Card Analysis Test Setup Test Performance Test Results VFAT Mount Times Setup Measurements Discussion Real-Time Support for Flash File Systems Problems of conventional flash file systems Implementation Proposals i

5 8 Further Work Collect more data Worst-case analysis of flash file systems Real-time support in flash file systems Conclusion 41 A Glossary 44

6 1 Introduction Over the last decade, flash memories have become cheaper and continuously enjoyed increasing importance. They enable the user to quickly transfer data from one device to another without the need for a computer network. They are compatible with a wide range of platforms because most of them support the FAT/FAT32 file system which has become a standard for removable flash devices. Flash based USB drives have almost completely replaced the conventional floppy disk because they are small in size, easy to handle and have a much higher storage capacity. Flash technology is superior to conventional random access memory because it is non-volatile. This means that none of the already stored data is lost when power is turned off or a power failure occurs. In contrast to hard disk drives, flash storage is silent, robust against shock, and the raw device is accessible shortly after power is turned on because there are no moving parts inside of a flash memory. A hard disk drive needs several seconds after power-on until its disks have spun up and its data is accessible [1]. With flash memory instead, no disks have to be spun up, no head has to be moved and there is no need to wait for the disk to spin until the data appears under the head because flash memory is implemented using transistor cells. This makes flash an interesting technology for building robust and fault-tolerant embedded systems which need a quick startup, but this requires some further investigation on the properties of flash devices especially with regard to their suitability for real-time systems. Chapter 2 gives an overview of the properties of common removable and non-removable flash technologies. It also describes interfaces which can be used to connect flash memory devices to a host. Chapter 3 presents a collection of tools which can be used to analyse flash devices and file systems. Further, it points to previous benchmarks which analysed the average read and write performance of many different flash devices and card readers. Chapter 4 describes the usage and functionality of a tool which was developed for this work to perform specific tests on flash cards. Chapter 5 describes the setup and discusses the results of the benchmarks which were performed on different flash cards. It highlights the most relevant tests and anomalies. In order to store files on flash devices, a file system is required, and when it comes to flash file systems, the mount time is often an issue. So the mount times of the VFAT file system, which is normally used on removable flash devices, were also measured. The results of these tests are described in chapter 6. This chapter is directly followed by chapter 7 which explains the main problems of conventional flash file systems and why they should not be used in hard real-time systems. Chapter 7 also highlights issues which should be considered in the design of new real-time file systems for flash devices. Chapter 8 summarizes work which still needs to be done and highlights topics for further research. 1

7 2 Basics In order to analyse the behavior and anomalies of flash memories, one has to understand the technology behind them. The following sections give an overview of basic flash technologies and common types of flash memories. 2.1 Flash Memory Technologies Flash is a kind of electrically erasable read only memory (EEPROM). Its memory cells are based on a metal-oxide-semiconductor field-effect transistor (MOSFET). These MOSFETs are extended by a socalled floating gate between control gate and substrate which works as a trap for electric charge and prevents leakage of electrons [2]. This makes the cell non-volatile and enables it to keep its state (1 or 0) over a long period of time without any refresh or applied voltage. Such memory cells are grouped into larger erase units (around 512 KBit). The erased logical state of each cell is one and write operations can only switch single bits from one to zero by applying a negative voltage (10 to 13 Volts). If bits of an erase unit need to be changed from zero to one, the whole erase unit must be cleared and set to logical one. This is where the name flash comes from because clearing one erase unit is like a flash of light erasing all the data in the unit [2]. Unlike other storage technologies, flash memory differentiates between read, write and erase. The timing of raw flash chips can be characterized by three basic values: maximum read cycle time, maximum write cycle time and maximum block erase time. Other important characteristics which influence performance are the flash type (NAND, NOR, etc.), the physical sector size, physical erase block size and the total size of the flash chip. For complex flash devices like flash cards or USB drives, the number of flash chips and the type of controller can also impact timing NOR Flash Some of the first and smaller flash devices (up to 8 MiB) used NOR flash where the transistor cells are connected in parallel (OR). Each bit can be separately accessed through a logic grid. This allows for arbitrary write operations where bits can be changed from one to zero, until all cells of the whole erase unit (usually 128 KiB) are set to zero because the cells do not interfere with each other. Being linearly accessible makes NOR flash well suitable as execute in place (XIP) program memories for micro processors. This is why NOR flash is still popular to boot embedded devices. The NOR technology can rarely be found in removable flash devices because of its low storage density and relatively high cost per chip [2], [3]. 2

8 2.1.2 NAND Flash The NAND flash technology is used in most current devices and connects the transistor memory cells in series. This makes a higher data density and lower cost per chip possible. A deficiency of this approach is that direct bit access is not possible. Any cell of a series which shall not be modified has to be masked with an offset voltage. Therefore, the data is written to an internal register which takes care of addressing the flash cell. Afterwards, a write command on the command bus of the flash device initiates a data transfer from the register to the flash cell [2]. Because of electron leakage in neighbour cells when writing a certain cell, NAND flash only allows a maximum number of about 10 write operations to a single erase unit until the whole erase unit must be erased. Erase units in NAND flash are usually divided into pages of 512 bytes. Each of these pages allows storing 16 extra bytes of out-of-band (OOB) information, like error correction codes (ECC), erase counters, and other meta-information [3]. NAND flash always requires ECC data since this technology is prone to page errors by design. There are two different types of NAND flash which heavily affect the speed and capacity of flash devices. A single level cell (SLC) can only store a single bit in its floating gate, the multi level cell (MLC) technology allows storing several bits in one cell. This can either be accomplished by using several floating gates or different levels of amperage to encode different states [2]. The x4 NAND flash technology [4] from m-systems allows storing 4 bits in each flash cell OneNAND Flash OneNAND [5] flash is a relatively new technology and has entered the market in It tries to combine the advantages of NOR and NAND flash and eliminate the need for an expensive NOR flash chip. Each OneNAND device contains a NAND flash chip, a NOR interface logic, and an SRAM buffer up to 5 KiB size. This way the device provides a NOR interface to the outside world which allows transparent XIP. Since the internal NAND flash allows high bandwidths up to 128 MB/s, OneNAND can reach much higher sustained read and write performance in comparison to conventional NOR flash. Access latencies might be better for NOR flash shortly after the device is powered on, but once the boot code is buffered in SRAM, the OneNAND device is able to provide better access latencies than any NAND or NOR flash chip. Since OneNAND flash has no moving parts and is very power-efficient, it will also be used in hybrid hard disk drives to save power and store a certain amount of data while the disks are not running. 2.2 Media Types There is a great variety of interfaces and protocols which provide access to the internal flash chips in different ways. The following sections will describe the most common interfaces for removable and embedded flash devices Smart Media (SM) Smart Media cards are based on NAND flash memory compatible to the Toshiba TC58 series. They have been introduced by Toshiba in Smart Media devices do not have an integrated controller, therefore any additional features like wear levelling (as described in section 2.4.1) or flash file systems need to 3

9 be implemented in external hardware or software. Originally, they were called Solid State Floppy Disk, being only a bit smaller than usual floppy disks. Later, they were renamed to Smart Media. With the help of an adapter, the 22-pad-connector of Smart Media devices can be connected to a floppy cable. There are also adapters which connect them to a PCMCIA slot or a Compact Flash Type II slot. A memory card reader can connect them via a controller to the Universal Serial Bus or IEEE-1394 (FireWire) port. A deficiency of the Smart Media technology is that its maximum capacity is limited to 128 MiB, which resulted in its replacement by the xd-picture Card xd-picture Card (xd) The xd-picture Card is a removable flash memory card with a proprietary interface, which has been introduced by Olympus and FujiFilm in It removes the capacity limit of Smart Media devices by implementing a memory logic which supports a theoretical maximum capacity of 8 GB. Like Smart Media devices, xd-picture Cards do not have an integrated controller. In order to access data stored on the internal NAND flash device through the 18-pin connector of the xd-picture Card, the host device must support the Single Level Cell or Multi Level Cell technology. Currently, there are three types of xd-picture Cards: The classic type without any label supports 16 to 512 MB, the Multi Level Cell type "M" supports 256 MB up to 2 GB, and the High Speed type "H" supports capacities from 512 MB up to 1 GB Memory Stick (MS) The Memory Stick has been developed by Sony in 1998 [6]. According to the Memory Stick Pro Specification [7] it has an integrated controller which implements an ECC error detection and correction mechanism to detect bad blocks and correct bit errors which are both common for NAND flash. It does not provide a default mass storage device interface, but a Memory Stick bus. The following paragraphs will give a short overview and try to clarify the names of the different Memory Sticks which can be found on the market. The standard Sony Memory Stick is 50x21.5x2.8mm large [8]. It is narrower than a Smart Media or a Compact Flash card and only supports capacities from 4 MB to 128 MB [9]. The interface provides one serial data line with a bus clock of 20 MHz [10] which corresponds to 20 MBit/s and therefore limits the maximum theoretical transfer rate to 2.5 MB/s. The Sony Memory Stick Select (MS Select) was an attempt to duplicate the capacity of a Standard Memory Stick by mechanically switching between two memories on one stick. Both halves of the stick act like a single Memory Stick. The advantage of this approach is that it is supported by any device which supports the standard Memory Stick, without the need for any new firmware or controller, but because the user needed to switch back and forth between the two halves, it has not prevailed and was superseded by superior technologies. The introduction of the Memory Stick Pro (MSP) improved the maximum bandwidth and capacity by implementing a new interface with 4 parallel lines with a bus clock of 40 Mhz. This allows a maximum theoretical bandwidth of 160 Mbit/s or 20 MB/s [10]. The Memory Stick Pro specification [7] defines a maximum possible capacity of 32 GB. The specification recommends the following layers for devices which make use of Memory Sticks: 4

10 Application layer takes care of file contents File management layer handles the FAT file system and defines the logical device structure Protocol layer provides a serial or parallel interface to the stick and handles incoming commands Physical layer specifies physical and electrical properties of the Memory Stick bus Only the lower three layers are covered by the specification. It is not possible to bypass the protocol layer in order to access flash memory directly. Memory Stick Pro are always controlled through a command interface provided by a controller. There are three types of commands: Transfer Protocol Commands (TPC) are used to control the memory stick and access the data buffer and registers of the Memory Stick Pro. Memory Access Commands are used to access the flash memory and allow READ/WRITE access to the user or information block area. This set of commands also includes an ERASE command which deletes data from the current address in the user area and a STOP command which terminates any of the above operations. Function Commands provide special functions to control the device. The FORMAT command formats the whole device and the SLEEP command allows the host controller to put the device into a low power consumption state. Memory Stick Interface Controller VSS Vcc BS DATA1 SDIO/ DATA0 DATA2 INS Serial Interface Register Data Buffer Memory Interface Sequencer Flash Memory Interface Flash Memory DATA3 SCLK VCC Parallel Interface ECC OSC Controller VSS Figure 2.1: Memory Stick Pro Scheme Both Memory Stick and Memory Stick Pro are available in a smaller form factor of 31x20x1.6mm. These versions have a suffix "Duo" and are called Memory Stick Duo (MSD) and Memory Stick Pro Duo (MSPD). Both Duo variants are usually shipped with simple plastic adapters to fit them mechanically 5

11 into normal sized Memory Stick slots because their electrical interface is identical to the interface of their larger counterparts. Memory Stick Micro has been introduced in the beginning of It also uses the Memory Stick Pro technology, but with a size of only 12.5x15x1.2mm it is even smaller than Memory Stick Pro Duo. It allows high capacities up to 32 GB as well as high transfer rates up to 20 MB/s (like Memory Stick Pro) and is additionally extremely small sized MultiMediaCard (MMC) The MultiMediaCard (MMC) was developed by Ingentix and SanDisk in It can be accessed using one of two different protocols which are described in the MultiMediaCard Specification [11]: The default bus protocol is the MultiMediaCard bus and the other protocol is the Serial Peripheral Interface (SPI). One of these two has to be selected when the card is powered on, and the protocol cannot be changed without power-cycling the card. The MMC bus protocol allows three different bus clock frequencies. The frequencies and bus protocols supported by the host s and card s controllers have to be negotiated when the card is powered on [2]. All Multimedia Cards are backwards compatible with the x1 mode. This mode uses a single data line running at a maximum clock speed of 20 MHz, which results in a theoretical maximum bandwidth of 2.5 MB/s. The MMC plus standard introduced the x4 mode using four parallel data lines running at a clock frequency of at least 26 MHz (which allows a maximum bandwidth of at least 13 MB/s) and the x8 mode using eight parallel data lines with a clock frequency of 52 MHz (which allows a maximum bandwidth of 52 MB/s). These high-speed modes have been developed to provide a minimum bandwidth for current multimedia applications like recording of video streams and for multi-shot digital cameras which have to store a lot of data in a short time. All MMC cards support a voltage of 3.3V. They require a minimum voltage of 2.7V and allow a maximum voltage of 3.6V. Cards with a DV label are dual voltage cards [11] and can be run with only 1.8V in a range from 1.65V to 1.95V. The standard form factor of MMC cards specifies dimensions of 24x32x1.4mm and a connector with 7 (for MMC) or 13 (for MMC plus) pins. The MMC plus standard allows higher transfer rates when using the x4 or x8 bus protocol. Smaller form factors have been developed to fit the cards into small size mobile devices. The reduced size (RS) form factor is a half-sized version of the standard MMC card with dimensions of 24x18x1.4mm. Reduced size cards use the same connector as standard sized cards and provide full electrical compatibility. A passive extender, which consists of a piece of plastic and a small clip, can be used to put reduced size MMC cards into standard sized MMC slots. The MMC Mini form factor (20x21.5x1.4mm) and the MMC Micro form factor (14x12x1.1mm) use 11 pins. MMC mobile sounds like another form factor, but in fact it is just a short marketing term for MMC DV RS Plus, which stands for a small reduced-size multimedia card with dual-voltage and high bandwidth support [2]. The MMC bus interface, as well as reading, writing and erasing of blocks, is handled by an internal controller and completely independent from the flash technology used internally. The MMC standard allows host software to work smoothly even with freshly developed MMC cards using new internal technologies. When data is written to an MMC card, the controller may induce a read after write and an ECC 6

12 error correction on newly erased units in order to verify if the data was written correctly. The internal controller also implements an error management algorithm to correct and replace bits or even whole sectors by reserved ones, which at least applies to the Samsung MMC device MC56U032NCFA [12]. MMC cards may have an automatic sleep mode to reduce power consumption when no command was received for 5 milliseconds. This is why MMC cards are in sleep mode most of the time when they are not being accessed. The sleep mode is transparent to the host, and the device automatically wakes up when any command is received. Even though the duration between sleep mode and ready state might be interesting to real-time developers, it is not mentioned in Samsungs data sheet [12]. The MMC specification [11] recommends that external MMC adapters should implement caching in order to increase throughput by speeding up random access or read-modify-write operations, where a cache can be very efficient on frequent updates. This recommendation indicates that MMC cards most probably contain only a small cache or no cache at all Secure Digital Memory Card (SD) The secure digital card was invented by SanDisk in SanDisk is a member of the SD Card Association which defines today s standards for secure digital cards. All secure digital cards have a standardized 9-pin connector and contain an integrated controller which implements the SD bus protocol, SD specific security features like secure digital music initiative (SDMI), and which does a simple form of wear levelling (see section 2.4.1) by distributing write operations across the device. In a flash memory overview article [2], the c t magazine states that the controller provides up to three different bus modes: The first one is the serial peripheral interface (SPI), the second one is a 1 bit SD transfer mode and the last one is a 4 bit SD data transfer mode. While SPI mode and 1 bit SD data transfer mode (x1) are mandatory for all cards, the 4 bit SD data transfer mode (x4) is optional for low speed cards and mandatory only for SD high speed cards [13]. The x4 mode supports two clock frequencies, a normal frequency of 25 MHz which allows a maximum theoretical bandwidth of 12.5 MB/s and a high speed frequency of 50 MHz which allows a maximum bandwidth of 25 MB/s. SD card slots are backwards compatible with MMC because the SD specification [14] is based on the MMC standard. Therefore, MMC cards work in SD slots, but not the other way round. To prevent SD cards from accidentally being put into an MMC slot, SD cards are usually thicker than MMC cards, but there are SD card form factors which are only 1.4 mm thin. These cards will not work in an MMC slot, in spite of the fact that they would mechanically fit in. SD cards are specified [14] in three different form factors. The standard SD card has dimensions of 24x32x2.1mm, MiniSD cards are only 20x21.5x1.4mm large and the size of MicroSD is 11x15x1mm which approximates the size of a fingertip. Therefore, MicroSD cards can be used in a wide range of small embedded devices like modern mobile phones. To provide a wide compatibility between devices supporting the SD protocol, the SD specification requires devices to support for the FAT16 file system and the SD High Capacity (SDHC) specification requires support for the FAT32 file system. Cards larger than 2 GByte which are not pre-formatted with the FAT32 file system do not comply with the SDHC standard because SDHC devices must support FAT32, but are not required to support any other file system [2]. 7

13 2.2.6 Compact Flash (CF) Compact Flash Cards were developed by SanDisk in 1994 as one of the first flash devices. They have become a quasi standard for professional digital photography because of their high capacity, high speed and strong mechanical robustness. Nevertheless, they are likely to be replaced by Secure Digital cards because SD cards are smaller and becoming cheaper than CF cards. The interface of compact flash cards is compatible to ATA and consists of 50 pins, of which 40 are an ATA connector and 10 are reserved for alternative operating modes and the power supply. There are two form factors for compact flash cards: Type I with dimensions of 42.8x36.4x3.3mm is usually used for flash memories, because most digital cameras merely contain a thin CF slot. The thicker type II with the dimensions 42.8x36.4x5.0mm is common for devices which need more space. Examples are devices like WLAN, ethernet and bluetooth adapters or barcode scanners which are connected through the Compact Flash In/Out (CFIO) interface. Hard disks called Microdrives also require a Compact Flash type II slot since they would not fit in the smaller type I slot. In fact, Microdrives do not contain any flash memory and this is why they only shall be of minor interest in this document. Four Compact Flash interface standards have been developed over time, and all of these standards are fully compatible with each other. The CF 1.0 standard allows programmed I/O (PIO) mode 2 with a maximum transfer rate of 8.3 MByte/s, the CF 2.0 standard allows PIO 4 with a transfer rate of 16.6 MByte/s, CF 3.0 allows using the UDMA-66 mode with a maximum theoretical transfer rate of 66 MByte/s, and the current CF 4.0 standard [15] which supports UDMA-133 and a maximum transfer rate of 133 MByte/s even exceeds the USB high-speed bandwidth of 400 MBit/s. Therefore, CF 4.0 based SanDisk Extreme IV cards can only unfold their full performance when used in a card reader which allows a maximum theoretical bandwidth of at least 100 MByte/s, e. g. a card reader supporting IEEE 1394b (FireWire-800 or FireWire-1600) [2] Disk On Chip (DOC) The DiskOnChip technology was developed by m-systems in 1997 [16] which has become a subsidiary of SanDisk in 2006 [2]. These integrated devices with 32 pins contain NAND flash memory and a microcontroller for bit error detection and correction (EDC). The internal controller also acts as glue logic to make a small memory window of the NAND flash directly visible and addressable like NOR flash [17]. This helps to reduce cost by omitting the NOR flash, but still allows booting from the device. DiskOnChip devices are independent from the flash technology used internally because the controller hides the specific NAND flash interface from the outside [18]. Specific information about mdoc devices like the format version and the type of physical flash media can be extracted using the MS-DOS utility dinfo provided by m-systems [19] USB Flash Drive USB flash drives contain a flash memory chip and an integrated controller placed together on a printed circuit board. Since there is no proper standard for the shape or weight of USB flash drives, there are various drives with different sizes, colours, controllers and flash memory technologies, but all have one thing in common: All USB drives have a male USB connector and implement the USB mass storage 8

14 device class (MSC) which means that the drive exports a linear device which hides the physical block structure, number of flash chips, erasure of blocks, flash addressing and wear levelling from the host system. This indirection allows using the same generic mass storage drivers for all USB drives but makes it more complicated to analyze the behavior of USB flash drives. Performance of USB drives may be influenced by the USB chipset on the mainboard, but for modern mainboards this is rarely the case, according to a c t article about USB flash drives [20]. The USB protocol overhead may also slow down the transfer. An important parameter for USB keydrives is the number of flash chips used because this determines whether the drive can be run in dual-channel mode with interleaved memory access (which can almost double the effective transfer rate) or needs to be run in single-channel mode. Another bottleneck for USB keydrives is the integrated mass storage controller because vendors tend to keep their devices as cheap as possible and therefore use cheap controllers Comparison of Flash Devices The following table shortly summarizes the developer companies, interfaces and features of the most common flash memory devices available. All of them are removable devices, except DiskOnChip (DOC). Device Developer Since Type Size Controller Bus PINs in mm SM Toshiba 1995 NAND 45/37/0.76 none SM 22 xd Olympus 2003 MLC NAND 25/20/1.7 none xd 18 Fujifilm MS Sony 1998 MLC NAND 50/21.5/2.8 ECC, MS (TPC) 10/11 31/20/1.6 Buffer 12.5/15/1.2 MMC Ingentix 1997 NAND 24/32/1.4 CRC MMC/SPI SanDisk 24/18/1.4 7/10/13 12/14/1.1 SD SanDisk 2001 NAND 24/32/2.1 CRC, SD/SPI 9 20/21.5/1.4 ECC 11/15/1.0 CF SanDisk 1994 NAND 42.8/36.4/3.3 CRC, ATA 50 (rarely NOR) 42.8/36.4/5.0 ECC DOC M-Systems 1997 NAND variable ECC SRAM 32 USB-Drive M-Systems 1998 NAND variable MSC USB (MSC) 4 Table 2.1: Comparison of flash memory interfaces 2.3 Interfaces Card readers and flash drives may be connected to the host system in various ways. The following sections will shortly summarize the most common interfaces and their capabilities. 9

15 2.3.1 USB - Universal Serial Bus A popular interface which can be found in all modern PCs and some embedded devices is the universal serial bus interface. The latest standard, USB 2.0, supports transfer rates up to 480 MBit/s. The actual transfer rate of a transmission is influenced by transfer mode, bus transaction delays and USB protocol overhead. The calculation of worst case transaction delays and protocol overhead can be obtained from the USB specification [21]. Chapter 5 of the specification describes the four different transfer modes supported by USB: Control transfers are bidirectional, best-effort transfers where a packet has a maximum payload size of 64 byte and every frame is acknowledged. These are intended to detect and configure USB devices but not to transmit real payload data. Isochronous transfers use a synchronized clock to deliver data in a timely manner in preference to correctness. This mode is best suited for multimedia streams, which require data to be either delivered in time or discarded if it contained one or more errors. Interrupt transfers provide a guaranteed maximum service period for transmission attempts, which is useful for real-time applications. After the desired period, the transmission is either finished or has failed, and it may be retried in the next period, to which the same guarantee applies. The last transfer mode is called bulk transfer. This mode does not provide any timing guarantee, but tries to take advantage of the highest bandwidth available. So the recommendation for real-time programmers is to make use of either isochronous or interrupt transfer, but not bulk transfer. The maximum theoretical transfer rate of USB 2.0 is high enough to access most of the currently available flash memories at their maximum transfer rate. USB 2.0 can still be a bottleneck for fast flash memory devices which comply with the Compact Flash 4.0 standard as explained in section So there is a need for alternative interfaces to connect high-speed card readers to a host system FireWire - IEEE-1394 Another high-speed serial bus is FireWire. It is hot-pluggable and well suited for transfer of multimedia streams. The first IEEE-1394 standard [22] from 1995 already specified transfer rates of up to 400 MBit/s. According to chapter of the specification, FireWire allows isochronous bus arbitration and bandwidth allocation in order to guarantee a timely delivery of data. In 2000, the first amendment IEEE-1394a [23] was added. The latest standard IEEE-1394b [24] appeared in This standard introduced specifications for maximum theoretical transfer rates of up to 800 and 1600 MBit/s. In contrast to USB 2.0, FireWire-800 and FireWire-1600 card readers are able to fulfill the bandwidth requirements of storage cards following the Compact Flash 4.0 specification [15] ATA Compact Flash cards can be connected to an ATA port using a simple Compact Flash to ATA adapter and communicate by using their True IDE I/O Transfer Function as specified in chapter 4.7 of the Compact Flash specification [15]. It is important to note that the compact flash interface is hotpluggable, but the ATA interface is not. So a compact flash may not be unplugged from a CF/ATA adapter while connected to the ATA port of a running PC, otherwise the system might lock up. 10

16 2.3.4 Memory Technology Devices Some flash memories, especially on-board devices and devices connected via serial peripheral interface, can be directly accessed using an appropriate hardware driver. Such a driver can be either generic code like the configurable MTD NAND driver which works out of the box, or a vendor customized driver which can perform better on the specific flash chip [25]. Since neither block nor character devices differentiate between read, write and erase operations, Linux supports a special Memory Technology Device (MTD), which provides a function set to access all features of flash memories. The MTD subsystem in the operating system provides generic access to the out-of-band and user area of flash memories while limiting the hardware specific driver part to the smallest amount possible and keeping driver development simple. The MTD interface allows application developers to create special flash file systems and block mapping layers like UBI (cf. section 2.4.3). Software based on the MTD subsystem is independent from specific hardware while still being able to access flash specific functions. There is a utility mtd_debug which uses the MTD subsystem to display detailed information about the underlying memory technology device, e. g. the type of flash (NAND or NOR) and the erase unit size. This information is gathered from the flash chip using the Common Flash Memory Interface (CFI) [26] Summary Timings can only be guaranteed if all layers from the physical flash memory device layer to the application layer specify and guarantee worst case timing constraints. Therefore, the data bus and its arbitration and transfer modes need to be carefully selected in order to guarantee the bandwidth and latency that a real-time application or memory device requires. Hard real-time systems additionally require real-time drivers for the chosen interface which are able to guarantee maximum latencies. Figure 2.2 shows a model of several possible flash architectures in which Linux is used as the operating system. It illustrates the involved layers and their purposes in each architecture. 2.4 Flash Management Wear Levelling Flash devices can be rewritten only a limited number of times because the oxide layer in the memory cells between control gate, floating gate and substrate gets worn out [2]. After a sufficient number of erase cycles, blocks will become erroneous or statistically unreliable. Vendors guarantee a minimum erase count from 10,000 up to 1,000,000 erase cycles, depending on the technology. In fact, torture tests [27] have shown that modern OneNAND flash memories survive about 6 million erase cycles until the first blocks become erroneous, which is about six times higher than vendors usually guarantee. In order to prolong the lifetime of flash memories, a technique called "wear levelling" exists. It equally distributes write operations across the whole device even if the same storage address is accessed again and again. Wear levelling algorithms may move infrequently changed data (or static data) to already worn out blocks in order to make use of relatively vital eraseblocks for data which has changed recently. Paper [28] describes different strategies for wear levelling, but mainly concentrates on algorithms and 11

17 Layers SM / xd Flash Card USB Drive MTD UBI Application Application Application Application Application Application OS Linux File System Flash Management Device Block File System Block Device Block File System Block Device Block File System Block Device Flash File System, Block Mapping MTD Bad Block H. UBI Flash FS or Block FS UBI Block Mapping MTD Bad Block H. Driver MSC MSC MSC NAND, NOR NAND, NOR Firmware Card Reader Flash Controller Driver Flash Management Driver NAND Card Protocol Flash Management NAND Flash Management NAND Flash Hardware NAND Flash NAND Flash NAND Flash NAND, NOR, OneNAND NAND, NOR, OneNAND Flash Management: Bad Block Handling + Wear Levelling + Block Mapping Algorithm (surrounded by black line) UBI: Logical Volume Manager for Embedded Flash MSC: Mass Storage Device Class (over USB or Firewire IEEE 1394) Figure 2.2: Layer model of Linux flash memory architectures data structures to increase flash lifetime and does not deal with real-time aspects. Most vendors implement wear levelling techniques on the internal microcontroller of their flash memories, but keep source codes and algorithms secret. Smart Media and xd-picture Cards do not provide any wear levelling because they do not include a controller. The simplest approach to implement wear levelling in software is to emulate a block device using a flash translation layer (FTL) as described in section Another approach is to develop specialized flash file systems which handle the wear levelling problem and care about the eraseblock size and other specifics of the flash memory, but their use only makes sense when no additional wear levelling mechanisms like flash translation layers or hardware based wear levelling are in place. Stacking several wear levelling layers would only decrease performance, so it should be implemented in exactly one layer. DiskOnChip devices like SanDisk mdoc H3 implement the TrueFFS file system as firmware on the integrated microcontroller. Even though SanDisk advertises all their DiskOnChip devices with buzzwords like wear levelling, the mdoc H1 only implements the TrueFFS file system as a software driver on the host system, in order to keep the hardware price low Flash Translation Layer The flash translation layer (FTL) [29] is a bridge between a Linux block device and a memory technology device. It is situated above the physical layer and is part of the PCMCIA specification because it was designed for PCMCIA flash devices on laptops. The FTL transparently maps blocks of a block device to physical erase units on the flash device. It takes care of wear levelling and therefore allows the transparent use of traditional block-oriented file systems on raw flash memories which do not provide any flash specific wear levelling mechanisms. This block device approach is highly inefficient because no matter how little data is written, any write 12

18 access to the device requires erasing a whole flash erase unit. If the data units written are smaller than one erase unit of the flash device, and the file system is not aware of the underlying hardware, it uses the hardware in a very inefficient way. The FTL is now deprecated because of the above problems and patent issues. It is not further developed and only supported by the Linux kernel for backward compatibility [30]. Modern approaches use log-structured flash file systems instead, which directly take care of wear levelling and other specifics of the underlying flash hardware. Some of these file systems will be presented in section Unsorted Block Images Unsorted block images (UBI) is a new logical volume and block management layer which relies on the memory technology device layer. It hides physical flash type, wear levelling, bit flipping issues and bad physical erase blocks (PEB) from upper layers and transparently provides generic UBI volumes. Such a UBI volume is seen by the application or file system as consecutive logical erase blocks (LEB) which support three flash operations: read, write and erase. On top of UBI volumes it is easier to implement block device emulation layers (similar to FTL) or flash file systems because the upper layers do not have to care about wear levelling or other specifics like the size of physical erase blocks. Further information about UBI concepts can be found in the UBI design paper [31] and the UBI Frequently Asked Questions [32] Erase before Write If a flash device makes use of the erase before write technique, any write access to a sector (512 bytes) on a flash device, causes an erase unit (16 to 128 Kilobytes) to be erased [33]. This technique has good timing predictability because the worst case delay for writing a sector is always the delay of erasing one erase unit plus the delay of modifying the data in the sector. MMC devices like the Samsung MultiMedia card MC56U032NCFA [12] implement the erase before write technique. Section (Memory Array Partitioning) of the corresponding data sheet explains that these cards always perform an implicit erase before write. There are good reasons to do this because NAND flash only allows a few write operations to a sector until it needs to be erased and flash generally does not allow resetting single bits to logical one without erasing the whole unit. 2.5 File Systems for Flash The following section will shortly describe file systems commonly used on flash devices. FAT and FAT32 are traditional block file systems which were originally designed for use on hard drives, but are now also widespread in flash storage and implemented in embedded devices which provide mass storage slots. The other file systems ExFAT, JFFS2, YAFFS2 and LogFS were specifically designed for use on flash devices. 13

19 2.5.1 FAT/FAT32 The most common file system used on removable flash memories is the File Allocation Table (FAT or FAT32) file system which is supported by many platforms and provides long filename support. For compatibility reasons, most digital cameras and operating systems can read and write this file system and it is used on a lot of removable flash devices, even though it does not care about wear levelling. This is not an issue because most of today s removable flash devices do wear levelling in firmware and can be seen as a block-oriented mass storage device to the outside world. FAT and FAT32 do not provide any access restrictions, therefore anyone with access to a FAT file system can create, alter or delete any file on it. There are some security mechanisms defined in the flash memory standards which work on the physical device layer, but these approaches have not become popular yet. There are also some memory devices (e. g. USB keydrives) which are bundled with software that adds security on the file system layer by creating an encrypted and an unencrypted partition. This vendorspecific software often runs only on a single operating system which nullifies the wide compatibility of the FAT file system and makes these partitions unreadable on other operating systems or devices. With regard to compatibility, FAT is the best choice for most modern flash memories which do internal wear levelling, but when using the FAT file system on a flash memory device without any wear levelling mechanism, the following problem occurs: The file system driver frequently changes the file allocation table at the beginning of the device. This happens at least each time a file is created, resized or deleted. If there is no wear levelling layer below the FAT file system, the same flash block will likely get updated a lot of times and worn out quickly which will render the flash device unusable very soon. This is why the FAT/FAT32 is often combined with software or firmware to provide a simple form of wear levelling and equally distribute write operations across the physical blocks of the device. Any file system can then work the same way as on other block devices because it sees the flash device as a logical block device which indirectly accesses the real physical device through the wear levelling component ExFAT The extended File Allocation Table [34] has been developed for mobile devices with non-removable NOR or NAND flash memory. It is an improved version of the FAT file system which supports files larger than 4GB and allows the file system to be customized [35] for device specific parameters like the physical erase block size of flash devices. Additionally, devices using an exfat file system may optionally implement a transaction-safe FAT file system (TFAT) [36] which increases reliability and allows crash recovery on interrupted file system operations. It is not recommended to use TFAT on removable devices in combination with desktop operating systems which do not know about TFAT because then the file system is treated like FAT and file system operations are not transaction-safe. TFAT manages two copies of the file allocation table and reroutes the FAT chain on file modifications. This causes a performance decrease in comparison with FAT JFFS2 The Journalling Flash File System is a log-structured file system based on a simple log-structured file system (LFS) which has been released by Axis Communications AB in Sweden in 1999 under the GNU 14

20 General Public License. JFFS only supported one inode type (raw inode) and therefore lacked support for hard links. Later, an advanced, portable version named JFFS2, with several inode types (raw inode, directory entry, clean marker) has been developed. JFFS2 [3] is designed to work on NOR and NAND flash chips and provides error correction (ECC) which is obligatory for NAND flash. The file system accesses flash devices through the Memory Technology Device (MTD) subsystem of Linux and takes care of wear levelling and bad block management. This avoids the inefficiencies which would occur with a journalling file system on top of the flash translation layer, which is some kind of journalling file system by itself. JFFS2 supports compression on the file system level. With file compression enabled, it is difficult to predict the duration of file system operations. Additionally, it provides a background garbage collection task which is triggered when necessary and reclaims unused erase units marked for deletion. Such a background task may cause unpredictable delays or even deadlocks, which makes it impossible to guarantee worst case delays for file operations on JFFS2. There is a problem that JFFS2 write access operations will hang during garbage collection, which has been confirmed on the Linux MTD mailing list by David Woodhouse, the maintainer of JFFS2 [37]. Another deficiency of JFFS2 is that it is only suitable for smaller flashes. It does not store a central index on flash, but each JFFS2 node contains all index information about itself. At mount time, it performs a multi-stage process including a full scan of the physical device in order to gather all inode information and store it in RAM, which is much faster than flash. For all nodes, the CRC value is checked and only the valid nodes are stored in inode cache structures, collected in a hash table [3]. The full scan requires a lot of time and largely depends on the size and read performance of the flash device. It also requires a lot of memory to hold the inode cache structures in RAM. Small embedded systems might not have enough RAM to cache all inode information and successfully mount a large JFFS2 partition containing multimedia files. JFFS2 prolongs flash lifetime by implementing wear levelling and garbage collection, but yet noone has proven the formal correctness of the JFFS2 garbage collection algorithm, and as long as it is not proven to be deadlock-free, JFFS2 is not a good choice for real-time systems. The above-mentioned problems make the JFFS2 file system unsuitable for hard real time systems which require big flashes or have small RAM. On power-failure, the file system needs to be remounted and the complete device scanned until the file system gets ready. This would make the file system unavailable for several seconds or even minutes which might lead to missed deadlines. Since JFFS2 does not scale well and only works on small flashes, it will likely be superseded by advanced flash file systems like JFFS3 [38] or LogFS [39] when they have reached production state. Both of these file systems use hierarchical tree structures and are designed to work on large flashes, save memory and aim to reduce mount time by implementing check point nodes which store accumulated information about inodes. The design papers do not discuss real-time issues, but there are already extensions [40] to JFFS3 and any further improvements are welcome on the linux-mtd mailing list LogFS LogFS [39] is a new log-structured flash file system designed to be scalable and provide an efficient, deterministic garbage collection algorithm which never uses more space than it reclaims. It aims to 15