LDA, the new family of Lortu Data Appliances

Transcription

1 LDA, the new family of Lortu Data Appliances Based on Lortu Byte-Level Deduplication Technology February, 2011 Copyright Lortu Software, S.L

2 Index Executive Summary 3 Lortu deduplication technology 4 What is deduplication? 4 Benefits of deduplication technology. 5 How does deduplication differ from other similar technologies? 6 How does Lortu deduplication technology differ from other deduplication technologies? 7 Post-process deduplication vs. in-line deduplication: 7 Byte-level differencing vs. pattern matching (storing a hash for each pattern or block): 7 Data agnostic vs. content-aware approach: 7 Lortu backup solutions 8 LDA (Lortu Data Appliance) 8 Procedure of backup 8 1-Generation of backups 8 2-Deduplication 9 3-Replication (optional feature) 9 Compaction process and policies of retention 11 Remote replication process: 11 Alerts and reports 12 Management of user accounts 14 Language configuration 14 Network and security configuration 14 Models of LDA appliances 15 LDA-Mini. 15 LDA1. 15 LDA2. 15 About Lortu 16 Lortu: innovation as a way of working. 16 Copyright Lortu Software, S.L

3 Executive Summary According to analysts, all data storage requirements have been growing exponentially in recent years. The trend indicates that they will grow further in coming years. This situation is linked to new data retention laws being implemented in Europe and the United States which require companies to store ever more information, and to maintain at least some of this information stored in remote locations. This situation contrasts with the limited storage space available, the limited bandwidth available for remote replication, and high energy costs involved in maintaining this information in data centers. The response of the data storage market to this reality is a new technology called deduplication which allows a high level of compaction of the information. Deduplication solves the main problems of the data storage market: 1. It reduces the space required to store information. 2. It allows the transfer of large amounts of information via conventional Internet connections (e.g. ADSL) 3. It is a green technology which greatly reduces the energy cost of a data center. However, not all deduplication technologies work in the same way and provide the same benefits. Lortu is a pioneer of deduplication technology, as we have been working on this technology since Our deduplication technology allows you to compact the data around 100 times. In this white paper we will explain how Lortu deduplication technology differs from other deduplication technologies, and the main features and benefits offered by the new LDA family of appliances. Copyright Lortu Software, S.L

4 Lortu deduplication technology What is deduplication? Deduplication (sometimes called Single-Instance Storage, Capacity Optimization or Factoring) is a data reduction technology intended to eliminate redundant (duplicate) data on a storage system by saving only one instance of each data item, in order to reduce disk space and network bandwidth. Deduplication technologies rely on an index which tracks the data in the repository and allows for the identification of data redundancy. The management software will look at the new data, compare it to the data which already exists on the system, and then store only the data which doesn't match existing data. For example, suppose that a company has 100 members and the mailbox of each member has around 1GB. However, most of the s are the same: s distributed and forwarded among company staff or s sent to several staff members from outside. That's 100 GB of disk space consumed to store basically the same information. Data deduplication ensures that only the unique data is saved to disk. Subsequent iterations of the data are only saved as references which point to the saved copy, so that end-users still see their own files in place. Copyright Lortu Software, S.L

5 There are three kinds of deduplication technology: File deduplication. Only one copy of each identical file is stored. This technology is also known as Single File Instance technology. Block-level deduplication. Divide the information into blocks and only one copy of each identical block is stored. Byte-level deduplication. Analyze the content of the information to be deduplicated at byte-level and store only the unique data. This is the only technology which guarantees fully redundant elimination. This means that different deduplication technologies can provide different granular control by removing redundant portions of files down to the block level or even to the byte level. When evaluating a deduplication product, it's important to understand the granularity offered by the platform. Benefits of deduplication technology. By not storing duplicate pieces of data, potentially huge savings in disk space result. For instance byte-level deduplication technologies can reduce the total amount of stored data by a ratio of 50:1 or more, depending on the environment. In other words, if you are keeping a terabyte of disk backups today, tomorrow that number reduces to 20GB. And the 980GB of storage that is left over means you can defer additional storage purchases for years before you will need to add more disks to your storage capacity. This also means that if you free up more storage capacity, you can choose to keep data online because it can be sent via secure WAN to remote sites for disaster recovery purposes or replication. Copyright Lortu Software, S.L

6 How does deduplication differ from other similar technologies? Data deduplication differs from compression in that compression looks only for repeating patterns of information and reduces them. For example, a compressed file cannot be compressed again because it has huge entropy. Data deduplication reduces the unique data regardless of its internal format. It just compares the content of the file with previous versions and extracts the new unique data. This provides a much greater data reduction capability than compression. In fact, most of the products apply compression algorithms after deduplicating the data to get an even higher data reduction. Deduplication also differs from incremental or differential backups in that only the bytelevel changes are backed up. Incremental backups scan selected files for changes. If there is a change in the file, even of a single bit, the whole file is saved in the newest backup file. If that file is 500 MB, it saves the whole file to the new backup. Datadeduplication technology will only store the pieces of data that have changed, not the entire file. Copyright Lortu Software, S.L

7 How does Lortu deduplication technology differ from other deduplication technologies? There are several approaches to implement deduplication, and even though each approach has its own pros and cons, some are much better than others. The main differences between the approaches: Post-process deduplication vs. in-line deduplication: The main advantage of post-process deduplication as opposed to in-line deduplication is a higher backup throughput and smaller backup time window. This is because the information is first stored in the appliance and then deduplicated later without interfering with the backup process. Lortu provides post-process deduplication. Byte-level differencing vs. pattern matching (storing a hash for each pattern or block): Pattern matching is less scalable than differencing as the data to be deduplicated grows, because the table with hashes uses more memory and CPU as it has to manage more data. However, its greater drawback is the restore time. If backup time is critical, the restore time is much more critical. Since the patterns are spread over the full disk in very small blocks of information, the system requires reads of one or two clusters for each small pattern. This means that restore time can be more than 10 times slower than copying the non-deduplicated information. With byte-level differencing, the information is stored in much larger blocks, and usually the restore time is very close to copying the non-deduplicated information. Also pattern matching technology requires several weeks before the deduplication process can be effective. With byte-level differencing the deduplication is very effective from the second backup, and effectiveness improves as new files are included in the vault. Lortu provides byte-level differencing deduplication. Data agnostic vs. content-aware approach: Data agnostic technologies work with any kind of information or file format. The drawback of the content-aware approach is that the technology needs to understand the format of the files. If the file format is different than expected (a new version of the application for instance), or if the application isn't supported by the technology, the deduplication process is not possible. Lortu deduplication technology is agnostic to the data. It can deduplicate data of any kind, any file format or file type. Copyright Lortu Software, S.L

8 Lortu backup solutions Lortu provides not only backup appliances for local storage but also a remote backup service, allowing to our customers to have a complete backup solution. It is based on a Disk to Disk to Cloud backup solution, without requiring our customers to purchase another appliance for remote backups or to have their own remote facilities. LDA (Lortu Data Appliance) The LDA is a massive storage device specially designed to store daily backups for months, and to replicate them to a remote server using a conventional Internet connection. For example, the model LDA2 stores around 100 backups of 1200GB locally and remotely, with a capacity of up to 250TB of data. This huge storage capacity is obtained due to the deduplication technology, designed and developed by Lortu, which is integrated into our appliances. The LDA is a massive storage data device, which can be used along with any existing backup applications in the market. In other words, it is the perfect substitute for conventional tapes and storage media. Procedure of backup The backup process has the following stages: 1-Generation of backups The generation of backups can be done by any existing backup application on the market. It is necessary to schedule a task in the backup application for each server to be backed up. Copyright Lortu Software, S.L

9 It is recommended, though not essential, to schedule the tasks so that the backup of each server can start once the previous server backup has finished. The way to indicate to the LDA that the backup of the last server has finished is by creating a file called backup.txt in the root of the LDA shared folder. The creation of this file acts as a trigger. As soon as the LDA detects the existence of this file, it deletes the file and begins the deduplication process, compacting the new backups stored in the LDA. If the trigger file is not created, the LDA automatically starts the deduplication process 12 hours after the previous deduplication process has finished. The trigger file backup.txt can be empty or can contain any information. It is important to make sure that the compression and encryption options are deactivated in the backup application. 2-Deduplication This process is done automatically inside the LDA. It does not require the intervention of the user. The deduplication process consists of comparing the data contained in the previous day s backups with all the data stored up to that time in the LDA. When this process ends, the duplicated information is replaced by references, and only the new information is stored in the LDA. The deduplication technology developed by Lortu makes this comparison at byte level, which allows maximum compaction. Once the new information is stored in an internal compartment of the LDA called the vault, all the deduplicated files are virtualized, releasing the space used by the stored daily backup files and leaving space for the following day s backups. When the LDA reaches the limit of its capacity, it automatically eliminates the older backups to leave space for new backups, based on the retention policies defined by the user. 3-Replication (optional feature) The last phase of the process is the replication of the daily backups to the remote replication server. The LDA does this in the following steps: 1- Extraction of the new information stored that day. 2- Compression of this information. 3- Encryption of the information using the AES algorithm. 4- Uploading the file with the replication data to the replication server. Copyright Lortu Software, S.L

10 Once compressed and encrypted, the size of the file that is sent to the remote replication server is on average 100 times smaller than the stored size. Thus a backup of 1200GB usually is compacted to 12GB. In the case of structured information (SQL, Oracle, Exchange Server, etc.) the compaction ratio can be double this, or even more. Copyright Lortu Software, S.L

11 Compaction process and policies of retention Whenever a deduplication process is finished, the LDA verifies whether it has sufficient free space to store at least one full backup without deduplication. If there is not sufficient free space, it begins a compaction process and deletes files in the following order: The oldest files are eliminated until there is sufficient space. In addition, the user can define retention policies from the web console, so that some old files are not eliminated. The LDA can apply the following types of retention policies: daily, weekly, monthly, quarterly and yearly. Remote replication process: The LDA can optionally replicate all the data added to the vault in each session to a remote location. The user can customize the following features from the web console: The list of the files and folders to exclude from remote replication. Bandwidth control. The user can configure which hours the LDA will operate in bandwidth control mode. (For example, during working hours or hours in which there is intensive bandwidth consumption by other servers.) The user can configure the following parameters: o Hours in which the restriction is applied. o Percentage of bandwidth which the LDA can use during these hours. o Apply the restriction only on working days (Monday through Friday). In the present implementation the LDA replicates to an IIS server, but in future versions we will extend the supported replication systems to other appliances and to cloud computing systems. The replication server has an IIS component which is in charge of receiving the packages from each appliance and storing them on disk. Copyright Lortu Software, S.L

12 Alerts and reports Each appliance sends reports and information about its configuration to the central server. The central server uses this information to generate and send alerts and reports. It also manages the updates for each appliance. The administrator can define which users will receive report and alert messages via the web console. Every day the system sends a message with an activity report, and if there are any errors, it sends a report of errors. This report additionally includes three graphs: Graphic of backups. This shows the size of the daily backups stored by each server. It allows you to visually determine important increases or decreases in the size of the daily backups. Copyright Lortu Software, S.L

13 Graphic of replication. This shows the process of replication for the daily backups. It allows detection of connection problems or bottlenecks in the network. Graphic of processes. This shows the time used by each internal process of the LDA. Copyright Lortu Software, S.L

14 Management of user accounts The LDA has two types of users: Console administrator. Can access all the pages of the console. In addition he can create new users and reset their passwords. Console user. Cannot access the network configuration page or the account management page. In addition, the administrator can create other restrictions for these users in the shared folders of the LDA, so that they can access only one or several folders. Language configuration The user can configure the language from the web console. This configuration affects the following components: Web console Local reports on the LDA Reports and alerts generated by the central server. Network and security configuration The user can configure the next options: o o Name of the Appliance Network configuration: Static IP or DHCP Mask Gateway DNS 1 DNS 2 o Firewall configuration: You can define the IPs or ranges of IPs that can have access to the LDA. Copyright Lortu Software, S.L

15 Models of LDA appliances LDA-Mini. Configuration Details: Chassis: NETTOP (190mm / 135mm / 25mm). Maximum daily capacity: 80GB Total Estimated capacity: 20TB Low noise and power consumption. LDA1. Configuration Details: Minitower chassis Maximum daily capacity: 300GB Total Estimated capacity: 80TB Storage: RAID 1 LDA2. Configuration Details: Chassis: RACK Maximum daily capacity: 1200GB Total Estimated capacity: 250TB Storage: RAID 6 Redundant NIC and power supply Copyright Lortu Software, S.L

16 About Lortu Lortu is a company dedicated to the development of storage appliances based on our proprietary data deduplication technologies. In 2003 Lortu Software began to investigate a technology which could drastically reduce the space needed for storing backups by replacing duplicated data with references. Lortu called that technology Kondar. The data storage industry did not have a commonly accepted term to describe this technology for several years. It was in the second half of 2006 when this technology began to be known as deduplication in the computer magazines. Lortu: innovation as a way of working. Lortu not only anticipated the solution to a future need by several years, we also followed up by innovating in both technology and marketing. In 2006 we began to commercialize our technology as components to be integrated into other software or hardware products. In 2007 we developed our first backup appliance, and decided to take a further step of innovation by becoming the first deduplication manufacturer in the world to offer its technology as a managed backup service. In 2011 we are taking another important step with our new family of data appliances (LDA), which provide improvements in key areas such as: performance, scalability, vault virtualization, in addition to several commodity features like reports, alerts and powerful retention policies. Additionally, we offer our deduplication technology to storage manufacturers and customers with special requirements in order to integrate one of the best byte-level deduplication technologies on the market into custom storage systems. Deduplication, like other areas of technology, is constantly evolving. At Lortu we are always working to offer the most innovative solutions to our customers and partners, as a pioneer company in this field. Copyright Lortu Software, S.L