Energy Consumption Analysis of ARMbased

Size: px
Start display at page:

Download "Energy Consumption Analysis of ARMbased"

Transcription

1 Aalto University School of Science Degree Programme of Mobile Computing Bo Pang Energy Consumption Analysis of ARMbased System Master s Thesis Espoo, Augest 31, 2011 Supervisor: Instructor: Professor Antti Ylä-Jääski, Aalto University Zhonghong Ou post-doc researcher

2 Aalto University School of Science Degree Programme of Mobile Computing Author: Bo Pang Title: Energy Consumption Analysis of ARM-based System ABSTRACT OF MASTER S THESIS Date: Augest 31, 2011 Pages: 68 Professorship: Data Communications Software Code: T-110 Supervisor: Instructor: Professor Antti Ylä-Jääski Zhonghong Ou post-doc researcher With growing computing demand, energy consumption is standing out as a salient concern of data centers with large numbers of servers. To build high energy-efficient data centers, the low power consumption ARM processors have been investigated as an alternative solution to replace the conventional high-end server processors e.g. Intel and AMD processors. This study exploits this concept and measures how much energy saving can be achieved by adopting ARM processors. We design four different experiments representing WEB server applications, In-memory database, video transcoding applications and Hadoop to evaluate whether ARM based server is able to provide the same level of service in a more energy-efficient way, and which application is more suitable for ARM based server. Additionally, the lesson learn from this study is discussed in the end of this study. Keywords: ARM Processor, Energy Efficiency, WEB, In-memory Database, Video Transcoding, Power Management Language: English 2

3 Acknowledgements First of all, I want to thank to my motherland, China, who gives me the opportunity to study abroad. Without the reform and opening-up policy, nothing I have done today would be possible. I would like to thank Aalto University for its two years high quality master education, from which I will benefit for my whole life. I also would like to express my sincere gratitude to my supervisor Professor Antti Ylä-Jääski and instructor Zhonghong Ou, who not only patiently guide my thesis writing but also teach me how to become a qualified scientific researcher. After many years, I may forget what this thesis is about, but the research attitude he taught me will always be remembered. At last I would like to thank my parents and my girlfriend, their consistent support helps me get through all these two years study. Without the faith of going home to reunite with them, I would not graduate on time. In the end, I want to thank all my friends, especially my good brother Horoscope Zhu who guided me in Matlab and Dota earnestly, accompanied me to pass rememberable days in Otaniemi. Espoo, Augest 31, 2011 Bo Pang 3

4 Abbreviations and Acronyms HDD EE LVS ARP MAC HDFS RAID OS HTTP Hard Disk Drive Energy Efficiency Linux Virtual Server Address Resolution Protocol Media Access Control Hadoop Distributed File System Redundant Array of Independent Disks Operating System Hypertext Transfer Protocol 4

5 Contents Abbreviations and Acronyms 4 1 Introduction Motivation and Challenge Technology Trends Problem Statement Scope of the Thesis Structure of the Thesis Background ARM Structure Web Server Technologies Apache Nginx LVS Multimedia Transcoding Libraries on Linux In-memory Database Hadoop Experimental Environment Pandaboard Specifications Intel Workstation Test Methodology Experimental Design HTTP Server In-memory Database Video Transcoding Hadoop Map-Reduce Task

6 5 Result Analysis HTTP Server Experiments In-memory Database Experiment Video Transcoding Experiment Hadoop Experiment Discussion 59 7 Conclusions Key Findings Limitations of Study Future Work

7 Chapter 1 Introduction 1.1 Motivation and Challenge Along with information technologies playing more and more important roles in modern society and economic, information processing becomes a common need for a variety of industries. New Internet services such as video streaming, social network sites, data mining and business intelligence and Internet search optimization require a huge amount of computing resource. Normally the scale of computing can not be achieved by workstations or personal computers, thus distributed servers cluster system built of thousands of servers in data center is a smart choice for massive-data processing, some Internet giant companies such as Google, possessing more than 200,000 servers from estimation in 2005, use a great amount of servers to build a distributed system to meet the big computing demand. However the significant growing number of servers makes power consumption stand out as a salient issue. The sum of electricity spending on servers computing, data processing, data storage and device cooling is becoming a big concern for world-spread data center owners. From Figure 1.1 below, the money spent on power supply and cooling is going to overtake the upfront investment on servers hardware, hence the reduction of power consumption on server is becoming a new concern of data center investors. Nowadays data center electricity consumption becomes a big challenge for the world. From Jonathan study[15], power consumption used by servers doubled from 2000 to And in 2005 approximately 0.6% electronic power was consumed by servers of all kinds in US, which is equal to the amount of electricity consumed by all the color televisions in US. In addition to the power directly consumed by servers, the same amount of power was used by auxiliary and cooling systems, which provide stable power and cooling to 7

8 CHAPTER 1. INTRODUCTION 8 Figure 1.1: IDC prediction of server spending in 2006 [14] processors to keep the servers working stably. The total data center power demand in 2005 required about five 1000 MW power plants to be built in US, and fourteen plants in the world. Moreover IDC also predicted the total electricity consumption on servers worldwide will be $44.5 billion by 2010 which requires at least 10 gigawatt power plants to be constructed for that[5]. Thus power saving solutions for computing may bring significant economic profit to data center and also protect the environment. 1.2 Technology Trends ARM, a micro processor architecture originally designed for mobile and embedded systems, starts to step into server market recently. Microsoft first announced ARM-based PC at the Computer Electronics Show (CES) in Las Vegas[17]. In addition, ZT Systems[51] announced a server powered by sixteen ARM cortex-a9 processor cores, which draws a maximum 80 watts system power that is even less than an Intel Xeon series processor s power consumption. Hence there is a new trend for data centers to adopt multicore ARM processors to build energy efficient and cooling efficient servers to execute computing tasks that originally executed by Intel processors based

9 CHAPTER 1. INTRODUCTION 9 servers. Because of the recent server cost s shrink, ARM processor s notable advantage of air-cooling and low power-consumption is becoming more important to data centers investment. And another motivation to adopt ARM processor is to solve the problem of increasing core density on processor, for accumulated devices, such as rack-mounted blade servers, the ARM processors low heat generation characteristic and scattering layout helps decrease the complexity of motherboard integration and cooling design. Nowadays most of data centers are adopting X86 architecture processor. From Figure 1.2, it is obvious that Intel Xeon series processor occupied roughly two thirds of server market, while the high-end server processor IBM power series penetrated 20% shares and AMD, Intel s strongest competitor, seized only 8.5%. Comparing with X86 architecture processor, the ARM based-server is still a new participator in the server market, only 2.3% users chose it. Thus replacing current high power consuming processors with ARM processors may bring significant potential power saving to data centers, ARM processor based server is a good candidate for leading a new evolution to high energy-efficiency computing. Figure 1.2: Server market share of China in 2010

10 CHAPTER 1. INTRODUCTION Problem Statement ARM-based server is upcoming, but there is no scientific research proved the feasibility and how power efficient ARM server would achieve. Thus the goal of this study is to answer the following questions: Is ARM-based servers/server cluster able to substitute Intel Xeon servers with the same performance but lower power consumption? Which kinds of applications are more suitable for ARM-base servers/server clusters? For answering these two questions, a ARM processor-based testbed, consisting of four Dual-core ARM Cortex-A9 processors, is built, running the same task against an Intel Server based processor to compare their power consumption and performance. To compare these two different platforms in a fair manner, we use the energy efficiency (EE) definition from Dimitris s paper[48]: the ratio of useful work done to the amount of used power. EE = Work done Energy = Work done Power Time = Performance Power (1.1) The EE will be a benchmark thoughout all the experiments in this study. 1.4 Scope of the Thesis This thesis focuses on metering and comparing power efficiency of ARM processor based server and Intel processor based server in some specific scenarios and analyses which kinds applications or services are better fit the ARM processor based system. In this thesis four sets of experiments are conducted on two platforms, which represent the following applications: Web application, In-memory SQL database, video transcoding application and Hadoop applications. 1.5 Structure of the Thesis This paper is organized as follow: chapter 2 introduces technologies and applications that are used as test objects in this paper, and a brief overview of their working principles are also given. And chapter 3 describes the test environment of experiments. Chapter 4 outlines the design details of each

11 CHAPTER 1. INTRODUCTION 11 experiments with brief descriptions about the involved benchmarks. Chapter 5 presents the experiment results and analyses the energy efficiency of both platforms in each experiment. Chapter 6 makes a discuss of strategy that will further lower the power consumption of the ARM processor based server cluster. Chapter 7 draws a conclusion and discusses the limitations of this study and plans the future work for continuing this work.

12 Chapter 2 Background This chapter introduces background information about technologies related to this study, including architecture of ARM processor and other application technologies which are going to be used as test tools or objects in this work. 2.1 ARM Structure The ARM is a 32-bit reduced instruction set computer (RISC) instruction set architecture (ISA) developed by ARM Holdings company [3]. The ARM Holding company only designs the architecture of processors and licenses other manufacturers to produce processors based on its design. The licensee list includes IBM, Texas Instruments, Samsung, etc, almost every big ICT manufacture company has involved in ARM architecture processor producing. The reason why so many producers choose ARM architecture is because of its advantages: low power consuming and simplicity. ARM s RISC keeps every instruction simple and of uniform length, the special design, unlike the complex instruction, allows RISC processors demanding less registers and circuits. Moreover the uniform length instruction offers ARM processor better performance from instruction pipeline technology that exploits processing circuit at its maximum efficiency. Thus the ARM s simple architecture generates less heat than Intel X86 architecture [12] but is still able to achieve good performance. Thanks to ARM processor s unique design, ARM counts on its high energy efficiency to dominate embedded system market [4], for instance, the prevailing Apple products, Iphone and Ipad families are all carrying ARM processors, moreover many mainstream operation systems has its ARM distribution, even Microsoft announced in Jan 2011 that it planned to support ARM-based system [2]. With several years accumulation and evolving, ARM processors gained enough computing power to participate 12

13 CHAPTER 2. BACKGROUND 13 server-end computing competition. For example, ARM cortex-a15, which contains four 2.5GHZ cores, is equal to Intel Core i7 2920XM processor at clock rate. Thus it is possible to adopt ARM-based processors to build a multi-processors server to overtake the jobs processed by Intel Xeon series servers. According to the news from industry, several manufacturers have already built their own ARM-based servers, for instance, ZT system proposed its R1801e 1U rackmount ARM server [51], which uses eight ARM Cortex-A9 processors with solid state disk. Surprisingly the total power spent on the whole server is less than 80W on 16*2GHz computing speed. All in all ARMbased server is becoming a competitive candidate for low power consumption servers [31]. 2.2 Web Server Technologies Apache Apache HTTP Server ( httpd ) [42] is a project under Apache Software Foundation, it is one of most popular Web server in the world and served for more than 224 million websites by June of 2011 [28]. Moreover it can be installed on a variety of operating systems, such as Linux, Solaris, Mac OS X and Microsoft Windows. What is more, Apache also provides an open interface for plug-ins, which enhances Apache with useful features. Installing the PHP plug-in, Apache obtains the ability to host PHP Web applications Nginx Nginx [29] is an open-source high-performance HTTP server and reverse proxy, which was first developed by Igor Sysoev in Nginx has a stunning performance efficiency when handling static-resource requests, and it is also able to be used as a load balancer. Since Nginx belongs to message-oriented asynchronous servers, unlike httpd, Nginx does not spawn new processes for each incoming connection, but rather uses a threads pool to handle incoming request events. This design uses a fixed number of worker threads and thus consumes limited memory resource when handling massive http connections. According to an experiment made in 2008 [13], Nginx defeated Apache at processor load and response time respectively.

14 CHAPTER 2. BACKGROUND LVS Linux Virtual Server (LVS) [21] is an advanced Linux load balancing solution, which provides server cluster high-scalability, high-performance and highavailability load balance service. LVS is an IP layer load balance software, thus it is able to serve any application beyond IP layer. LVS supports three load balance modes and eight schedule algorithms. The three load balance modes include Virtual Server via Network Address Translation (VS/NAT), Virtual Server via IP Tunneling (VS/TUN) and Virtual Server via Direct Routing(VS/DR). The VS/NAT mode is the simplest. Application servers are assigned private IP addresses, and LVS in this mode plays a role as a Network Address Translation server and also the only gateway between the server cluster and clients. When clients send requests to the LVS, it will look up in its server address table to find a real server in the cluster and map the destination public address to the private address of the real server, and then sends the request to the real server directly. And when the real server replies to the client, the LVS will translate the address back and deliver the reply package back to the client outside the private subnet. The VS/TUN mode works similar to virtual private network (VPN) tunnel mode. The LVS adds a new IP destination header, containing the real server s IP address, in front of the incoming IP package. Through the LVS tunnel the new header will be removed by the tunneling protocol on the real server, and after unpacking the payload of the request, the real server processes the request and replies clients directly. And the last mode is VS/DR which, unlike the former two modes, works at data-link layer, in this mode all the real servers and LVS share the same IP address, but only LVS replies the Address Resolution Protocol (ARP) requests from the gateway, so all the requests are forwarded by gateway to LVS and LVS replaces the request s destination MAC address with one certain the real server s and forward it to that real server, and the real server will reply clients directly. Corresponding to these three modes, eight schedule schemes decide which real server would be chosen to reply requests. The eight schedule schemes are listed as follows: Round-Robin Scheduling Weighted Round-Robin Scheduling Least-Connection Scheduling Weighted Least-Connection Scheduling Locality-Based Least Connections Scheduling Locality-Based Least Connections with Replication Scheduling

15 CHAPTER 2. BACKGROUND 15 Destination Hashing Scheduling Source Hashing Scheduling Round-Robin schedule strategy polls real servers list and selects one fairly, this scheme is the most fair scheme, while other schemes are suitable for capability-unbalanced server cluster. In this paper s experiments, LVS in VS/NAT mode with Round-Robin schedule strategy is selected as front-end load balancer configuration. 2.3 Multimedia Transcoding Libraries on Linux Nowadays, video broadcasting Web service is becoming an inseparable part of people s lives. Those video broadcasting websites, such as YouTube and YouKu, allow users to upload their own videos in various formats and transcode them into uniform Flash compatible format and present the videos on their pages to viewers. This thesis will study the transcoding process as a benchmark for energy efficiency comparison. On Linux, there are two popular multimedia transcoding libraries/tools. The first is MEncoder [47], which is included in MPlayer project. MEncoder, as MPlayer s codec library, supports diversity of video and audio formats covering most common-use video formats. The other one is FFmpeg [46] that is also an open-source project providing library and program for multimedia data processing. FFmpeg is famous for its library named Libavcode that is a video/audio codec used by several other projects. FFmpeg supports many video and audio formats as well, and many hardware platforms are on its support list, which includes x86(ia-32 and x86-64), PPC (PowerPC), ARM, DEC, SPARC, and MIPS architecture. Besides these two tools, Java has its own media library called Java Media Framework, which enables audio, video and other time-based media to be added to applications and applets built on Java technology. This framework can capture, playback, stream, and transcoding multiple media formats and has cross-platform feature [30]. In this thesis, the MEncode is responsible for splitting and merging the video files and the video slice will be transcoded by the FFmpeg. 2.4 In-memory Database In this study, SQLite [38] is used as an in-memory database. The in-memory database is response-speed-oriented database that relies on main memory rather than hard disk. Because all the data are stored in main memory,

16 CHAPTER 2. BACKGROUND 16 there is no I/O operation needed, therefore time critical tasks and high frequency tasks can be completed in high speed by in-memory database. SQLite is a popular in-memory database, of which the biggest advantage is simplicity, but SQLite supports ACID (atomicity, consistency, isolation, durability) properties as well, as traditional database does. Although the SQLite is normally used in embedded systems and in small or middle website projects, with Partition and Shard technologies getting mature, some SQLite databases cluster may outperform a single industrial database server. 2.5 Hadoop Hadoop is an open-source project under Apache foundation for distributed, reliable and scalable computing. Hadoop, inspired by the Google s Google File System (GFS) [11] and MapReduce [8], implements most functions of GFS and Google s MapReduce. Its goal is to conduct data intensive computing on large cluster of commodity hardwares. This framework is spreading around several giant IT companies, such as China Mobile, TaoBao and Yahoo, which adopt this software for analyzing data for their business and also contribute back to the Hadoop project s source code [49]. The latest stable version is [43], which is used as test benchmark in this thesis. Hadoop is comprised of three parts: Hadoop common project, Hadoop Distributed File System (HDFS) and Hadoop MapReduce. The common project [39] is common utilities for supporting other Hadooprelated projects. This package includes library jar files and scripts to manipulate Hadoop instances and also other materials, such as source codes, documents and examples. HDFS [40] is the foundation for Hadoop Map-reduce. It provides a distributed file system for upper applications. And since it is written in Java, it gains portability to run on heterogeneous operating systems, moreover its architecture s simplicity enables Hadoop s scalability. HDFS is designed to process batch jobs rather than user interactive jobs, hence low data access latency is a less important goal for HDFS whose actual main goal is high throughput of streaming data access. Since it is not for general purpose application, HDFS only implements part of POSIX standard and ignores some requirements for better data throughput rate. The most significant feature of HDFS is its write-once-read-many file access mode, which means once a file s writing process is finished it can not be changed anymore. This feature dramatically advances the simplicity of system structure, there would not be write lock issue for designers. But in the future HDFS may support appending write operation. Thus HDFS is not suitable for storing

17 CHAPTER 2. BACKGROUND 17 data which is incline to changing, such as user s information. Instead of data representing in-time situation, system logs and historical data are suitable to be stored in HDFS and be analyzed later on. HDFS file organization is very similar to traditional directory-file hierarchy, in which user can create directory and put files under it. But there is a small difference from existing file systems, for example, it does not support soft or hard link which is prevailingly supported by most file systems. Moreover files and directories on HDFS will be split into two parts: meta-data and data file, which are stored in different parts of system. Meta-data is managed on the Name node and real data file is stored as file in local file system of Data node, those two kinds of nodes will be discussed in the s subsequent paragraphs. A node is a Java software instance running upon commodity machines. There are two types of nodes: Data node, Name node, and Name node is the core of the HDFS. Because HDFS is a star-style structure, there is a unique Name node in the center of the system, which takes charge of maintaining meta data and controlling data blocks replication (data file is comprised of series of blocks) and interacts with clients about the block locations. This design also simplifies the HDFS structure. Name node as a central server controls every movement of Data node. Data node is the real worker, which manages file s content and replication process. In HDFS files are split into a sequence of equal-size blocks (except the last block). For fault tolerance, each block is replicated in several locations (the configuration of number of replicas is stored in the Name node), and all the replicas and the original blocks are spread across the HDFS cluster, like RAID 5 working principle. Figure 2.1 depicts the block and replica distribution across Data nodes. Data node reports its status and a block list of its repository to Name node periodically, and then the Name node will give reply with dictations about hierarchy changes, such as directory changing and files deleting, etc. Figure 2.2 depicts the architecture of HDFS and communication transaction process. There are two communication protocols in this diagram, one is from client to nodes, and the other is from Data node to Name node. Both these two protocols are based on TCP/IP protocol and support remote procedure call (RPC). The protocol between Data node and Name node is used for periodically reporting. In addition to this internal protocol, as data keeper, Data node is queried directly by clients to read and write file data, and clients also need to change meta data on the Name node to update file system information, thus the client protocol is adopted for file operations. To tackle with hardware failure, HDFS s data block replication and check point mechanism helps data availability and robustness. But the Name node

18 CHAPTER 2. BACKGROUND 18 is a single point of failure, and its failure may cause the whole HDFS fails to carry on any functions. Thus a secondary Name node is essential as a backup of Name node, and manual intervention is needed as well. Figure 2.1: Distribution of replicas of data in HDFS [45] Running on the HDFS, the Map-Reduce programming paradigm [41] is the last component of Hadoop. Similar to Google s MapReduce, Hadoop MapReduce programming model is also for parallelized computing of big set of data. This programming model has three phrases: map, shuffle and reduce. In map phrase, some processes, so called mappers, will analyze raw data and output in key-value pair style into intermediate files. The second phrase is called combine or shuffle, which sorts the key-value pairs in the intermediate files and gathers the data with the same key into one node via HTTP for the following reduce phrase. Data in the reduce phrase will go through some processes, such as summing up or counting number, and output in key-value style to the final output file. From the Figure 2.3, it is obvious that one job contains several map tasks and these tasks are running on separated nodes and each one processes a piece of input file (normally a HDFS file block) and output intermediate file on the local file system. The framework will collect those intermediate files and distribute them into different nodes by keys. After that reducer will fetch the intermediate files as input, and output result into output file. Got

19 CHAPTER 2. BACKGROUND 19 Figure 2.2: Architecture of HDFS [44] the job finished signal, client can fetch its job s result file from HDFS. Hadoop implements Map-reduce by two kinds of nodes, called Job Tracker and Task Tracker. Like HDFS structure, Job Tracker manages job information and assigns tasks to Task Tracker. The design principle of Hadoop is moving computing rather than data, thus Job Track will get data file block information from Name node and assign task to the Data node, which holds the file block or the closest to the data file block node, for saving backbone network traffic. Figure 2.4 depicts the classical deployment of Hadoop: Job Tracker and Name node are deployed on one master node, and Task Trackers and Data nodes are deployed on several slave nodes. This design takes advantage that map task may compute local data block instead of fetching blocks from other slave nodes [8].

20 CHAPTER 2. BACKGROUND 20 Figure 2.3: Map-reduce paradigm [1] Figure 2.4: Architecture of HDFS [24]

21 Chapter 3 Experimental Environment This thesis focuses on comparing performance and power consumption of ARM processors and Intel processors at various application domains. For building up ARM-base server cluster, four Pandaboards are used to set up a test platform. A workstation carrying Intel processor is deployed as its adversary. This chapter is about the configurations of these two test platforms and each one s power consumption metering methods. 3.1 Pandaboard Specifications The ARM test platform is based on Pandaboard which includes one OMAP4430 processor, 1 GB low power DDR2 RAM and a 8 GB Kingston SD card. The OMAP4430, as the core logic unit of Pandaboard, contains Dual-core ARM Cortex-A9 MPCore with Symmetric Multiprocessing (SMP) at 1 GHz each, which allowing for 150% performance increase over previous ARM Cortex- A8 cores [33]. What s more, OMAP4430 also supports several image and video processing technology. Figure 3.1 depicts Pandaboard s layout and its periphery components. 3.2 Intel Workstation As the counterpart of ARM test platform, we deploy a Intel processor based workstation as the control group. The processor inside the Intel workstation is Core2-Q9400 of Xeon series. This CPU is quad-core, thus this Intel CPU could run four processes simultaneously. For reducing the difference between two platforms, the Intel workstation installs the same version of operating system and software stack to make sure 21

22 CHAPTER 3. EXPERIMENTAL ENVIRONMENT 22 Figure 3.1: PandaBoard Platform [34] both test platforms have the same application foundations. The configurations of two test beds, Pandaboard and Intel workstation, are compared in Table Test Methodology For metering ARM platform, Mosoon Power Monitor [26] is used for measuring the power consumption. Figure 3.2 shows the power monitor and a self-made power supply plug compatible with Pandaboard s socket. This device is able to supply power to Pandaboard meanwhile recording power and reporting the average value in a real time fashion. For Intel workstation power measurement, we use Mastech product: MS2102 AC/DC clamp meter. The MS2102 clamp meter is able to measure maximum 200A current with 2.5% accuracy. To isolate the power used by hard disk devices (HDD) and other periphery devices, the clamp meter attaches the 5V (red) and 12V (yellow) lines, which mainly supply CPU and its cooling fans power, to record the currents, as showed in Figure 3.3. And then

CSE-E5430 Scalable Cloud Computing Lecture 2

CSE-E5430 Scalable Cloud Computing Lecture 2 CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing

More information

Chapter 7. Using Hadoop Cluster and MapReduce

Chapter 7. Using Hadoop Cluster and MapReduce Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in

More information

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information

More information

Accelerating Enterprise Applications and Reducing TCO with SanDisk ZetaScale Software

Accelerating Enterprise Applications and Reducing TCO with SanDisk ZetaScale Software WHITEPAPER Accelerating Enterprise Applications and Reducing TCO with SanDisk ZetaScale Software SanDisk ZetaScale software unlocks the full benefits of flash for In-Memory Compute and NoSQL applications

More information

Hadoop and Map-Reduce. Swati Gore

Hadoop and Map-Reduce. Swati Gore Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data

More information

Energy Efficient MapReduce

Energy Efficient MapReduce Energy Efficient MapReduce Motivation: Energy consumption is an important aspect of datacenters efficiency, the total power consumption in the united states has doubled from 2000 to 2005, representing

More information

Hypertable Architecture Overview

Hypertable Architecture Overview WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for

More information

Accelerating and Simplifying Apache

Accelerating and Simplifying Apache Accelerating and Simplifying Apache Hadoop with Panasas ActiveStor White paper NOvember 2012 1.888.PANASAS www.panasas.com Executive Overview The technology requirements for big data vary significantly

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms Distributed File System 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributed File System Don t move data to workers move workers to the data! Store data on the local disks of nodes

More information

Big Data Storage, Management and challenges. Ahmed Ali-Eldin

Big Data Storage, Management and challenges. Ahmed Ali-Eldin Big Data Storage, Management and challenges Ahmed Ali-Eldin (Ambitious) Plan What is Big Data? And Why talk about Big Data? How to store Big Data? BigTables (Google) Dynamo (Amazon) How to process Big

More information

Hadoop IST 734 SS CHUNG

Hadoop IST 734 SS CHUNG Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to

More information

EWeb: Highly Scalable Client Transparent Fault Tolerant System for Cloud based Web Applications

EWeb: Highly Scalable Client Transparent Fault Tolerant System for Cloud based Web Applications ECE6102 Dependable Distribute Systems, Fall2010 EWeb: Highly Scalable Client Transparent Fault Tolerant System for Cloud based Web Applications Deepal Jayasinghe, Hyojun Kim, Mohammad M. Hossain, Ali Payani

More information

Benchmarking Cassandra on Violin

Benchmarking Cassandra on Violin Technical White Paper Report Technical Report Benchmarking Cassandra on Violin Accelerating Cassandra Performance and Reducing Read Latency With Violin Memory Flash-based Storage Arrays Version 1.0 Abstract

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

CLOUDDMSS: CLOUD-BASED DISTRIBUTED MULTIMEDIA STREAMING SERVICE SYSTEM FOR HETEROGENEOUS DEVICES

CLOUDDMSS: CLOUD-BASED DISTRIBUTED MULTIMEDIA STREAMING SERVICE SYSTEM FOR HETEROGENEOUS DEVICES CLOUDDMSS: CLOUD-BASED DISTRIBUTED MULTIMEDIA STREAMING SERVICE SYSTEM FOR HETEROGENEOUS DEVICES 1 MYOUNGJIN KIM, 2 CUI YUN, 3 SEUNGHO HAN, 4 HANKU LEE 1,2,3,4 Department of Internet & Multimedia Engineering,

More information

Evaluating HDFS I/O Performance on Virtualized Systems

Evaluating HDFS I/O Performance on Virtualized Systems Evaluating HDFS I/O Performance on Virtualized Systems Xin Tang xtang@cs.wisc.edu University of Wisconsin-Madison Department of Computer Sciences Abstract Hadoop as a Service (HaaS) has received increasing

More information

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1 Performance Study Performance Characteristics of and RDM VMware ESX Server 3.0.1 VMware ESX Server offers three choices for managing disk access in a virtual machine VMware Virtual Machine File System

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction

More information

International Journal of Advance Research in Computer Science and Management Studies

International Journal of Advance Research in Computer Science and Management Studies Volume 2, Issue 8, August 2014 ISSN: 2321 7782 (Online) International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online

More information

Apache Hadoop. Alexandru Costan

Apache Hadoop. Alexandru Costan 1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open

More information

Introduction to Cloud Computing

Introduction to Cloud Computing Introduction to Cloud Computing Cloud Computing I (intro) 15 319, spring 2010 2 nd Lecture, Jan 14 th Majd F. Sakr Lecture Motivation General overview on cloud computing What is cloud computing Services

More information

Purpose-Built Load Balancing The Advantages of Coyote Point Equalizer over Software-based Solutions

Purpose-Built Load Balancing The Advantages of Coyote Point Equalizer over Software-based Solutions Purpose-Built Load Balancing The Advantages of Coyote Point Equalizer over Software-based Solutions Abstract Coyote Point Equalizer appliances deliver traffic management solutions that provide high availability,

More information

Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data

Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data WHITE PAPER Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents Abstract... 3 What Is Big Data?...

More information

High Performance Cluster Support for NLB on Window

High Performance Cluster Support for NLB on Window High Performance Cluster Support for NLB on Window [1]Arvind Rathi, [2] Kirti, [3] Neelam [1]M.Tech Student, Department of CSE, GITM, Gurgaon Haryana (India) arvindrathi88@gmail.com [2]Asst. Professor,

More information

HDFS Users Guide. Table of contents

HDFS Users Guide. Table of contents Table of contents 1 Purpose...2 2 Overview...2 3 Prerequisites...3 4 Web Interface...3 5 Shell Commands... 3 5.1 DFSAdmin Command...4 6 Secondary NameNode...4 7 Checkpoint Node...5 8 Backup Node...6 9

More information

Comparative analysis of mapreduce job by keeping data constant and varying cluster size technique

Comparative analysis of mapreduce job by keeping data constant and varying cluster size technique Comparative analysis of mapreduce job by keeping data constant and varying cluster size technique Mahesh Maurya a, Sunita Mahajan b * a Research Scholar, JJT University, MPSTME, Mumbai, India,maheshkmaurya@yahoo.co.in

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

SiteCelerate white paper

SiteCelerate white paper SiteCelerate white paper Arahe Solutions SITECELERATE OVERVIEW As enterprises increases their investment in Web applications, Portal and websites and as usage of these applications increase, performance

More information

Accelerating Hadoop MapReduce Using an In-Memory Data Grid

Accelerating Hadoop MapReduce Using an In-Memory Data Grid Accelerating Hadoop MapReduce Using an In-Memory Data Grid By David L. Brinker and William L. Bain, ScaleOut Software, Inc. 2013 ScaleOut Software, Inc. 12/27/2012 H adoop has been widely embraced for

More information

Scala Storage Scale-Out Clustered Storage White Paper

Scala Storage Scale-Out Clustered Storage White Paper White Paper Scala Storage Scale-Out Clustered Storage White Paper Chapter 1 Introduction... 3 Capacity - Explosive Growth of Unstructured Data... 3 Performance - Cluster Computing... 3 Chapter 2 Current

More information

Multi-Channel Clustered Web Application Servers

Multi-Channel Clustered Web Application Servers THE AMERICAN UNIVERSITY IN CAIRO SCHOOL OF SCIENCES AND ENGINEERING Multi-Channel Clustered Web Application Servers A Masters Thesis Department of Computer Science and Engineering Status Report Seminar

More information

Energy- and Cost-Efficiency Analysis of ARM-Based Clusters

Energy- and Cost-Efficiency Analysis of ARM-Based Clusters Energy- and Cost-Efficiency Analysis of ARM-Based Clusters Zhonghong Ou, Bo Pang, Yang Deng, Jukka K. Nurminen, Antti Ylä-Jääski Department of Computer Science and Engineering Aalto University Helsinki,

More information

Jeffrey D. Ullman slides. MapReduce for data intensive computing

Jeffrey D. Ullman slides. MapReduce for data intensive computing Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very

More information

NoSQL and Hadoop Technologies On Oracle Cloud

NoSQL and Hadoop Technologies On Oracle Cloud NoSQL and Hadoop Technologies On Oracle Cloud Vatika Sharma 1, Meenu Dave 2 1 M.Tech. Scholar, Department of CSE, Jagan Nath University, Jaipur, India 2 Assistant Professor, Department of CSE, Jagan Nath

More information

Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com

Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop, Why? Need to process huge datasets on large clusters of computers

More information

GraySort and MinuteSort at Yahoo on Hadoop 0.23

GraySort and MinuteSort at Yahoo on Hadoop 0.23 GraySort and at Yahoo on Hadoop.23 Thomas Graves Yahoo! May, 213 The Apache Hadoop[1] software library is an open source framework that allows for the distributed processing of large data sets across clusters

More information

SERVER CLUSTERING TECHNOLOGY & CONCEPT

SERVER CLUSTERING TECHNOLOGY & CONCEPT SERVER CLUSTERING TECHNOLOGY & CONCEPT M00383937, Computer Network, Middlesex University, E mail: vaibhav.mathur2007@gmail.com Abstract Server Cluster is one of the clustering technologies; it is use for

More information

Can High-Performance Interconnects Benefit Memcached and Hadoop?

Can High-Performance Interconnects Benefit Memcached and Hadoop? Can High-Performance Interconnects Benefit Memcached and Hadoop? D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University,

More information

A Middleware Strategy to Survive Compute Peak Loads in Cloud

A Middleware Strategy to Survive Compute Peak Loads in Cloud A Middleware Strategy to Survive Compute Peak Loads in Cloud Sasko Ristov Ss. Cyril and Methodius University Faculty of Information Sciences and Computer Engineering Skopje, Macedonia Email: sashko.ristov@finki.ukim.mk

More information

HDFS Architecture Guide

HDFS Architecture Guide by Dhruba Borthakur Table of contents 1 Introduction... 3 2 Assumptions and Goals... 3 2.1 Hardware Failure... 3 2.2 Streaming Data Access...3 2.3 Large Data Sets... 3 2.4 Simple Coherency Model...3 2.5

More information

Application Performance Analysis of the Cortex-A9 MPCore

Application Performance Analysis of the Cortex-A9 MPCore This project in ARM is in part funded by ICT-eMuCo, a European project supported under the Seventh Framework Programme (7FP) for research and technological development Application Performance Analysis

More information

Benchmarking Hadoop & HBase on Violin

Benchmarking Hadoop & HBase on Violin Technical White Paper Report Technical Report Benchmarking Hadoop & HBase on Violin Harnessing Big Data Analytics at the Speed of Memory Version 1.0 Abstract The purpose of benchmarking is to show advantages

More information

Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database

Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Built up on Cisco s big data common platform architecture (CPA), a

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the The Israeli Association of Grid Technologies July 15, 2009 Outline Architecture

More information

Understanding the Benefits of IBM SPSS Statistics Server

Understanding the Benefits of IBM SPSS Statistics Server IBM SPSS Statistics Server Understanding the Benefits of IBM SPSS Statistics Server Contents: 1 Introduction 2 Performance 101: Understanding the drivers of better performance 3 Why performance is faster

More information

Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela

Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance

More information

Big Data With Hadoop

Big Data With Hadoop With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials

More information

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Edward Bortnikov & Ronny Lempel Yahoo Labs, Haifa Indexing in Search Engines Information Retrieval s two main stages: Indexing process

More information

Integrating Big Data into the Computing Curricula

Integrating Big Data into the Computing Curricula Integrating Big Data into the Computing Curricula Yasin Silva, Suzanne Dietrich, Jason Reed, Lisa Tsosie Arizona State University http://www.public.asu.edu/~ynsilva/ibigdata/ 1 Overview Motivation Big

More information

GeoGrid Project and Experiences with Hadoop

GeoGrid Project and Experiences with Hadoop GeoGrid Project and Experiences with Hadoop Gong Zhang and Ling Liu Distributed Data Intensive Systems Lab (DiSL) Center for Experimental Computer Systems Research (CERCS) Georgia Institute of Technology

More information

- An Essential Building Block for Stable and Reliable Compute Clusters

- An Essential Building Block for Stable and Reliable Compute Clusters Ferdinand Geier ParTec Cluster Competence Center GmbH, V. 1.4, March 2005 Cluster Middleware - An Essential Building Block for Stable and Reliable Compute Clusters Contents: Compute Clusters a Real Alternative

More information

Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage

Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage White Paper Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage A Benchmark Report August 211 Background Objectivity/DB uses a powerful distributed processing architecture to manage

More information

Virtuoso and Database Scalability

Virtuoso and Database Scalability Virtuoso and Database Scalability By Orri Erling Table of Contents Abstract Metrics Results Transaction Throughput Initializing 40 warehouses Serial Read Test Conditions Analysis Working Set Effect of

More information

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database An Oracle White Paper June 2012 High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database Executive Overview... 1 Introduction... 1 Oracle Loader for Hadoop... 2 Oracle Direct

More information

Maximizing Hadoop Performance and Storage Capacity with AltraHD TM

Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Executive Summary The explosion of internet data, driven in large part by the growth of more and more powerful mobile devices, has created

More information

Can the Elephants Handle the NoSQL Onslaught?

Can the Elephants Handle the NoSQL Onslaught? Can the Elephants Handle the NoSQL Onslaught? Avrilia Floratou, Nikhil Teletia David J. DeWitt, Jignesh M. Patel, Donghui Zhang University of Wisconsin-Madison Microsoft Jim Gray Systems Lab Presented

More information

Mobile Cloud Computing for Data-Intensive Applications

Mobile Cloud Computing for Data-Intensive Applications Mobile Cloud Computing for Data-Intensive Applications Senior Thesis Final Report Vincent Teo, vct@andrew.cmu.edu Advisor: Professor Priya Narasimhan, priya@cs.cmu.edu Abstract The computational and storage

More information

Detection of Distributed Denial of Service Attack with Hadoop on Live Network

Detection of Distributed Denial of Service Attack with Hadoop on Live Network Detection of Distributed Denial of Service Attack with Hadoop on Live Network Suchita Korad 1, Shubhada Kadam 2, Prajakta Deore 3, Madhuri Jadhav 4, Prof.Rahul Patil 5 Students, Dept. of Computer, PCCOE,

More information

Chapter 1 - Web Server Management and Cluster Topology

Chapter 1 - Web Server Management and Cluster Topology Objectives At the end of this chapter, participants will be able to understand: Web server management options provided by Network Deployment Clustered Application Servers Cluster creation and management

More information

Scaling Hadoop for Multi-Core and Highly Threaded Systems

Scaling Hadoop for Multi-Core and Highly Threaded Systems Scaling Hadoop for Multi-Core and Highly Threaded Systems Jangwoo Kim, Zoran Radovic Performance Architects Architecture Technology Group Sun Microsystems Inc. Project Overview Hadoop Updates CMT Hadoop

More information

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION A DIABLO WHITE PAPER AUGUST 2014 Ricky Trigalo Director of Business Development Virtualization, Diablo Technologies

More information

Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware

Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after

More information

Weekly Report. Hadoop Introduction. submitted By Anurag Sharma. Department of Computer Science and Engineering. Indian Institute of Technology Bombay

Weekly Report. Hadoop Introduction. submitted By Anurag Sharma. Department of Computer Science and Engineering. Indian Institute of Technology Bombay Weekly Report Hadoop Introduction submitted By Anurag Sharma Department of Computer Science and Engineering Indian Institute of Technology Bombay Chapter 1 What is Hadoop? Apache Hadoop (High-availability

More information

A programming model in Cloud: MapReduce

A programming model in Cloud: MapReduce A programming model in Cloud: MapReduce Programming model and implementation developed by Google for processing large data sets Users specify a map function to generate a set of intermediate key/value

More information

A Performance Analysis of Distributed Indexing using Terrier

A Performance Analysis of Distributed Indexing using Terrier A Performance Analysis of Distributed Indexing using Terrier Amaury Couste Jakub Kozłowski William Martin Indexing Indexing Used by search

More information

marlabs driving digital agility WHITEPAPER Big Data and Hadoop

marlabs driving digital agility WHITEPAPER Big Data and Hadoop marlabs driving digital agility WHITEPAPER Big Data and Hadoop Abstract This paper explains the significance of Hadoop, an emerging yet rapidly growing technology. The prime goal of this paper is to unveil

More information

DATA MINING WITH HADOOP AND HIVE Introduction to Architecture

DATA MINING WITH HADOOP AND HIVE Introduction to Architecture DATA MINING WITH HADOOP AND HIVE Introduction to Architecture Dr. Wlodek Zadrozny (Most slides come from Prof. Akella s class in 2014) 2015-2025. Reproduction or usage prohibited without permission of

More information

Fault Tolerance in Hadoop for Work Migration

Fault Tolerance in Hadoop for Work Migration 1 Fault Tolerance in Hadoop for Work Migration Shivaraman Janakiraman Indiana University Bloomington ABSTRACT Hadoop is a framework that runs applications on large clusters which are built on numerous

More information

Hadoop at Yahoo! Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com

Hadoop at Yahoo! Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Hadoop at Yahoo! Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since Feb

More information

Energy-Saving Cloud Computing Platform Based On Micro-Embedded System

Energy-Saving Cloud Computing Platform Based On Micro-Embedded System Energy-Saving Cloud Computing Platform Based On Micro-Embedded System Wen-Hsu HSIEH *, San-Peng KAO **, Kuang-Hung TAN **, Jiann-Liang CHEN ** * Department of Computer and Communication, De Lin Institute

More information

Performance Analysis of IPv4 v/s IPv6 in Virtual Environment Using UBUNTU

Performance Analysis of IPv4 v/s IPv6 in Virtual Environment Using UBUNTU Performance Analysis of IPv4 v/s IPv6 in Virtual Environment Using UBUNTU Savita Shiwani Computer Science,Gyan Vihar University, Rajasthan, India G.N. Purohit AIM & ACT, Banasthali University, Banasthali,

More information

Analysing Large Web Log Files in a Hadoop Distributed Cluster Environment

Analysing Large Web Log Files in a Hadoop Distributed Cluster Environment Analysing Large Files in a Hadoop Distributed Cluster Environment S Saravanan, B Uma Maheswari Department of Computer Science and Engineering, Amrita School of Engineering, Amrita Vishwa Vidyapeetham,

More information

Introduction 1 Performance on Hosted Server 1. Benchmarks 2. System Requirements 7 Load Balancing 7

Introduction 1 Performance on Hosted Server 1. Benchmarks 2. System Requirements 7 Load Balancing 7 Introduction 1 Performance on Hosted Server 1 Figure 1: Real World Performance 1 Benchmarks 2 System configuration used for benchmarks 2 Figure 2a: New tickets per minute on E5440 processors 3 Figure 2b:

More information

Scalable Cloud Computing Solutions for Next Generation Sequencing Data

Scalable Cloud Computing Solutions for Next Generation Sequencing Data Scalable Cloud Computing Solutions for Next Generation Sequencing Data Matti Niemenmaa 1, Aleksi Kallio 2, André Schumacher 1, Petri Klemelä 2, Eija Korpelainen 2, and Keijo Heljanko 1 1 Department of

More information

NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB, Cassandra, and MongoDB

NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB, Cassandra, and MongoDB bankmark UG (haftungsbeschränkt) Bahnhofstraße 1 9432 Passau Germany www.bankmark.de info@bankmark.de T +49 851 25 49 49 F +49 851 25 49 499 NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB,

More information

Muse Server Sizing. 18 June 2012. Document Version 0.0.1.9 Muse 2.7.0.0

Muse Server Sizing. 18 June 2012. Document Version 0.0.1.9 Muse 2.7.0.0 Muse Server Sizing 18 June 2012 Document Version 0.0.1.9 Muse 2.7.0.0 Notice No part of this publication may be reproduced stored in a retrieval system, or transmitted, in any form or by any means, without

More information

www.basho.com Technical Overview Simple, Scalable, Object Storage Software

www.basho.com Technical Overview Simple, Scalable, Object Storage Software www.basho.com Technical Overview Simple, Scalable, Object Storage Software Table of Contents Table of Contents... 1 Introduction & Overview... 1 Architecture... 2 How it Works... 2 APIs and Interfaces...

More information

Introducing the Singlechip Cloud Computer

Introducing the Singlechip Cloud Computer Introducing the Singlechip Cloud Computer Exploring the Future of Many-core Processors White Paper Intel Labs Jim Held Intel Fellow, Intel Labs Director, Tera-scale Computing Research Sean Koehl Technology

More information

Kentico CMS 6.0 Performance Test Report. Kentico CMS 6.0. Performance Test Report February 2012 ANOTHER SUBTITLE

Kentico CMS 6.0 Performance Test Report. Kentico CMS 6.0. Performance Test Report February 2012 ANOTHER SUBTITLE Kentico CMS 6. Performance Test Report Kentico CMS 6. Performance Test Report February 212 ANOTHER SUBTITLE 1 Kentico CMS 6. Performance Test Report Table of Contents Disclaimer... 3 Executive Summary...

More information

PARALLELS CLOUD STORAGE

PARALLELS CLOUD STORAGE PARALLELS CLOUD STORAGE Performance Benchmark Results 1 Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements...

More information

Big Fast Data Hadoop acceleration with Flash. June 2013

Big Fast Data Hadoop acceleration with Flash. June 2013 Big Fast Data Hadoop acceleration with Flash June 2013 Agenda The Big Data Problem What is Hadoop Hadoop and Flash The Nytro Solution Test Results The Big Data Problem Big Data Output Facebook Traditional

More information

Apache Hadoop FileSystem and its Usage in Facebook

Apache Hadoop FileSystem and its Usage in Facebook Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs

More information

Mobile Storage and Search Engine of Information Oriented to Food Cloud

Mobile Storage and Search Engine of Information Oriented to Food Cloud Advance Journal of Food Science and Technology 5(10): 1331-1336, 2013 ISSN: 2042-4868; e-issn: 2042-4876 Maxwell Scientific Organization, 2013 Submitted: May 29, 2013 Accepted: July 04, 2013 Published:

More information

Windows Server 2008 R2 Hyper-V Live Migration

Windows Server 2008 R2 Hyper-V Live Migration Windows Server 2008 R2 Hyper-V Live Migration Table of Contents Overview of Windows Server 2008 R2 Hyper-V Features... 3 Dynamic VM storage... 3 Enhanced Processor Support... 3 Enhanced Networking Support...

More information

Web Application Deployment in the Cloud Using Amazon Web Services From Infancy to Maturity

Web Application Deployment in the Cloud Using Amazon Web Services From Infancy to Maturity P3 InfoTech Solutions Pvt. Ltd http://www.p3infotech.in July 2013 Created by P3 InfoTech Solutions Pvt. Ltd., http://p3infotech.in 1 Web Application Deployment in the Cloud Using Amazon Web Services From

More information

Storage Technologies for Video Surveillance

Storage Technologies for Video Surveillance The surveillance industry continues to transition from analog to digital. This transition is taking place on two fronts how the images are captured and how they are stored. The way surveillance images

More information

Chapter-1 : Introduction 1 CHAPTER - 1. Introduction

Chapter-1 : Introduction 1 CHAPTER - 1. Introduction Chapter-1 : Introduction 1 CHAPTER - 1 Introduction This thesis presents design of a new Model of the Meta-Search Engine for getting optimized search results. The focus is on new dimension of internet

More information

POWER ALL GLOBAL FILE SYSTEM (PGFS)

POWER ALL GLOBAL FILE SYSTEM (PGFS) POWER ALL GLOBAL FILE SYSTEM (PGFS) Defining next generation of global storage grid Power All Networks Ltd. Technical Whitepaper April 2008, version 1.01 Table of Content 1. Introduction.. 3 2. Paradigm

More information

Map- reduce, Hadoop and The communica3on bo5leneck. Yoav Freund UCSD / Computer Science and Engineering

Map- reduce, Hadoop and The communica3on bo5leneck. Yoav Freund UCSD / Computer Science and Engineering Map- reduce, Hadoop and The communica3on bo5leneck Yoav Freund UCSD / Computer Science and Engineering Plan of the talk Why is Hadoop so popular? HDFS Map Reduce Word Count example using Hadoop streaming

More information

Hadoop Architecture. Part 1

Hadoop Architecture. Part 1 Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,

More information

Data processing goes big

Data processing goes big Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,

More information

Availability Digest. www.availabilitydigest.com. Redundant Load Balancing for High Availability July 2013

Availability Digest. www.availabilitydigest.com. Redundant Load Balancing for High Availability July 2013 the Availability Digest Redundant Load Balancing for High Availability July 2013 A large data center can comprise hundreds or thousands of servers. These servers must not only be interconnected, but they

More information

DELL. Virtual Desktop Infrastructure Study END-TO-END COMPUTING. Dell Enterprise Solutions Engineering

DELL. Virtual Desktop Infrastructure Study END-TO-END COMPUTING. Dell Enterprise Solutions Engineering DELL Virtual Desktop Infrastructure Study END-TO-END COMPUTING Dell Enterprise Solutions Engineering 1 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL

More information

ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy

ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy OVERVIEW The global communication and the continuous growth of services provided through the Internet or local infrastructure require to

More information

How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time

How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time SCALEOUT SOFTWARE How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time by Dr. William Bain and Dr. Mikhail Sobolev, ScaleOut Software, Inc. 2012 ScaleOut Software, Inc. 12/27/2012 T wenty-first

More information

Web Development. Owen Sacco. ICS2205/ICS2230 Web Intelligence

Web Development. Owen Sacco. ICS2205/ICS2230 Web Intelligence Web Development Owen Sacco ICS2205/ICS2230 Web Intelligence 2. Web Servers Introduction Web content lives on Web servers Web servers speak the platform independent HyperText Transfer Protocol (HTTP) (so

More information

VMware Virtual SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014

VMware Virtual SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014 VMware SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014 VMware SAN Backup Using VMware vsphere Table of Contents Introduction.... 3 vsphere Architectural Overview... 4 SAN Backup

More information

Reduction of Data at Namenode in HDFS using harballing Technique

Reduction of Data at Namenode in HDFS using harballing Technique Reduction of Data at Namenode in HDFS using harballing Technique Vaibhav Gopal Korat, Kumar Swamy Pamu vgkorat@gmail.com swamy.uncis@gmail.com Abstract HDFS stands for the Hadoop Distributed File System.

More information

PackeTV Mobile. http://www.vsicam.com. http://www.linkedin.com/company/visionary- solutions- inc. http://www.facebook.com/vsiptv

PackeTV Mobile. http://www.vsicam.com. http://www.linkedin.com/company/visionary- solutions- inc. http://www.facebook.com/vsiptv PackeTV Mobile Delivering HLS Video to Mobile Devices White Paper Created by Visionary Solutions, Inc. July, 2013 http://www.vsicam.com http://www.linkedin.com/company/visionary- solutions- inc. http://www.facebook.com/vsiptv

More information