BPP: Large Graph Storage for Efficient Disk-Based Processing
|
|
- Neal Henderson
- 7 years ago
- Views:
Transcription
1 , pp BPP: Large Graph Storage for Efficient Disk-Based Processing Kamran Najeebullah, Kifayat Ullah Khan, Muhammad Waqas Nawaz, Young-Koo Lee Department of Computer Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Korea {kamran, Kifayat, Abstract. Processing very large graphs like social networks, biological and chemical compounds is a challenging task. Distributed graph processing systems process the billion-scale graphs efficiently but incur overheads of efficient partitioning and distribution of the graph over a cluster of nodes. Distributed processing also requires cluster management and fault tolerance. In order to overcome these problems GraphChi was proposed recently. GraphChi significantly outperformed all the representative distributed processing frameworks. Still, we observe that GraphChi incurs some serious degradation in performance due to 1) high number of non-sequential I/Os for processing every chunk of graph; and 2) lack of true parallelism to process the graph. In this paper we propose a simple yet powerful engine BiShard Parallel Processor (BPP) to efficiently process billions-scale graphs on a single PC. We extend the storage structure proposed by GraphChi and introduce a new processing model called BiShard Parallel (BP). BP enables full CPU parallelism for processing the graph and significantly reduces the number of non-sequential I/Os required to process every chunk of the graph. Our experiments on real large graphs show that our solution significantly outperforms GraphChi. Keywords: Graph processing; Big data; Parallel processing; BiShard Parallel. 1 Introduction Graph processing has been a popular research area in the last decade and a lot of research has been targeted at the most common graph processing algorithms such as shortest path and some variations of clustering and page rank. Algorithms like connected components and minimum cut also have their own vital value. Graphs like social networks, biological and chemical compounds are difficult to process because of their massive size. With the growing size of the graph datasets, processing graphs has become more challenging. GraphChi [1] processes very large graphs on a single PC using an asynchronous computation model. It introduced a naive processing model, Parallel Sliding Window (PSW). PSW performs computation in execution intervals. An execution interval consist of three steps 1) load a subgraph of the input graph into memory; 2) process all the vertices of the subgraph in parallel and modify the values associated with the ISSN: ASTL Copyright 2013 SERSC
2 vertices and their incident edges; and 3) write the updates back to the disk. PSW passes information between the vertices using out-edges. GraphChi significantly outperformed the distributed processing systems on per node basis. We observe PSW inherits two serious bottlenecks. First, in every execution interval, PSW needs to perform non-sequential reads proportional to the number of the subgraphs. For a very large graph, in every execution interval the number of nonsequential reads will be significantly high. Second, for edges having both endpoints inside the same interval, PSW needs to avoid the race conditions between the two endpoints of the edge. As GraphChi maintains only one copy of the edges, both of the endpoints may end up accessing the edge at the same time. To avoid race conditions, PSW marks such edges as critical and processes the endpoint vertices sequentially. If graph is dense and all the edges have both endpoints inside the memory, all the vertices will be processed sequentially without any parallelism. We propose a disk-based graph processing engine following asynchronous model of computation which efficiently processes billion-scale graphs on a single PC. We introduce a new processing model BiShard Parallel (BSP) based on a vertex-centric approach [2]. BSP divides the graph into several subgraphs. For every subgraph, it manages the in and out edges separately, which allows to load a subgraph with only two reads. This storage structure manages two copies of every edge (one is each direction). This setting allows every vertex to have its own copy of the edges, and ensures full parallel processing. Our contributions are two folds: 1) a new storage structure that reduces the number of I/Os significantly; and 2) a new processing model that ensures full CPU parallelism. The rest of this paper is organized as follows. Section 2 reviews related works. Section 3 lists the core idea of the paper in details. Section 4 discusses experimental settings, results and comparison with state-of-the-art. Finally Section 5 summarizes and concludes the paper. 2 Related Work GraphChi extended the work of Bender [3] and Chen et. al. [4] and proposed a mechanism that stores and processes billion-scale graphs on a single consumer PC. Their system implements a naive technique for disk-based graph processing called Parallel Sliding Window (PSW). PSW exploits the sequential I/Os and parallel computation using vertex-centric approach for processing the graph. Their results show that GraphChi out-performs all the representative disk-based distributed systems on per node basis. However, experimental results also show some bottlenecks in part of parallel graph processing and number of disk reads in every execution interval. Recently, another disk-based graph processing framework TurboGraph [5] was proposed. TurboGraph is also designed to process very large graphs on modern consumer level machine with a flash drive. It implements a naive technique called Pin and Slide. TurboGraph fully exploits parallelism of flash disk and multi-core CPU. Their results show that their system outperforms GraphChi by an order-of-magnitude. However their solution exploits some specific properties of the flash drive which are not available on the rotational disks. 118 Copyright 2013 SERSC
3 3 BiShard Parallel Processor This section describes our proposed solution BiShard Parallel Processor (BPP). BPP is an asynchronous disk-based framework for processing very large graph on a single PC. We introduce a new processing model BiShard Parallel (BP). BP extends the storage structure proposed by GraphChi to reduce the non-sequential I/Os and eliminate the race conditions while accessing the edges shared by the vertices in memory. Vertices are allowed to modify their associated values and the values associated with their out-edges. BP processes the graph one chunk at a time. Processing of every interval consists of 3 steps 1) load a chunk of graph inside the memory; 2) perform computation on the chunk and modify the associated values of the vertices and out-edges; and 3) write the updated values back to the disk. 3.1 Loading of Subgraph We divide the graph G into P number of intervals. Every interval p consists of a subset of vertices V. We associate two shards in-shard(p) and out-shard(p) with every interval p. in-shard(p) contains all the in-edges of the vertices in interval p, while outshard(p) contains all the out-edges. In both of the shards edges are stored in order of their source vertex. Size and number of P are chosen such that any interval p can be loaded inside the memory. In order to process the interval subgraph we need to read all its vertices and their edges from the disk. We load the in-edges of the subgraph from the in-shard(p) and the out-edges of the subgraph from the out-shard(p). We perform only 2 nonsequential disk reads while processing an interval, irrespective of the graph size and total number of shards P. 3.2 Parallel vertex updates Graph algorithms are executed on graph by defining an update function. After all the interval vertices along their in and out edges are loaded inside the memory we run the update function for all the vertices in parallel. As every vertex has its own copy of the edges, there is no race condition while accessing any of the edges. We utilize true CPU parallelism by eliminating the race conditions for accessing the edges. 3.3 Writing back to disk To keep the asynchronous processing model intact we must write back the updated values to the disk as updates need to be available to any subsequent processing steps. We distribute and write the updated out-edges to all of the in-shards. Updated edges occur in sequential chunks in every shard. We keep track of the offsets in all the shards where we need to write the updates. Copyright 2013 SERSC 119
4 4 Experimental Evaluation We now list our experimental settings, details of the experiments and comparison with state-of-the-art. 4.1 Experiments setup Experiments were performed on a compatible PC with 3.3GHz Intel Core i CPU, 4GB of installed main memory and a 500GB 7200 rpm disk drive. We ran Microsoft Windows 7 64-bit with default settings. We disabled the file caching to get meaningful comparisons for small and large input files. 4.2 Experimental results and comparisons We implemented page rank algorithm in the same way as implemented by GraphChi. We conducted our experiments on the wiki-vote [6] data set with 7 thousand plus vertices and more than 1 hundred thousand edges. We varied the number of shards to evaluate the effect of different number of inter-interval edges and number nonsequential I/O. Our results showed that BPP significantly outperform GraphChi with both small and large number of shards. We noticed that the performance margin is larger when number of inter-interval edges is large and performance margin gets smaller with increase in number of shards. Fig.1. Graph visualizing the comparison of execution time of GraphChi and BPP while running Page Rank algorithm. 5 Conclusion GraphChi is a single PC, disk-based, graph processing engine. It solves the problems observed in distributed processing system but suffers from serious performance issues. In this work we extend the storage structure proposed by GraphChi, and proposed a new processing method called BiShard Parallel (BP). We showed by theoretical analysis that our solution reduced the number of non-sequential seeks and 120 Copyright 2013 SERSC
5 I/Os incurred in GraphChi to almost half, which makes it a better choice for processing graph on SSD. We also eliminated the race conditions between the vertices to access a common edge, which hindered GraphChi from full parallel processing of the graph vertices. Acknowledgements. This research was supported by the MSIP (Ministry of Science, ICT & Future Planning), Korea, under the ITRC(Information Technology Research Center) support program (NIPA H ) supervised by the NIPA(National IT Industry Promotion Agency). References 1. Kyrola, A., Blelloch, G., Guestrin, C.: Graphchi: large-scale graph computation on just a pc. In: OSDI, pp (2012). 2. Malewicz, G., Austern, M. H., Bik, A. J., Dehnert, J., Horn, I., Leiser, N., Czajkowski, G.: Pregel: a system for large-scale graph processing, In: SIGMOD ACM (2010). 3. Bender, M., Brodal, G., Fagerberg, R., Jacob, R., Vicari, E.: Optimal sparse matrix dense vector multiplication in the i/o-model. Theory of Computing Systems 47(4), pp (2010). 4. Chen, Y., Gan, Q., Suel, T.: I/O-efficient techniques for computing PageRank. In: Proceedings of the eleventh international conference on Information and knowledge management, pp , McLean, USA (2002) 5. Han, W.-S., Sangyeon, L., Park, K., Lee, J.-H., Kim, M.-S., Kim, J., Yu, H. Turbograph: A fast parallel graph engine handling billion-scale graphs in a single pc, In: Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data mining ACM (2013) 6. Leskovec, J., Huttenlocher, D., Kleinberg, J.: Predicting Positive and Negative Links in Online Social Networks, In: WWW (2010) Copyright 2013 SERSC 121
Software tools for Complex Networks Analysis. Fabrice Huet, University of Nice Sophia- Antipolis SCALE (ex-oasis) Team
Software tools for Complex Networks Analysis Fabrice Huet, University of Nice Sophia- Antipolis SCALE (ex-oasis) Team MOTIVATION Why do we need tools? Source : nature.com Visualization Properties extraction
More informationFast Iterative Graph Computation with Resource Aware Graph Parallel Abstraction
Human connectome. Gerhard et al., Frontiers in Neuroinformatics 5(3), 2011 2 NA = 6.022 1023 mol 1 Paul Burkhardt, Chris Waring An NSA Big Graph experiment Fast Iterative Graph Computation with Resource
More informationBSPCloud: A Hybrid Programming Library for Cloud Computing *
BSPCloud: A Hybrid Programming Library for Cloud Computing * Xiaodong Liu, Weiqin Tong and Yan Hou Department of Computer Engineering and Science Shanghai University, Shanghai, China liuxiaodongxht@qq.com,
More informationBig Graph Processing: Some Background
Big Graph Processing: Some Background Bo Wu Colorado School of Mines Part of slides from: Paul Burkhardt (National Security Agency) and Carlos Guestrin (Washington University) Mines CSCI-580, Bo Wu Graphs
More informationEvaluating partitioning of big graphs
Evaluating partitioning of big graphs Fredrik Hallberg, Joakim Candefors, Micke Soderqvist fhallb@kth.se, candef@kth.se, mickeso@kth.se Royal Institute of Technology, Stockholm, Sweden Abstract. Distributed
More informationLARGE-SCALE GRAPH PROCESSING IN THE BIG DATA WORLD. Dr. Buğra Gedik, Ph.D.
LARGE-SCALE GRAPH PROCESSING IN THE BIG DATA WORLD Dr. Buğra Gedik, Ph.D. MOTIVATION Graph data is everywhere Relationships between people, systems, and the nature Interactions between people, systems,
More informationGraph Processing and Social Networks
Graph Processing and Social Networks Presented by Shu Jiayu, Yang Ji Department of Computer Science and Engineering The Hong Kong University of Science and Technology 2015/4/20 1 Outline Background Graph
More informationMMap: Fast Billion-Scale Graph Computation on a PC via Memory Mapping
: Fast Billion-Scale Graph Computation on a PC via Memory Mapping Zhiyuan Lin, Minsuk Kahng, Kaeser Md. Sabrin, Duen Horng (Polo) Chau Georgia Tech Atlanta, Georgia {zlin48, kahng, kmsabrin, polo}@gatech.edu
More informationMap-Based Graph Analysis on MapReduce
2013 IEEE International Conference on Big Data Map-Based Graph Analysis on MapReduce Upa Gupta, Leonidas Fegaras University of Texas at Arlington, CSE Arlington, TX 76019 {upa.gupta,fegaras}@uta.edu Abstract
More informationOptimization and analysis of large scale data sorting algorithm based on Hadoop
Optimization and analysis of large scale sorting algorithm based on Hadoop Zhuo Wang, Longlong Tian, Dianjie Guo, Xiaoming Jiang Institute of Information Engineering, Chinese Academy of Sciences {wangzhuo,
More informationBENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next
More informationMizan: A System for Dynamic Load Balancing in Large-scale Graph Processing
/35 Mizan: A System for Dynamic Load Balancing in Large-scale Graph Processing Zuhair Khayyat 1 Karim Awara 1 Amani Alonazi 1 Hani Jamjoom 2 Dan Williams 2 Panos Kalnis 1 1 King Abdullah University of
More informationExploring RAID Configurations
Exploring RAID Configurations J. Ryan Fishel Florida State University August 6, 2008 Abstract To address the limits of today s slow mechanical disks, we explored a number of data layouts to improve RAID
More informationStorage Sizing Issue of VDI System
, pp.89-94 http://dx.doi.org/10.14257/astl.2014.49.19 Storage Sizing Issue of VDI System Jeongsook Park 1, Cheiyol Kim 1, Youngchang Kim 1, Youngcheol Kim 1, Sangmin Lee 1 and Youngkyun Kim 1 1 Electronics
More informationMassive Data Analysis Using Fast I/O
Abstract Massive Data Analysis Using Fast I/O Da Zheng Department of Computer Science The Johns Hopkins University Baltimore, Maryland, 21218 dzheng5@jhu.edu The advance of new I/O technologies has tremendously
More informationScalable Data Analysis in R. Lee E. Edlefsen Chief Scientist UserR! 2011
Scalable Data Analysis in R Lee E. Edlefsen Chief Scientist UserR! 2011 1 Introduction Our ability to collect and store data has rapidly been outpacing our ability to analyze it We need scalable data analysis
More informationEfficient and Customizable Data Partitioning Framework for Distributed Big RDF Data Processing in the Cloud
Efficient and Customizable Data ing Framework for Distributed Big RDF Data Processing in the Cloud Kisung Lee, Ling Liu, Yuzhe Tang, Qi Zhang, Yang Zhou DiSL, College of Computing, Georgia Institute of
More informationGridGraph: Large-Scale Graph Processing on a Single Machine Using 2-Level Hierarchical Partitioning
GridGraph: Large-Scale Graph Processing on a Single Machine Using -Level Hierarchical Partitioning Xiaowei Zhu, Wentao Han, and Wenguang Chen, Tsinghua University https://www.usenix.org/conference/atc5/technical-session/presentation/zhu
More informationMapReduce Algorithms. Sergei Vassilvitskii. Saturday, August 25, 12
MapReduce Algorithms A Sense of Scale At web scales... Mail: Billions of messages per day Search: Billions of searches per day Social: Billions of relationships 2 A Sense of Scale At web scales... Mail:
More informationEnergy Efficient MapReduce
Energy Efficient MapReduce Motivation: Energy consumption is an important aspect of datacenters efficiency, the total power consumption in the united states has doubled from 2000 to 2005, representing
More informationRecommended hardware system configurations for ANSYS users
Recommended hardware system configurations for ANSYS users The purpose of this document is to recommend system configurations that will deliver high performance for ANSYS users across the entire range
More informationLarge Scale Social Network Analysis
Large Scale Social Network Analysis DATA ANALYTICS 2013 TUTORIAL Rui Sarmento email@ruisarmento.com João Gama jgama@fep.up.pt Outline PART I 1. Introduction & Motivation Overview & Contributions 2. Software
More informationBenchmarking Hadoop & HBase on Violin
Technical White Paper Report Technical Report Benchmarking Hadoop & HBase on Violin Harnessing Big Data Analytics at the Speed of Memory Version 1.0 Abstract The purpose of benchmarking is to show advantages
More informationMining Large Datasets: Case of Mining Graph Data in the Cloud
Mining Large Datasets: Case of Mining Graph Data in the Cloud Sabeur Aridhi PhD in Computer Science with Laurent d Orazio, Mondher Maddouri and Engelbert Mephu Nguifo 16/05/2014 Sabeur Aridhi Mining Large
More informationDistributed Framework for Data Mining As a Service on Private Cloud
RESEARCH ARTICLE OPEN ACCESS Distributed Framework for Data Mining As a Service on Private Cloud Shraddha Masih *, Sanjay Tanwani** *Research Scholar & Associate Professor, School of Computer Science &
More informationEnhancing Dataset Processing in Hadoop YARN Performance for Big Data Applications
Enhancing Dataset Processing in Hadoop YARN Performance for Big Data Applications Ahmed Abdulhakim Al-Absi, Dae-Ki Kang and Myong-Jong Kim Abstract In Hadoop MapReduce distributed file system, as the input
More informationGraph Mining on Big Data System. Presented by Hefu Chai, Rui Zhang, Jian Fang
Graph Mining on Big Data System Presented by Hefu Chai, Rui Zhang, Jian Fang Outline * Overview * Approaches & Environment * Results * Observations * Notes * Conclusion Overview * What we have done? *
More informationPart 2: Community Detection
Chapter 8: Graph Data Part 2: Community Detection Based on Leskovec, Rajaraman, Ullman 2014: Mining of Massive Datasets Big Data Management and Analytics Outline Community Detection - Social networks -
More informationDistance Degree Sequences for Network Analysis
Universität Konstanz Computer & Information Science Algorithmics Group 15 Mar 2005 based on Palmer, Gibbons, and Faloutsos: ANF A Fast and Scalable Tool for Data Mining in Massive Graphs, SIGKDD 02. Motivation
More informationGraphChi: Large-Scale Graph Computation on Just a PC
GraphChi: Large-Scale Graph Computation on Just a PC Aapo Kyrola Carnegie Mellon University akyrola@cs.cmu.edu Guy Blelloch Carnegie Mellon University guyb@cs.cmu.edu Carlos Guestrin University of Washington
More informationStorage and Retrieval of Large RDF Graph Using Hadoop and MapReduce
Storage and Retrieval of Large RDF Graph Using Hadoop and MapReduce Mohammad Farhan Husain, Pankil Doshi, Latifur Khan, and Bhavani Thuraisingham University of Texas at Dallas, Dallas TX 75080, USA Abstract.
More informationA Case for Flash Memory SSD in Hadoop Applications
A Case for Flash Memory SSD in Hadoop Applications Seok-Hoon Kang, Dong-Hyun Koo, Woon-Hak Kang and Sang-Won Lee Dept of Computer Engineering, Sungkyunkwan University, Korea x860221@gmail.com, smwindy@naver.com,
More informationGPU File System Encryption Kartik Kulkarni and Eugene Linkov
GPU File System Encryption Kartik Kulkarni and Eugene Linkov 5/10/2012 SUMMARY. We implemented a file system that encrypts and decrypts files. The implementation uses the AES algorithm computed through
More informationBig Data Systems CS 5965/6965 FALL 2015
Big Data Systems CS 5965/6965 FALL 2015 Today General course overview Expectations from this course Q&A Introduction to Big Data Assignment #1 General Course Information Course Web Page http://www.cs.utah.edu/~hari/teaching/fall2015.html
More informationHPC performance applications on Virtual Clusters
Panagiotis Kritikakos EPCC, School of Physics & Astronomy, University of Edinburgh, Scotland - UK pkritika@epcc.ed.ac.uk 4 th IC-SCCE, Athens 7 th July 2010 This work investigates the performance of (Java)
More informationRingo: Interactive Graph Analytics on Big-Memory Machines
Ringo: Interactive Graph Analytics on Big-Memory Machines Yonathan Perez, Rok Sosič, Arijit Banerjee, Rohan Puttagunta, Martin Raison, Pararth Shah, Jure Leskovec Stanford University {yperez, rok, arijitb,
More informationScale up Vs. Scale out in Cloud Storage and Graph Processing Systems
Scale up Vs. Scale out in Cloud Storage and Graph Processing Systems Wenting Wang, Le Xu, Indranil Gupta Department of Computer Science, University of Illinois, Urbana Champaign {wwang84, lexu1, indy}@illinois.edu
More informationBig Data and Scripting Systems beyond Hadoop
Big Data and Scripting Systems beyond Hadoop 1, 2, ZooKeeper distributed coordination service many problems are shared among distributed systems ZooKeeper provides an implementation that solves these avoid
More informationHadoop Based Link Prediction Performance Analysis
Hadoop Based Link Prediction Performance Analysis Yuxiao Dong, Casey Robinson, Jian Xu Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556, USA Email: ydong1@nd.edu,
More informationUPS battery remote monitoring system in cloud computing
, pp.11-15 http://dx.doi.org/10.14257/astl.2014.53.03 UPS battery remote monitoring system in cloud computing Shiwei Li, Haiying Wang, Qi Fan School of Automation, Harbin University of Science and Technology
More informationFault Tolerance in Hadoop for Work Migration
1 Fault Tolerance in Hadoop for Work Migration Shivaraman Janakiraman Indiana University Bloomington ABSTRACT Hadoop is a framework that runs applications on large clusters which are built on numerous
More informationThe Development of an Intellectual Tracking App System based on IoT and RTLS
, pp.9-13 http://dx.doi.org/10.14257/astl.2015.85.03 The Development of an Intellectual Tracking App System based on IoT and RTLS Hak-Jun Lee 1, Ju-Su Kim 1, Umarov Jamshid 1, Man-Kyo Han 2, Ryum-Duck
More informationApache Hama Design Document v0.6
Apache Hama Design Document v0.6 Introduction Hama Architecture BSPMaster GroomServer Zookeeper BSP Task Execution Job Submission Job and Task Scheduling Task Execution Lifecycle Synchronization Fault
More informationA REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM
A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information
More informationBig Data, Statistics, and the Internet
Big Data, Statistics, and the Internet Steven L. Scott April, 4 Steve Scott (Google) Big Data, Statistics, and the Internet April, 4 / 39 Summary Big data live on more than one machine. Computing takes
More informationResearch Article Hadoop-Based Distributed Sensor Node Management System
Distributed Networks, Article ID 61868, 7 pages http://dx.doi.org/1.1155/214/61868 Research Article Hadoop-Based Distributed Node Management System In-Yong Jung, Ki-Hyun Kim, Byong-John Han, and Chang-Sung
More informationMALLET-Privacy Preserving Influencer Mining in Social Media Networks via Hypergraph
MALLET-Privacy Preserving Influencer Mining in Social Media Networks via Hypergraph Janani K 1, Narmatha S 2 Assistant Professor, Department of Computer Science and Engineering, Sri Shakthi Institute of
More informationComparision of k-means and k-medoids Clustering Algorithms for Big Data Using MapReduce Techniques
Comparision of k-means and k-medoids Clustering Algorithms for Big Data Using MapReduce Techniques Subhashree K 1, Prakash P S 2 1 Student, Kongu Engineering College, Perundurai, Erode 2 Assistant Professor,
More informationPerformance Comparison of VMware and Xen Hypervisor on Guest OS
ISSN: 2393-8528 Contents lists available at www.ijicse.in International Journal of Innovative Computer Science & Engineering Volume 2 Issue 3; July-August-2015; Page No. 56-60 Performance Comparison of
More informationOn Benchmarking Popular File Systems
On Benchmarking Popular File Systems Matti Vanninen James Z. Wang Department of Computer Science Clemson University, Clemson, SC 2963 Emails: {mvannin, jzwang}@cs.clemson.edu Abstract In recent years,
More informationAnalysis of Web Archives. Vinay Goel Senior Data Engineer
Analysis of Web Archives Vinay Goel Senior Data Engineer Internet Archive Established in 1996 501(c)(3) non profit organization 20+ PB (compressed) of publicly accessible archival material Technology partner
More informationEfficient Load Balancing using VM Migration by QEMU-KVM
International Journal of Computer Science and Telecommunications [Volume 5, Issue 8, August 2014] 49 ISSN 2047-3338 Efficient Load Balancing using VM Migration by QEMU-KVM Sharang Telkikar 1, Shreyas Talele
More informationPerformance Characteristics of VMFS and RDM VMware ESX Server 3.0.1
Performance Study Performance Characteristics of and RDM VMware ESX Server 3.0.1 VMware ESX Server offers three choices for managing disk access in a virtual machine VMware Virtual Machine File System
More informationHadoop MapReduce using Cache for Big Data Processing
Hadoop MapReduce using Cache for Big Data Processing Janani.J Dept. of computer science and engineering Arasu Engineering College Kumbakonam Kalaivani.K Dept. of computer science and engineering Arasu
More information! E6893 Big Data Analytics Lecture 9:! Linked Big Data Graph Computing (I)
! E6893 Big Data Analytics Lecture 9:! Linked Big Data Graph Computing (I) Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science Mgr., Dept. of Network Science and
More informationBoosting Database Batch workloads using Flash Memory SSDs
Boosting Database Batch workloads using Flash Memory SSDs Won-Gill Oh and Sang-Won Lee School of Information and Communication Engineering SungKyunKwan University, 27334 2066, Seobu-Ro, Jangan-Gu, Suwon-Si,
More informationSIGMOD RWE Review Towards Proximity Pattern Mining in Large Graphs
SIGMOD RWE Review Towards Proximity Pattern Mining in Large Graphs Fabian Hueske, TU Berlin June 26, 21 1 Review This document is a review report on the paper Towards Proximity Pattern Mining in Large
More informationEvaluating HDFS I/O Performance on Virtualized Systems
Evaluating HDFS I/O Performance on Virtualized Systems Xin Tang xtang@cs.wisc.edu University of Wisconsin-Madison Department of Computer Sciences Abstract Hadoop as a Service (HaaS) has received increasing
More informationBig Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network
, pp.273-284 http://dx.doi.org/10.14257/ijdta.2015.8.5.24 Big Data Analytics of Multi-Relationship Online Social Network Based on Multi-Subnet Composited Complex Network Gengxin Sun 1, Sheng Bin 2 and
More informationEfficient Data Replication Scheme based on Hadoop Distributed File System
, pp. 177-186 http://dx.doi.org/10.14257/ijseia.2015.9.12.16 Efficient Data Replication Scheme based on Hadoop Distributed File System Jungha Lee 1, Jaehwa Chung 2 and Daewon Lee 3* 1 Division of Supercomputing,
More informationCrime Hotspots Analysis in South Korea: A User-Oriented Approach
, pp.81-85 http://dx.doi.org/10.14257/astl.2014.52.14 Crime Hotspots Analysis in South Korea: A User-Oriented Approach Aziz Nasridinov 1 and Young-Ho Park 2 * 1 School of Computer Engineering, Dongguk
More informationChing-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science IBM Chief Scientist, Graph Computing. October 29th, 2015
E6893 Big Data Analytics Lecture 8: Spark Streams and Graph Computing (I) Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science IBM Chief Scientist, Graph Computing
More informationBenchmark Hadoop and Mars: MapReduce on cluster versus on GPU
Benchmark Hadoop and Mars: MapReduce on cluster versus on GPU Heshan Li, Shaopeng Wang The Johns Hopkins University 3400 N. Charles Street Baltimore, Maryland 21218 {heshanli, shaopeng}@cs.jhu.edu 1 Overview
More informationMapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research
MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With
More informationDELL SOLID STATE DISK (SSD) DRIVES
DELL SOLID STATE DISK (SSD) DRIVES STORAGE SOLUTIONS FOR SELECT POWEREDGE SERVERS By Bryan Martin, Dell Product Marketing Manager for HDD & SSD delltechcenter.com TAB LE OF CONTENTS INTRODUCTION 3 DOWNFALLS
More informationThe Shortcut Guide to Balancing Storage Costs and Performance with Hybrid Storage
The Shortcut Guide to Balancing Storage Costs and Performance with Hybrid Storage sponsored by Dan Sullivan Chapter 1: Advantages of Hybrid Storage... 1 Overview of Flash Deployment in Hybrid Storage Systems...
More informationMachine Learning over Big Data
Machine Learning over Big Presented by Fuhao Zou fuhao@hust.edu.cn Jue 16, 2014 Huazhong University of Science and Technology Contents 1 2 3 4 Role of Machine learning Challenge of Big Analysis Distributed
More informationSeptember 25, 2007. Maya Gokhale Georgia Institute of Technology
NAND Flash Storage for High Performance Computing Craig Ulmer cdulmer@sandia.gov September 25, 2007 Craig Ulmer Maya Gokhale Greg Diamos Michael Rewak SNL/CA, LLNL Georgia Institute of Technology University
More informationThe Bayesian Network Methodology for Industrial Control System with Digital Technology
, pp.157-161 http://dx.doi.org/10.14257/astl.2013.42.37 The Bayesian Network Methodology for Industrial Control System with Digital Technology Jinsoo Shin 1, Hanseong Son 2, Soongohn Kim 2, and Gyunyoung
More informationDistributed R for Big Data
Distributed R for Big Data Indrajit Roy, HP Labs November 2013 Team: Shivara m Erik Kyungyon g Alvin Rob Vanish A Big Data story Once upon a time, a customer in distress had. 2+ billion rows of financial
More informationOracle Database Scalability in VMware ESX VMware ESX 3.5
Performance Study Oracle Database Scalability in VMware ESX VMware ESX 3.5 Database applications running on individual physical servers represent a large consolidation opportunity. However enterprises
More informationDistributed forests for MapReduce-based machine learning
Distributed forests for MapReduce-based machine learning Ryoji Wakayama, Ryuei Murata, Akisato Kimura, Takayoshi Yamashita, Yuji Yamauchi, Hironobu Fujiyoshi Chubu University, Japan. NTT Communication
More informationCOMP 598 Applied Machine Learning Lecture 21: Parallelization methods for large-scale machine learning! Big Data by the numbers
COMP 598 Applied Machine Learning Lecture 21: Parallelization methods for large-scale machine learning! Instructor: (jpineau@cs.mcgill.ca) TAs: Pierre-Luc Bacon (pbacon@cs.mcgill.ca) Ryan Lowe (ryan.lowe@mail.mcgill.ca)
More informationAgenda. Enterprise Application Performance Factors. Current form of Enterprise Applications. Factors to Application Performance.
Agenda Enterprise Performance Factors Overall Enterprise Performance Factors Best Practice for generic Enterprise Best Practice for 3-tiers Enterprise Hardware Load Balancer Basic Unix Tuning Performance
More informationTwo-Level Metadata Management for Data Deduplication System
Two-Level Metadata Management for Data Deduplication System Jin San Kong 1, Min Ja Kim 2, Wan Yeon Lee 3.,Young Woong Ko 1 1 Dept. of Computer Engineering, Hallym University Chuncheon, Korea { kongjs,
More informationHealthcare Big Data Exploration in Real-Time
Healthcare Big Data Exploration in Real-Time Muaz A Mian A Project Submitted in partial fulfillment of the requirements for degree of Masters of Science in Computer Science and Systems University of Washington
More informationA Study on UNIX Server Consolidation by Virtualization
, pp.43-47 http://dx.doi.org/10.14257/astl.2014.46.10 A Study on UNIX Server Consolidation by Virtualization Lee Yong Hui 1, Kim Hyun Gi 2 1 Shinsung University, Chief of Information Service Center, DaehackRo1
More informationLarge-Scale Data Processing
Large-Scale Data Processing Eiko Yoneki eiko.yoneki@cl.cam.ac.uk http://www.cl.cam.ac.uk/~ey204 Systems Research Group University of Cambridge Computer Laboratory 2010s: Big Data Why Big Data now? Increase
More informationEfficient Parallel Execution of Sequence Similarity Analysis Via Dynamic Load Balancing
Efficient Parallel Execution of Sequence Similarity Analysis Via Dynamic Load Balancing James D. Jackson Philip J. Hatcher Department of Computer Science Kingsbury Hall University of New Hampshire Durham,
More informationCloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com
Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...
More informationThe Pitfalls of Deploying Solid-State Drive RAIDs
The Pitfalls of Deploying Solid-State Drive RAIDs Nikolaus Jeremic 1, Gero Mühl 1, Anselm Busse 2 and Jan Richling 2 Architecture of Application Systems Group 1 Faculty of Computer Science and Electrical
More informationSolid State Drive Architecture
Solid State Drive Architecture A comparison and evaluation of data storage mediums Tyler Thierolf Justin Uriarte Outline Introduction Storage Device as Limiting Factor Terminology Internals Interface Architecture
More informationMulti-level Metadata Management Scheme for Cloud Storage System
, pp.231-240 http://dx.doi.org/10.14257/ijmue.2014.9.1.22 Multi-level Metadata Management Scheme for Cloud Storage System Jin San Kong 1, Min Ja Kim 2, Wan Yeon Lee 3, Chuck Yoo 2 and Young Woong Ko 1
More informationExplore Efficient Data Organization for Large Scale Graph Analytics and Storage
Explore Efficient Data Organization for Large Scale Graph Analytics and Storage Yinglong Xia 1,IlieGabrielTanase 1,LifengNai 2,WeiTan 1,YanbinLiu 1,JasonCrawford 1,andChing-YungLin 1 1 IBM Research, Yorktown
More informationMoving Virtual Storage to the Cloud
Moving Virtual Storage to the Cloud White Paper Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage www.parallels.com Table of Contents Overview... 3 Understanding the Storage
More informationRevoScaleR Speed and Scalability
EXECUTIVE WHITE PAPER RevoScaleR Speed and Scalability By Lee Edlefsen Ph.D., Chief Scientist, Revolution Analytics Abstract RevoScaleR, the Big Data predictive analytics library included with Revolution
More informationIn-Situ Bitmaps Generation and Efficient Data Analysis based on Bitmaps. Yu Su, Yi Wang, Gagan Agrawal The Ohio State University
In-Situ Bitmaps Generation and Efficient Data Analysis based on Bitmaps Yu Su, Yi Wang, Gagan Agrawal The Ohio State University Motivation HPC Trends Huge performance gap CPU: extremely fast for generating
More informationHYBRID PROBABILITY BASED ENSEMBLES FOR BANKRUPTCY PREDICTION
HYBRID PROBABILITY BASED ENSEMBLES FOR BANKRUPTCY PREDICTION Chihli Hung 1, Jing Hong Chen 2, Stefan Wermter 3, 1,2 Department of Management Information Systems, Chung Yuan Christian University, Taiwan
More informationA Content-Based Load Balancing Algorithm for Metadata Servers in Cluster File Systems*
A Content-Based Load Balancing Algorithm for Metadata Servers in Cluster File Systems* Junho Jang, Saeyoung Han, Sungyong Park, and Jihoon Yang Department of Computer Science and Interdisciplinary Program
More informationDistributed Dynamic Load Balancing for Iterative-Stencil Applications
Distributed Dynamic Load Balancing for Iterative-Stencil Applications G. Dethier 1, P. Marchot 2 and P.A. de Marneffe 1 1 EECS Department, University of Liege, Belgium 2 Chemical Engineering Department,
More informationGraph Database Proof of Concept Report
Objectivity, Inc. Graph Database Proof of Concept Report Managing The Internet of Things Table of Contents Executive Summary 3 Background 3 Proof of Concept 4 Dataset 4 Process 4 Query Catalog 4 Environment
More informationA Dynamic Resource Management with Energy Saving Mechanism for Supporting Cloud Computing
A Dynamic Resource Management with Energy Saving Mechanism for Supporting Cloud Computing Liang-Teh Lee, Kang-Yuan Liu, Hui-Yang Huang and Chia-Ying Tseng Department of Computer Science and Engineering,
More informationHardware/Software Guidelines
There are many things to consider when preparing for a TRAVERSE v11 installation. The number of users, application modules and transactional volume are only a few. Reliable performance of the system is
More informationA Study on Workload Imbalance Issues in Data Intensive Distributed Computing
A Study on Workload Imbalance Issues in Data Intensive Distributed Computing Sven Groot 1, Kazuo Goda 1, and Masaru Kitsuregawa 1 University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan Abstract.
More informationAffinity Aware VM Colocation Mechanism for Cloud
Affinity Aware VM Colocation Mechanism for Cloud Nilesh Pachorkar 1* and Rajesh Ingle 2 Received: 24-December-2014; Revised: 12-January-2015; Accepted: 12-January-2015 2014 ACCENTS Abstract The most of
More informationDWMiner : A tool for mining frequent item sets efficiently in data warehouses
DWMiner : A tool for mining frequent item sets efficiently in data warehouses Bruno Kinder Almentero, Alexandre Gonçalves Evsukoff and Marta Mattoso COPPE/Federal University of Rio de Janeiro, P.O.Box
More informationQuantcast Petabyte Storage at Half Price with QFS!
9-131 Quantcast Petabyte Storage at Half Price with QFS Presented by Silvius Rus, Director, Big Data Platforms September 2013 Quantcast File System (QFS) A high performance alternative to the Hadoop Distributed
More informationBig Fast Data Hadoop acceleration with Flash. June 2013
Big Fast Data Hadoop acceleration with Flash June 2013 Agenda The Big Data Problem What is Hadoop Hadoop and Flash The Nytro Solution Test Results The Big Data Problem Big Data Output Facebook Traditional
More informationHadoop Cluster Applications
Hadoop Overview Data analytics has become a key element of the business decision process over the last decade. Classic reporting on a dataset stored in a database was sufficient until recently, but yesterday
More informationMoving Virtual Storage to the Cloud. Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage
Moving Virtual Storage to the Cloud Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage Table of Contents Overview... 1 Understanding the Storage Problem... 1 What Makes
More information