Redundant Storage Cluster

Size: px
Start display at page:

Download "Redundant Storage Cluster"

Transcription

1 Redundant Storage Cluster For When It's Just Too Big Bob Burgess radian 6 Technologies MySQL User Conference 2009

2 Scope Fundamentals of MySQL Proxy Fundamentals of LuaSQL Description of the Redundant Storage Cluster Architecture Sample code

3 Scope NOT Complete Course on MySQL Proxy Lua Programming Course Amazing Lua Tricks Complete code listing of cluster (Posted online)

4 The Problem

5 Growth! 12 Blog Posts millions per day

6 ... Current Solution month table myisam merge table month table month table month table Data over NFS can be unreliable Separate copy for each DB that could use it (Master / Replicas)

7 Goals Single place for all content Redundancy without complete duplication Add storage by adding nodes Survive a server failure

8 Existing Products HiveDB Java/Hibernate-based No redundancy Spock Proxy No redundancy MySQL Cluster All indexes in RAM

9 Cluster Master DB content (federated) Other DBs Other DBs DBs content (federated) Load Balancer MySQL Proxy Lua MySQL Proxy Lua MySQL Proxy Lua content 1 1 dir content 2 content n dir dir 2... n

10 Component Walk-Through Federated Engine Load Balancer MySQL Proxy Lua LuaSQL

11 Federated Engine Just a pointer to another table create table sample ( id int primary key, value varchar(100) ) engine=federated connection=

12 Federated Engine create table sample ( id int primary key, val int, name varchar(50) ) select id,name from sample; SELECT `id`, `val`, `name` FROM `sample`; select id,name from sample where id=2; SELECT `id`, `val`, `name` FROM sample WHERE `id` = '2'; select val,name from sample where val=2; SELECT `id`, `val`, `name` FROM `sample`; select * from sample limit 10; SELECT `id`, `val`, `name` FROM `sample`;

13 Federated Engine select max(id)from sample; SELECT `id`, `val`, `name` FROM `sample`; select count(*) from sample; SELECT `id`, `val`, `name` FROM `sample`; insert into sample (id,name) values (5,'bob'); INSERT INTO `sample` (`id`, `val`, `name`) VALUES ('5', NULL, 'bob'); insert into sample values (5,10,'bob'); INSERT INTO `sample` (`id`, `val`, `name`) VALUES ('5', '10', 'bob');

14 Load Balancer Options MySQL Proxy Load Balancer For read balancing only For traditional master/slave architecture Custom load balancer MySQL Proxy & Lua script Still an option, depending on future architecture

15 Load Balancer Options Linux Networking NAT LB stays involved in session Limited Scalability Slower option Linux Networking Direct Routing LB hands off session Node answers client directly Much more scalable

16 MySQL Proxy Communicates with a MySQL client using MySQL Network Protocol Provides an API to a script environment Pass query from client Return result set to the client Uses Lua scripting language

17 MySQL Proxy Supplied Constants and Methods global proxy.global.connect_retry=3 proxy.global.conremote={} proxy.global.mysqlenv=luasql.mysql() proxy.global.conremote[tonumber(details.nodeid)] = proxy.global.mysqlenv:connect(...) queries list of queries going to the server proxy.queries:reset()

18 MySQL Proxy Supplied Constants and Methods response response from server to client proxy.response.type = proxy.mysqld_packet_ok proxy.response.resultset = {...} Constants MYSQL_PACKET_OK MYSQL_PACKET_ERR MYSQL_TYPE_LONG PROXY_SEND_RESULT

19 MySQL Proxy MySQL client 3306 MySQL server MySQL client 4040 MySQL Proxy 3306 MySQL server MySQL client 4040 MySQL Proxy Lua LuaSQL 3306 MySQL server

20 MySQL Proxy MySQL client MySQL server MySQL Proxy Lua read_query( ) read_query_result( )

21 MySQL Proxy MySQL client MySQL server MySQL Proxy Lua LuaSQL calls

22 LuaSQL Connect directly to databases from Lua scripts environment :connect connection :execute (select) :execute (ins/upd/del) cursor return code & error msg :fetch return set

23 LuaSQL Connection Timeouts Connection drops after sleeping 10s call the execute method if cursor object is nil: call connect method for the environment call execute again

24 LuaSQL Connection Timeouts function select_db (node,sql) local reconnects = 0 local cur local err repeat cur, err = proxy.global.conremote[tonumber(node)]:execute(sql) if cur == nil then print ('Reconnecting select_db. Error='..err) reopen_remote_db (node) end reconnects = reconnects+1 until (cur ~= nil or reconnects==proxy.global.connect_retry) if cur==nil then error("could not reconnect / No result set in select_db",2) else return cur:fetch() end end

25 LuaSQL Connection Timeouts function execute_db (node,sql) local reconnects = 0 local LOST_CONNECTION = "MySQL server has gone away" local rc local err repeat rc, err = proxy.global.conremote[tonumber(node)]:execute(sql) if rc ~= nil then print ("execute_db RC="..rc); end if err~=nil then print ("execute_db error="..err); end if rc == nil and err:find(lost_connection) then print ('Reconnecting execute_db. Error='..err) reopen_remote_db (node) end reconnects = reconnects+1 until (rc ~= nil or reconnects==proxy.global.connect_retry or not err:find(lost_connection) ) if rc == nil and err:find(lost_connection) then error("could not reconnect in execute_db.",2) else return rc,err end end

26 System Info Script Keeps cluster up to date on disk usage Update Node table on all nodes with disk size & free of this node

27 Accepting Queries from Client Query comes in to Proxy, read_query is called The query appears in read_query's parameter First byte of variable indicates the type of query: proxy.com_query (Query) proxy.com_process_info (Process List ) proxy.com_connect (Connect) proxy.com_process_kill (Kill)

28 Returning a result set Two response types "OK" Error Set properties of the "response" object in the "proxy" environment

29 Returning a result set: Error Array: type errmsg errcode sqlstate proxy.response={ type= proxy.mysqld_packet_err, errmsg= "Malformed INSERT statement.", errcode= 1064, sqlstate= "42000" } return proxy.proxy_send_result

30 Returning a result set: OK Empty Result Set Array: type proxy.response.type=proxy.mysqld_packet_ok return proxy.proxy_send_result

31 Returning a result set: OK Full result set Array: type (value) resultset (table) fields (table) rows (table) type proxy resultset (etc.) fields rows name value value value value value value

32 Returning a result set: OK fields rows type name value value value value value value proxy.response.type = proxy.mysqld_packet_ok proxy.response.resultset = { fields = { {type=proxy.mysql_type_longlong, name="blogpostid" }, {type=proxy.mysql_type_long, name="partitionkey"}, {type=proxy.mysql_type_var_string, name="rawcontent" } }, rows = { { tonumber(itemid), tonumber(partkey), contentvalue } } } return proxy.proxy_send_result

33 Cluster Operation Overview

34 Cluster: Librarian Talks to the client Accepts items to store Retrieves items / gives them to client The single Lua script that runs under Proxy

35 Cluster: Librarian Directory itemid nodeid partitionkey Event table serial no. event type event data

36 Cluster: Librarian Content_partitionKey itemid item compressed into largeblob Node table nodeid connection / authentication info system status (disk) capacity factor

37 Cluster: Librarian insert no yes syntax good bad error exists no store on this node yes Error: table doesn't exist error create table store on this node update ALL directories store Event return OK to client

38 Cluster: Librarian Returning an error proxy.response={ type= proxy.mysqld_packet_err, errmsg= "Duplicate entry '"..itemid.."' for key 1", errcode= 1062, sqlstate= "23000"} return proxy.proxy_send_result Content Tables MyISAM (concurrent_insert=2) One table per partition key For us: 300 MB / hour

39 Cluster: Librarian select no yes syntax good bad error 1=0 no yes return empty result set to client info table no yes calculate max(id) and count(*) return result set to client get item from remote db remote read dir local doesn't exist return empty result set to client get item from this db return result set to client

40 Cluster: Librarian Federated select max(id)from sample; SELECT `id`, `val`, `name` FROM `sample`; Client create table _content_info ( _max_id bigint unsigned, _count bigint unsigned) engine=federated connection=(...schema/_content_info...); Target create view _content_info as select max(id) _max_id, count(*) _count from _content;

41 Cluster: Librarian show table status like `_content`; Auto_increment 0 cluster node add cluster node status cluster node offline

42 Cluster: Rebalancer Enforces redundancy policy for new inserts to heal from node loss Rebalance node added equal disk usage

43 Cluster: Rebalancer Policy enforce for new items get earliest new item no new items find another node with the most free space copy the item there update all directories next step

44 Cluster: Rebalancer Policy enforce for other items get one item from the Directory which exists on an insufficient number of nodes and has "me" as the first listed node no items copy the item to the node with the most free space that doesn't already have it update all directories next step

45 Cluster: Rebalancer Rebalancer do I have the most free space of all the nodes yes no find the node with the least free space find an item on that node that's not on this node (that obeys the redundancy policy) move the item from that node to me update all directories done

46 Member Add Add new node to Node table Get a copy of the directory from all nodes (a piece from each)

47 Member Remove Set free-space margin to 0 Force "free disk space" for this node to 0 Wait for Rebalance to copy everything off Restore the free-space margin and update the Node list

48 Member Fail Update node list Remove all directory entries for this node

49 Information Age-Out Drop tables for obsolete partition keys Remove directory entries for those partitions

50 Table Optimizer information_schema.tables compare data_free to data_length "Lock" that partition on that node (row in PartitionLock table) Run optimize table but abandon if any node fails Unlock table

51 Backup "Write Lock" one partition across all nodes Copy that partition table for all nodes Copy directory entries for that node Unlock

52 Development Directions Bulletproof error handling Performance tuning

53 Alternative Architectures Load Balancer choices Move SQL parsing to a complex proxy-based load balancer, communicate with nodes on network sockets Librarian in Perl, Java, C Alternative databases

54 Resources MySQL Documentation

55 Thank you! Bob Burgess

Tushar Joshi Turtle Networks Ltd

Tushar Joshi Turtle Networks Ltd MySQL Database for High Availability Web Applications Tushar Joshi Turtle Networks Ltd www.turtle.net Overview What is High Availability? Web/Network Architecture Applications MySQL Replication MySQL Clustering

More information

SCALABLE DATA SERVICES

SCALABLE DATA SERVICES 1 SCALABLE DATA SERVICES 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2 Overview MySQL Database Clustering GlusterFS Memcached 3 Overview Problems of Data Services 4 Data retrieval

More information

Zero Downtime Deployments with Database Migrations. Bob Feldbauer twitter: @bobfeldbauer email: bob.feldbauer@timgroup.com

Zero Downtime Deployments with Database Migrations. Bob Feldbauer twitter: @bobfeldbauer email: bob.feldbauer@timgroup.com Zero Downtime Deployments with Database Migrations Bob Feldbauer twitter: @bobfeldbauer email: bob.feldbauer@timgroup.com Deployments Two parts to deployment: Application code Database schema changes (migrations,

More information

Database Administration with MySQL

Database Administration with MySQL Database Administration with MySQL Suitable For: Database administrators and system administrators who need to manage MySQL based services. Prerequisites: Practical knowledge of SQL Some knowledge of relational

More information

Tier Architectures. Kathleen Durant CS 3200

Tier Architectures. Kathleen Durant CS 3200 Tier Architectures Kathleen Durant CS 3200 1 Supporting Architectures for DBMS Over the years there have been many different hardware configurations to support database systems Some are outdated others

More information

Part 3. MySQL DBA I Exam

Part 3. MySQL DBA I Exam Part 3. MySQL DBA I Exam Table of Contents 23. MySQL Architecture... 3 24. Starting, Stopping, and Configuring MySQL... 6 25. Client Programs for DBA Work... 11 26. MySQL Administrator... 15 27. Character

More information

New Features in MySQL 5.0, 5.1, and Beyond

New Features in MySQL 5.0, 5.1, and Beyond New Features in MySQL 5.0, 5.1, and Beyond Jim Winstead jimw@mysql.com Southern California Linux Expo February 2006 MySQL AB 5.0: GA on 19 October 2005 Expanded SQL standard support: Stored procedures

More information

MySQL Storage Engines

MySQL Storage Engines MySQL Storage Engines Data in MySQL is stored in files (or memory) using a variety of different techniques. Each of these techniques employs different storage mechanisms, indexing facilities, locking levels

More information

ODBC Client Driver Help. 2015 Kepware, Inc.

ODBC Client Driver Help. 2015 Kepware, Inc. 2015 Kepware, Inc. 2 Table of Contents Table of Contents 2 4 Overview 4 External Dependencies 4 Driver Setup 5 Data Source Settings 5 Data Source Setup 6 Data Source Access Methods 13 Fixed Table 14 Table

More information

COSC 6397 Big Data Analytics. 2 nd homework assignment Pig and Hive. Edgar Gabriel Spring 2015

COSC 6397 Big Data Analytics. 2 nd homework assignment Pig and Hive. Edgar Gabriel Spring 2015 COSC 6397 Big Data Analytics 2 nd homework assignment Pig and Hive Edgar Gabriel Spring 2015 2 nd Homework Rules Each student should deliver Source code (.java files) Documentation (.pdf,.doc,.tex or.txt

More information

Partitioning under the hood in MySQL 5.5

Partitioning under the hood in MySQL 5.5 Partitioning under the hood in MySQL 5.5 Mattias Jonsson, Partitioning developer Mikael Ronström, Partitioning author Who are we? Mikael is a founder of the technology behind NDB

More information

In Memory Accelerator for MongoDB

In Memory Accelerator for MongoDB In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000

More information

HDFS Users Guide. Table of contents

HDFS Users Guide. Table of contents Table of contents 1 Purpose...2 2 Overview...2 3 Prerequisites...3 4 Web Interface...3 5 Shell Commands... 3 5.1 DFSAdmin Command...4 6 Secondary NameNode...4 7 Checkpoint Node...5 8 Backup Node...6 9

More information

Scaling up = getting a better machine. Scaling out = use another server and add it to your cluster.

Scaling up = getting a better machine. Scaling out = use another server and add it to your cluster. MongoDB 1. Introduction MongoDB is a document-oriented database, not a relation one. It replaces the concept of a row with a document. This makes it possible to represent complex hierarchical relationships

More information

Comparing SQL and NOSQL databases

Comparing SQL and NOSQL databases COSC 6397 Big Data Analytics Data Formats (II) HBase Edgar Gabriel Spring 2015 Comparing SQL and NOSQL databases Types Development History Data Storage Model SQL One type (SQL database) with minor variations

More information

IBM Redistribute Big SQL v4.x Storage Paths IBM. Redistribute Big SQL v4.x Storage Paths

IBM Redistribute Big SQL v4.x Storage Paths IBM. Redistribute Big SQL v4.x Storage Paths Redistribute Big SQL v4.x Storage Paths THE GOAL The Big SQL temporary tablespace is used during high volume queries to spill sorts or intermediate data to disk. To improve I/O performance for these queries,

More information

How to backup a remote MySQL server with ZRM over the Internet

How to backup a remote MySQL server with ZRM over the Internet How to backup a remote MySQL server with ZRM over the Internet White paper "As MySQL gains widespread adoption and moves more broadly into the enterprise, ZRM for MySQL addresses the growing need among

More information

MySQL Enterprise Monitor

MySQL Enterprise Monitor MySQL Enterprise Monitor Lynn Ferrante Principal Sales Consultant 1 Program Agenda MySQL Enterprise Monitor Overview Architecture Roles Demo 2 Overview 3 MySQL Enterprise Edition Highest Levels of Security,

More information

IceWarp to IceWarp Server Migration

IceWarp to IceWarp Server Migration IceWarp to IceWarp Server Migration Registered Trademarks iphone, ipad, Mac, OS X are trademarks of Apple Inc., registered in the U.S. and other countries. Microsoft, Windows, Outlook and Windows Phone

More information

Workflow Templates Library

Workflow Templates Library Workflow s Library Table of Contents Intro... 2 Active Directory... 3 Application... 5 Cisco... 7 Database... 8 Excel Automation... 9 Files and Folders... 10 FTP Tasks... 13 Incident Management... 14 Security

More information

High Availability Solutions for the MariaDB and MySQL Database

High Availability Solutions for the MariaDB and MySQL Database High Availability Solutions for the MariaDB and MySQL Database 1 Introduction This paper introduces recommendations and some of the solutions used to create an availability or high availability environment

More information

Open Source DBMS CUBRID 2008 & Community Activities. Byung Joo Chung bjchung@cubrid.com

Open Source DBMS CUBRID 2008 & Community Activities. Byung Joo Chung bjchung@cubrid.com Open Source DBMS CUBRID 2008 & Community Activities Byung Joo Chung bjchung@cubrid.com Agenda Open Source DBMS CUBRID 2008 CUBRID Community Activities Open Source DBMS CUBRID 2008 Open Source DBMS CUBRID

More information

MarkLogic Server. Database Replication Guide. MarkLogic 8 February, 2015. Copyright 2015 MarkLogic Corporation. All rights reserved.

MarkLogic Server. Database Replication Guide. MarkLogic 8 February, 2015. Copyright 2015 MarkLogic Corporation. All rights reserved. Database Replication Guide 1 MarkLogic 8 February, 2015 Last Revised: 8.0-1, February, 2015 Copyright 2015 MarkLogic Corporation. All rights reserved. Table of Contents Table of Contents Database Replication

More information

Real-time reporting at 10,000 inserts per second. Wesley Biggs CTO 25 October 2011 Percona Live

Real-time reporting at 10,000 inserts per second. Wesley Biggs CTO 25 October 2011 Percona Live Real-time reporting at 10,000 inserts per second Wesley Biggs CTO 25 October 2011 Percona Live Agenda 1. Who we are, what we do, and (maybe) why we do it 2. Solution architecture and evolution 3. Top 5

More information

Application note: SQL@CHIP Connecting the IPC@CHIP to a Database

Application note: SQL@CHIP Connecting the IPC@CHIP to a Database Application note: SQL@CHIP Connecting the IPC@CHIP to a Database 1. Introduction This application note describes how to connect an IPC@CHIP to a database and exchange data between those. As there are no

More information

Scalability of web applications. CSCI 470: Web Science Keith Vertanen

Scalability of web applications. CSCI 470: Web Science Keith Vertanen Scalability of web applications CSCI 470: Web Science Keith Vertanen Scalability questions Overview What's important in order to build scalable web sites? High availability vs. load balancing Approaches

More information

Tipping The Scale Tips, Tools, and Techniques For Building Scalable. Steve French Senior Software Engineer digg.com

Tipping The Scale Tips, Tools, and Techniques For Building Scalable. Steve French Senior Software Engineer digg.com Tipping The Scale Tips, Tools, and Techniques For Building Scalable Steve French Senior Software Engineer digg.com First Thing s First... The Stack Server OS Linux, MacOS X, UNIX, Windows Web Server apache,

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

A Brief Introduction to MySQL

A Brief Introduction to MySQL A Brief Introduction to MySQL by Derek Schuurman Introduction to Databases A database is a structured collection of logically related data. One common type of database is the relational database, a term

More information

Troubleshooting. Section 7. Introduction. EB Gensym G2 Interface Error Codes

Troubleshooting. Section 7. Introduction. EB Gensym G2 Interface Error Codes Troubleshooting Section 7 Introduction This section provides troubleshooting information for the Gensym G2 interface software. EB Gensym G2 Interface Error s Table 7-1. Sub Error s Table 7-1 lists the

More information

Microsoft SQL Server OLTP Best Practice

Microsoft SQL Server OLTP Best Practice Microsoft SQL Server OLTP Best Practice The document Introduction to Transactional (OLTP) Load Testing for all Databases provides a general overview on the HammerDB OLTP workload and the document Microsoft

More information

MYSQL DATABASE ACCESS WITH PHP

MYSQL DATABASE ACCESS WITH PHP MYSQL DATABASE ACCESS WITH PHP Fall 2009 CSCI 2910 Server Side Web Programming Typical web application interaction Database Server 3 tiered architecture Security in this interaction is critical Web Server

More information

D61830GC30. MySQL for Developers. Summary. Introduction. Prerequisites. At Course completion After completing this course, students will be able to:

D61830GC30. MySQL for Developers. Summary. Introduction. Prerequisites. At Course completion After completing this course, students will be able to: D61830GC30 for Developers Summary Duration Vendor Audience 5 Days Oracle Database Administrators, Developers, Web Administrators Level Technology Professional Oracle 5.6 Delivery Method Instructor-led

More information

USING FILERELICATIONPRO TO REPLICATE SQL SERVER

USING FILERELICATIONPRO TO REPLICATE SQL SERVER USING FILERELICATIONPRO TO REPLICATE SQL SERVER Abstract FileReplicationPro (FRP) can be used to backup your SQL Server databases. The replication procedure is not as straight forward as replicating other

More information

Using SQL Server Management Studio

Using SQL Server Management Studio Using SQL Server Management Studio Microsoft SQL Server Management Studio 2005 is a graphical tool for database designer or programmer. With SQL Server Management Studio 2005 you can: Create databases

More information

Apache Hadoop. Alexandru Costan

Apache Hadoop. Alexandru Costan 1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open

More information

Appendix A Core Concepts in SQL Server High Availability and Replication

Appendix A Core Concepts in SQL Server High Availability and Replication Appendix A Core Concepts in SQL Server High Availability and Replication Appendix Overview Core Concepts in High Availability Core Concepts in Replication 1 Lesson 1: Core Concepts in High Availability

More information

MySQL: Cloud vs Bare Metal, Performance and Reliability

MySQL: Cloud vs Bare Metal, Performance and Reliability MySQL: Cloud vs Bare Metal, Performance and Reliability Los Angeles MySQL Meetup Vladimir Fedorkov, March 31, 2014 Let s meet each other Performance geek All kinds MySQL and some Sphinx Working for Blackbird

More information

How, What, and Where of Data Warehouses for MySQL

How, What, and Where of Data Warehouses for MySQL How, What, and Where of Data Warehouses for MySQL Robert Hodges CEO, Continuent. Introducing Continuent The leading provider of clustering and replication for open source DBMS Our Product: Continuent Tungsten

More information

Integrating VoltDB with Hadoop

Integrating VoltDB with Hadoop The NewSQL database you ll never outgrow Integrating with Hadoop Hadoop is an open source framework for managing and manipulating massive volumes of data. is an database for handling high velocity data.

More information

"SQL Database Professional " module PRINTED MANUAL

SQL Database Professional  module PRINTED MANUAL "SQL Database Professional " module PRINTED MANUAL "SQL Database Professional " module All rights reserved. No parts of this work may be reproduced in any form or by any means - graphic, electronic, or

More information

Whitepaper: performance of SqlBulkCopy

Whitepaper: performance of SqlBulkCopy We SOLVE COMPLEX PROBLEMS of DATA MODELING and DEVELOP TOOLS and solutions to let business perform best through data analysis Whitepaper: performance of SqlBulkCopy This whitepaper provides an analysis

More information

Full Text Search in MySQL 5.1 New Features and HowTo

Full Text Search in MySQL 5.1 New Features and HowTo Full Text Search in MySQL 5.1 New Features and HowTo Alexander Rubin Senior Consultant, MySQL AB 1 Full Text search Natural and popular way to search for information Easy to use: enter key words and get

More information

In-Memory Databases MemSQL

In-Memory Databases MemSQL IT4BI - Université Libre de Bruxelles In-Memory Databases MemSQL Gabby Nikolova Thao Ha Contents I. In-memory Databases...4 1. Concept:...4 2. Indexing:...4 a. b. c. d. AVL Tree:...4 B-Tree and B+ Tree:...5

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction

More information

MySQL Technical Overview

MySQL Technical Overview MySQL Technical Overview Mayuresh Nirhali Member Technical Staff Solaris RPE Mayuresh.Nirhali@Sun.COM 1 Agenda MySQL AB > Sun Microsystems Technology Stacks MySQL Server Architecture Pluggable Storage

More information

Table of Contents Introduction and System Requirements 9 Installing VMware Server 35

Table of Contents Introduction and System Requirements 9 Installing VMware Server 35 Table of Contents Introduction and System Requirements 9 VMware Server: Product Overview 10 Features in VMware Server 11 Support for 64-bit Guest Operating Systems 11 Two-Way Virtual SMP (Experimental

More information

Using Object Database db4o as Storage Provider in Voldemort

Using Object Database db4o as Storage Provider in Voldemort Using Object Database db4o as Storage Provider in Voldemort by German Viscuso db4objects (a division of Versant Corporation) September 2010 Abstract: In this article I will show you how

More information

Overview of Databases On MacOS. Karl Kuehn Automation Engineer RethinkDB

Overview of Databases On MacOS. Karl Kuehn Automation Engineer RethinkDB Overview of Databases On MacOS Karl Kuehn Automation Engineer RethinkDB Session Goals Introduce Database concepts Show example players Not Goals: Cover non-macos systems (Oracle) Teach you SQL Answer what

More information

YouTube Vitess. Cloud-Native MySQL. Oracle OpenWorld Conference October 26, 2015. Anthony Yeh, Software Engineer, YouTube. http://vitess.

YouTube Vitess. Cloud-Native MySQL. Oracle OpenWorld Conference October 26, 2015. Anthony Yeh, Software Engineer, YouTube. http://vitess. YouTube Vitess Cloud-Native MySQL Oracle OpenWorld Conference October 26, 2015 Anthony Yeh, Software Engineer, YouTube http://vitess.io/ Spoiler Alert Spoilers 1. History of Vitess 2. What is Cloud-Native

More information

Using MySQL for Big Data Advantage Integrate for Insight Sastry Vedantam sastry.vedantam@oracle.com

Using MySQL for Big Data Advantage Integrate for Insight Sastry Vedantam sastry.vedantam@oracle.com Using MySQL for Big Data Advantage Integrate for Insight Sastry Vedantam sastry.vedantam@oracle.com Agenda The rise of Big Data & Hadoop MySQL in the Big Data Lifecycle MySQL Solutions for Big Data Q&A

More information

MySQL Administration and Management Essentials

MySQL Administration and Management Essentials MySQL Administration and Management Essentials Craig Sylvester MySQL Sales Consultant 1 Safe Harbor Statement The following is intended to outline our general product direction. It

More information

Database Administration

Database Administration Unified CCE, page 1 Historical Data, page 2 Tool, page 3 Database Sizing Estimator Tool, page 11 Administration & Data Server with Historical Data Server Setup, page 14 Database Size Monitoring, page 15

More information

Administração e Optimização de BDs

Administração e Optimização de BDs Departamento de Engenharia Informática 2010/2011 Administração e Optimização de BDs Aula de Laboratório 1 2º semestre In this lab class we will address the following topics: 1. General Workplan for the

More information

MySQL Cluster Deployment Best Practices

MySQL Cluster Deployment Best Practices MySQL Cluster Deployment Best Practices Johan ANDERSSON Joffrey MICHAÏE MySQL Cluster practice Manager MySQL Consultant The presentation is intended to outline our general product

More information

SQL SERVER 2008 DATABASE MANAGEMENT. PART I: Writing Queries using MS Server 2008 Transact-SQL

SQL SERVER 2008 DATABASE MANAGEMENT. PART I: Writing Queries using MS Server 2008 Transact-SQL SQL SERVER 2008 DATABASE MANAGEMENT PART I: Writing Queries using MS Server 2008 Transact-SQL Module 1: Querying and Filtering Data Using the SELECT Statement Filtering Data Working with NULL Values Formatting

More information

database abstraction layer database abstraction layers in PHP Lukas Smith BackendMedia smith@backendmedia.com

database abstraction layer database abstraction layers in PHP Lukas Smith BackendMedia smith@backendmedia.com Lukas Smith database abstraction layers in PHP BackendMedia 1 Overview Introduction Motivation PDO extension PEAR::MDB2 Client API SQL syntax SQL concepts Result sets Error handling High level features

More information

Ultimate Guide to Oracle Storage

Ultimate Guide to Oracle Storage Ultimate Guide to Oracle Storage Presented by George Trujillo George.Trujillo@trubix.com George Trujillo Twenty two years IT experience with 19 years Oracle experience. Advanced database solutions such

More information

MySQL. Leveraging. Features for Availability & Scalability ABSTRACT: By Srinivasa Krishna Mamillapalli

MySQL. Leveraging. Features for Availability & Scalability ABSTRACT: By Srinivasa Krishna Mamillapalli Leveraging MySQL Features for Availability & Scalability ABSTRACT: By Srinivasa Krishna Mamillapalli MySQL is a popular, open-source Relational Database Management System (RDBMS) designed to run on almost

More information

Comparing Microsoft SQL Server 2005 Replication and DataXtend Remote Edition for Mobile and Distributed Applications

Comparing Microsoft SQL Server 2005 Replication and DataXtend Remote Edition for Mobile and Distributed Applications Comparing Microsoft SQL Server 2005 Replication and DataXtend Remote Edition for Mobile and Distributed Applications White Paper Table of Contents Overview...3 Replication Types Supported...3 Set-up &

More information

InfiniteGraph: The Distributed Graph Database

InfiniteGraph: The Distributed Graph Database A Performance and Distributed Performance Benchmark of InfiniteGraph and a Leading Open Source Graph Database Using Synthetic Data Objectivity, Inc. 640 West California Ave. Suite 240 Sunnyvale, CA 94086

More information

Sharding with postgres_fdw

Sharding with postgres_fdw Sharding with postgres_fdw Postgres Open 2013 Chicago Stephen Frost sfrost@snowman.net Resonate, Inc. Digital Media PostgreSQL Hadoop techjobs@resonateinsights.com http://www.resonateinsights.com Stephen

More information

Acronis Backup & Recovery: Events in Application Event Log of Windows http://kb.acronis.com/content/38327

Acronis Backup & Recovery: Events in Application Event Log of Windows http://kb.acronis.com/content/38327 Acronis Backup & Recovery: Events in Application Event Log of Windows http://kb.acronis.com/content/38327 Mod ule_i D Error _Cod e Error Description 1 1 PROCESSOR_NULLREF_ERROR 1 100 ERROR_PARSE_PAIR Failed

More information

Hadoop: A Framework for Data- Intensive Distributed Computing. CS561-Spring 2012 WPI, Mohamed Y. Eltabakh

Hadoop: A Framework for Data- Intensive Distributed Computing. CS561-Spring 2012 WPI, Mohamed Y. Eltabakh 1 Hadoop: A Framework for Data- Intensive Distributed Computing CS561-Spring 2012 WPI, Mohamed Y. Eltabakh 2 What is Hadoop? Hadoop is a software framework for distributed processing of large datasets

More information

Accessing External Databases from Mobile Applications

Accessing External Databases from Mobile Applications CENTER FOR CONVERGENCE AND EMERGING NETWORK TECHNOLOGIES CCENT Syracuse University TECHNICAL REPORT: T.R. 2014-003 Accessing External Databases from Mobile Applications Version 2.0 Authored by: Anirudh

More information

Dynamics NAV/SQL Server Configuration Recommendations

Dynamics NAV/SQL Server Configuration Recommendations Dynamics NAV/SQL Server Configuration Recommendations This document describes SQL Server configuration recommendations that were gathered from field experience with Microsoft Dynamics NAV and SQL Server.

More information

Performance rule violations usually result in increased CPU or I/O, time to fix the mistake, and ultimately, a cost to the business unit.

Performance rule violations usually result in increased CPU or I/O, time to fix the mistake, and ultimately, a cost to the business unit. Is your database application experiencing poor response time, scalability problems, and too many deadlocks or poor application performance? One or a combination of zparms, database design and application

More information

Recovery Principles in MySQL Cluster 5.1

Recovery Principles in MySQL Cluster 5.1 Recovery Principles in MySQL Cluster 5.1 Mikael Ronström Senior Software Architect MySQL AB 1 Outline of Talk Introduction of MySQL Cluster in version 4.1 and 5.0 Discussion of requirements for MySQL Cluster

More information

Establishing Environmental Best Practices. Brendan Law Blaw@td.com.au @FlamerNZ Flamer.co.nz/spag/

Establishing Environmental Best Practices. Brendan Law Blaw@td.com.au @FlamerNZ Flamer.co.nz/spag/ Establishing Environmental Best Practices Brendan Law Blaw@td.com.au @FlamerNZ Flamer.co.nz/spag/ Agenda Active Directory Service Accounts Database Platform Windows Platform Data Storage Planning Virtualisation

More information

Asterisk Cluster with MySQL Replication. JR Richardson Engineering for the Masses Hubguru@gmail.com

Asterisk Cluster with MySQL Replication. JR Richardson Engineering for the Masses Hubguru@gmail.com Asterisk Cluster with MySQL Replication JR Richardson Engineering for the Masses Hubguru@gmail.com Presentation Overview Reasons to cluster Asterisk Load distribution Scalability This presentation focuses

More information

Expert Oracle Exadata

Expert Oracle Exadata Expert Oracle Exadata Kerry Osborne Randy Johnson Tanel Poder Apress Contents J m About the Authors About the Technical Reviewer a Acknowledgments Introduction xvi xvii xviii xix Chapter 1: What Is Exadata?

More information

Gladinet Cloud Backup V3.0 User Guide

Gladinet Cloud Backup V3.0 User Guide Gladinet Cloud Backup V3.0 User Guide Foreword The Gladinet User Guide gives step-by-step instructions for end users. Revision History Gladinet User Guide Date Description Version 8/20/2010 Draft Gladinet

More information

SQL Databases Course. by Applied Technology Research Center. This course provides training for MySQL, Oracle, SQL Server and PostgreSQL databases.

SQL Databases Course. by Applied Technology Research Center. This course provides training for MySQL, Oracle, SQL Server and PostgreSQL databases. SQL Databases Course by Applied Technology Research Center. 23 September 2015 This course provides training for MySQL, Oracle, SQL Server and PostgreSQL databases. Oracle Topics This Oracle Database: SQL

More information

Designing, Optimizing and Maintaining a Database Administrative Solution for Microsoft SQL Server 2008

Designing, Optimizing and Maintaining a Database Administrative Solution for Microsoft SQL Server 2008 Course 50400A: Designing, Optimizing and Maintaining a Database Administrative Solution for Microsoft SQL Server 2008 Length: 5 Days Language(s): English Audience(s): IT Professionals Level: 300 Technology:

More information

Perforce Backup Strategy & Disaster Recovery at National Instruments. Steven Lysohir

Perforce Backup Strategy & Disaster Recovery at National Instruments. Steven Lysohir Perforce Backup Strategy & Disaster Recovery at National Instruments Steven Lysohir 1 Why This Topic? Case study on large Perforce installation Something for smaller sites to ponder as they grow Stress

More information

Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming. by Dibyendu Bhattacharya

Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming. by Dibyendu Bhattacharya Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming by Dibyendu Bhattacharya Pearson : What We Do? We are building a scalable, reliable cloud-based learning platform providing services

More information

Raima Database Manager Version 14.0 In-memory Database Engine

Raima Database Manager Version 14.0 In-memory Database Engine + Raima Database Manager Version 14.0 In-memory Database Engine By Jeffrey R. Parsons, Senior Engineer January 2016 Abstract Raima Database Manager (RDM) v14.0 contains an all new data storage engine optimized

More information

SQLITE C/C++ TUTORIAL

SQLITE C/C++ TUTORIAL http://www.tutorialspoint.com/sqlite/sqlite_c_cpp.htm SQLITE C/C++ TUTORIAL Copyright tutorialspoint.com Installation Before we start using SQLite in our C/C++ programs, we need to make sure that we have

More information

Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com

Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com What s Hadoop Framework for running applications on large clusters of commodity hardware Scale: petabytes of data

More information

Nortel Networks Symposium Call Center Server Symposium Database Integration User s Guide

Nortel Networks Symposium Call Center Server Symposium Database Integration User s Guide 297-2183-911 Nortel Networks Symposium Call Center Server Symposium Database Integration User s Guide Product release 5.0 Standard 1.0 April 2004 Nortel Networks Symposium Call Center Server Symposium

More information

High Availability Solutions with MySQL

High Availability Solutions with MySQL High Availability Solutions with MySQL best OpenSystems Day Fall 2008 Ralf Gebhardt Senior Systems Engineer MySQL Global Software Practice ralf.gebhardt@sun.com 1 HA Requirements and Considerations HA

More information

Oracle Database 11g Comparison Chart

Oracle Database 11g Comparison Chart Key Feature Summary Express 10g Standard One Standard Enterprise Maximum 1 CPU 2 Sockets 4 Sockets No Limit RAM 1GB OS Max OS Max OS Max Database Size 4GB No Limit No Limit No Limit Windows Linux Unix

More information

ANDROID APPS DEVELOPMENT FOR MOBILE GAME

ANDROID APPS DEVELOPMENT FOR MOBILE GAME ANDROID APPS DEVELOPMENT FOR MOBILE GAME Lecture 7: Data Storage and Web Services Overview Android provides several options for you to save persistent application data. Storage Option Shared Preferences

More information

DBA Tutorial Kai Voigt Senior MySQL Instructor Sun Microsystems kai@sun.com Santa Clara, April 12, 2010

DBA Tutorial Kai Voigt Senior MySQL Instructor Sun Microsystems kai@sun.com Santa Clara, April 12, 2010 DBA Tutorial Kai Voigt Senior MySQL Instructor Sun Microsystems kai@sun.com Santa Clara, April 12, 2010 Certification Details http://www.mysql.com/certification/ Registration at Conference Closed Book

More information

Optimizing Performance. Training Division New Delhi

Optimizing Performance. Training Division New Delhi Optimizing Performance Training Division New Delhi Performance tuning : Goals Minimize the response time for each query Maximize the throughput of the entire database server by minimizing network traffic,

More information

Online Transaction Processing in SQL Server 2008

Online Transaction Processing in SQL Server 2008 Online Transaction Processing in SQL Server 2008 White Paper Published: August 2007 Updated: July 2008 Summary: Microsoft SQL Server 2008 provides a database platform that is optimized for today s applications,

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Synchronous multi-master clusters with MySQL: an introduction to Galera

Synchronous multi-master clusters with MySQL: an introduction to Galera Synchronous multi-master clusters with : an introduction to Galera Henrik Ingo OUGF Harmony conference Aulanko, Please share and reuse this presentation licensed under Creative Commonse Attribution license

More information

SQL Query Performance Tuning: Tips and Best Practices

SQL Query Performance Tuning: Tips and Best Practices SQL Query Performance Tuning: Tips and Best Practices Pravasini Priyanka, Principal Test Engineer, Progress Software INTRODUCTION: In present day world, where dozens of complex queries are run on databases

More information

Integrating Big Data into the Computing Curricula

Integrating Big Data into the Computing Curricula Integrating Big Data into the Computing Curricula Yasin Silva, Suzanne Dietrich, Jason Reed, Lisa Tsosie Arizona State University http://www.public.asu.edu/~ynsilva/ibigdata/ 1 Overview Motivation Big

More information

Customer Bank Account Management System Technical Specification Document

Customer Bank Account Management System Technical Specification Document Customer Bank Account Management System Technical Specification Document Technical Specification Document Page 1 of 15 Table of Contents Contents 1 Introduction 3 2 Design Overview 4 3 Topology Diagram.6

More information

MongoDB Developer and Administrator Certification Course Agenda

MongoDB Developer and Administrator Certification Course Agenda MongoDB Developer and Administrator Certification Course Agenda Lesson 1: NoSQL Database Introduction What is NoSQL? Why NoSQL? Difference Between RDBMS and NoSQL Databases Benefits of NoSQL Types of NoSQL

More information

MS-50400 - Design, Optimize and Maintain Database for Microsoft SQL Server 2008

MS-50400 - Design, Optimize and Maintain Database for Microsoft SQL Server 2008 MS-50400 - Design, Optimize and Maintain Database for Microsoft SQL Server 2008 Table of Contents Introduction Audience At Completion Prerequisites Microsoft Certified Professional Exams Student Materials

More information

Configuring Apache Derby for Performance and Durability Olav Sandstå

Configuring Apache Derby for Performance and Durability Olav Sandstå Configuring Apache Derby for Performance and Durability Olav Sandstå Sun Microsystems Trondheim, Norway Agenda Apache Derby introduction Performance and durability Performance tips Open source database

More information

SQL Server. 2012 for developers. murach's TRAINING & REFERENCE. Bryan Syverson. Mike Murach & Associates, Inc. Joel Murach

SQL Server. 2012 for developers. murach's TRAINING & REFERENCE. Bryan Syverson. Mike Murach & Associates, Inc. Joel Murach TRAINING & REFERENCE murach's SQL Server 2012 for developers Bryan Syverson Joel Murach Mike Murach & Associates, Inc. 4340 N. Knoll Ave. Fresno, CA 93722 www.murach.com murachbooks@murach.com Expanded

More information

Apache Hadoop FileSystem and its Usage in Facebook

Apache Hadoop FileSystem and its Usage in Facebook Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs

More information

Databases and SQL. Homework. Matthias Danner. June 11, 2013. Matthias Danner Databases and SQL June 11, 2013 1 / 16

Databases and SQL. Homework. Matthias Danner. June 11, 2013. Matthias Danner Databases and SQL June 11, 2013 1 / 16 Databases and SQL Homework Matthias Danner June 11, 2013 Matthias Danner Databases and SQL June 11, 2013 1 / 16 Install and configure a MySQL server Installation of the mysql-server package apt-get install

More information

6. Storage and File Structures

6. Storage and File Structures ECS-165A WQ 11 110 6. Storage and File Structures Goals Understand the basic concepts underlying different storage media, buffer management, files structures, and organization of records in files. Contents

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the The Israeli Association of Grid Technologies July 15, 2009 Outline Architecture

More information

High Availability and Scalability for Online Applications with MySQL

High Availability and Scalability for Online Applications with MySQL High Availability and Scalability for Online Applications with MySQL Part 1I - Advanced Replication Ivan Zoratti Sales Engineering Manager EMEA ivan@mysql.com April 2007 Agenda Welcome back! and Welcome

More information