Microsoft SQL Server PDW 新世代 MPP 資料倉儲解決方案

Transcription

1 DBI304 Microsoft SQL Server PDW 新世代 MPP 資料倉儲解決方案 徐園程 Sr. Technical Account Manager

2 微軟資料倉儲的願景 未來趨勢 Appliance PDW AU3 新功能 Hub & Spoke 架構運用 PDW & Big Data 大綱 客戶案例分享 與其他 MPP 比較

3 100% 75% Flat growth, good/ moderate commitment 50% DBMS Built for Transactions Mixed Workloads 25% DW Bundles Data Federation Columnar DBMS Streaming Data 0% Low-Power Hardware In-Memory DBMS SaaS Open Source Reporting Open Source Data Integration Open Source DBMS Public Cloud Areas of strategic investment for Microsoft 25% 50% 未來 3 年真正會成長 Good growth, moderate commitment SOA Open Source OS Software Appliance Declining usage despite commitment -25% Real-time DW Security Server Virtualization MDM Web Services DW Appliance SMP 64-bit Big Data MPP Decreasing Usage HA for DW DBMS Built for DW Blades in Racks -50% Data Quality Analytics within EDW Analytics Outside EDW Good growth, good commitment Advanced Analytics Centralized EDW 0% Narrow Commitment 客戶計畫使用 Broad Commitment 新世代資料倉儲平台功能一覽表 Good growth, small commitment 75% 100% Increasing Usage Source: TDWI Microsoft Confidential

4 未來三年資料倉儲使用空間 了解市場需求 資料倉儲所管理的資料量 Less than 1TB 41% 17% 21% 18% 1-3 TB 效能 : 分析大量資料的能力. 資料持續持續成長, 彈性擴充, 是考量 主要關鍵 10s of TBs, to 100s of TB, to PBs 19% 25% 3-10 TB 17% More than 10 TB 34% 2% 6% Don't Know 0% 10% 20% 30% 40% 50% In 3 years Today Source: TDWI Report Next Generation DW 大量使用 appliance model 主因在於 balanced appliance 的效能與彈性擴充能力. (MPP solutions) Jim Cobelius, Forrester Research

5 資料倉儲未來趨勢 DW Software License Revenue US$ Billions CAGR Share( 15) % 4.6% Private Cloud 7.1% 5.0% Appliances/RA 26.2% 30.0% Traditional -0.3% 60.4% Public Cloud FY10 FY11 FY12 FY13 FY14 FY15 Appliances 未來四年是主要趨勢 (2015: US: 40 億 ) Cloud DW longer-term 單機緩慢下降 Source: MS internal analysis, DBSMIT Cloud Market Opportunity Forecast

6 SQL server 2012 資料平台 Third-Party BI Applications Reporting Services Reports Excel Workbooks PowerPivot Applications 分析服務 報表服務 整合服務 Master Data Services Data Quality Services Stream Insight App Fabric SharePoint Dashboards and Scorecards BI Applications BI Platform EIM Platform Hadoop Windows /Azure Data Sources

7 微軟 MDS / QDS 解決方案 在 Excel 產生快速與簡單 model 強大 MDM 功能 : hierarchies, validation r ules, versions, and workflows De-duplication and matching through int egration with Data Quality Services DQS -> Data Quality Service 未來規劃 Data quality in the Azure cloud Data cleansing in Excel DQS API

8 企業資訊管理與資料倉儲 SQL Server DQS + MDS 解決方案 協同資料加值 公有資料 (of data, data services and data models) 連結外部 Public curated discover analyze create transform clean curate govern secure publish Publisher Information Workers & Developers 發布內部資料讓 user 挖掘 Microsoft Confidential data analyst POWER BI USER DATA STEWARD Data Scientist Analytics developer APP Developer Consumer Information Workers Business Decision DW + EIM Visibility 內外部資料 取得關聯 models Web data 內部資料 用戶端 secure govern transform publish Control IT PROs discover create clean Maker Business Data Manager

9 微軟領導資料倉儲與商業智慧 資料倉儲 Challengers 執 行 能 力 商業智慧 Challengers Leaders 執 行 能 力 Microsoft Niche players 發展目標完整性 Microsoft Niche players Visionaries As of February 2012 Leaders Visionaries 發展目標完整性 Microsoft exhibits one of the best value propositions on the market with a low cost and a highly favorable price/performance ratio - Gartner, February 2012 [Gartner, Inc., Magic Quadrant for Data Warehouse Database Management Systems Magic Quadrant, Mark A. Beyer, Donald Feinberg, Merv Adrian, Roxane Edjlali, February 6, The Magic Quadrant is copyrighted 2012 by Gartner, Inc. and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner's analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the "Leaders" quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. 9

10 資料倉儲全方位解決方案 Software Reference Architectures Fast Track for Appliances Dell Parallel Data Warehouse HP Enterprise Data Warehouse Dell Quickstart Data Warehouse HP Business Data Warehouse

11 比較 Fast Track 與 PDW 之差異性 Fast Track PDW (Parallel Data Warehousing ) 版本 AU3 資料量 1~80TB 20~500TB 架構 SMP (Symmetric MultiProcessing) 對稱多處理技術 1 SQL node MPP (Massive Parallel Processing ): 大規模並行處理技術 8~10 SQL nodes 高可用性 可建置 內建 主要觀點 從硬體與軟體平衡來加速效能與 降低 TCO. 快速建置 ( 同左 ) 大規模並行處理, Share Nothing, 簡易擴充 硬體廠商 HP, Dell, BULL, IBM HP, Dell 擴充性 Scale-Up Scale-Up & Scale-Out (~500TB)

12 SQL Server PDW 硬體架構 Control Rack Data Rack Storage Nodes Compute Nodes Control Nodes SQL Active / Passive SQL SQL Infiniband SQL SQL SQL SQL Dual Fiber Channel Landing Zone Dual Management Node SQL SQL SQL Backup Node SQL Spare Compute Node

13 SQL Server Parallel Data Warehouse Control Rack Overall Architecture Data Rack Control Node Client Interface (JDBC, ODBC, OLE-DB, ADO.NET) PDW Engine DMS Core Compute Node 1 PDW Agent DMS Manager PDW Agent DMS Core Compute Node 2 PDW Agent Legend: = = Landing Zone Node Bulk Data Loader ETL Interface PDW Agent Management Active Node Directory PDW Agent PDW service DMS= DMS Core PDW Agent Data Movement Service Compute Node 10

14 PDW 支援 SQL(Native) Client

15 PDW 核心效益 高擴充性 : 10s to 100s of TB 高效能 : MPP 大規模並行處理, 實體分散 Server workload Appliance: 軟硬體結合最佳化 sequential IO 低成本 : 真正 Share Nothing, Node 獨立運用所有硬體資 源 開放性 : Open Platform 可使新硬體技術容易導入, 不會區限於單一硬體廠商 靈活度 : 使用者建立的 Tables 以 replicated 或是 hash distributed 分散於 所有 nodes 選擇建置架構 Hub and Spoke 中央 DW 架構 高整合性 : 快速與簡易整合微軟 BI (SSIS,SSAS,SSRS) 完整工具支援 : BI, ETL, MDM and Streaming data Parallel loading 1 rack ~750GB/hr

16 PDW 管理 全部管理 Appliance 都透過單一管理介面 Console Dashboard Query activity Load activity Backup and restore Active locks Active sessions Alerts Appliance state

17 PDW Configuration Manager Appliance topology Services status Network configuration Privileges Central Configuration of SQL instances across each node

18 整合監控 System Center Full Management Pack for System Center provides key monitoring and alerting capabilities

19 透過 PDW Query 管理介面 看 create distribution table 過程

20 Hub & Spoke 架構運用案例 SQL Data Service DB Cloud OLAP SQL Azure Sync Framework Remote Table Copy OLTP PDW Central DW Hub DW Loader ETL Tools 20 Microsoft Confidential

21 Hub-And-Spoke 架構優點 所有系統都是透過高速網路連接再一起 Parallel database copy 速度高達 400 GB/min 簡易的 data mart ETL / ELT 流程 各部門預算獨立使用, 系統統一控管 每一部門 (BU) 可保有自己的分析超市, 無需更動 可即時滿足各部門 (BU) 原業務性之需求 當系統 Capacity 飽合時 將不用再各別添購 加強跨業務行銷能力 分開管理與使用者的工作 避免單一系統管理與使用都在同一套系統上 可整合 SMP/MPP, OLTP/OLAP & Cloud 系統 獨立延展或是建立子系統 增加節點 (spokes) 不會影響到其他使用者 Microsoft Confidential

22 規劃 PDW / TERADATA 架構範例 Proposed Architecture Data Sources Cloud Services Cloud BI Platform Parallel Data Warehouse In the future Data Mart Internal Users DM1 New Data SSIS Staging DM2 Web Portal Casual User Web APP Aggregation AS400/DB2 DM3 Raw data Sharepoint New Data and Aggregation Unix/Oracle DM4 test Power User Hourly Daily test ETL Area Oracle DB2 Informix Daily TERADATA Weekly Weekly Daily BI Applications Frond-End ELT Raw data Staging Real-time/Directly

23 及時整合多種資料庫 SSIS CDC Data Center SQL SSIS CDC /SSIS Attunity CDC AS400 DB2 S/390 DB2 Unix Oracle Windows SQL

24 PDW 透過 CDC 及時同步整 合 Oracle, DB2 & mysql

25 SQL Server PDW AU3 重點 Release Themes 效能再提高 Less work for the same results Do the same work more efficiently 與 SQL Server 整 合 BI, Analytics, & ETL 工具整合 Broader functionality Native Support for - Analysis Services - Reporting Services - PowerPivot Full Alignment Lay the foundation for broad connectivity support

26 SQL Server PDW AU3 架構 PDW AU3 Architecture with Shell DB User issues a query 2. Query is sent to the Shell through sp_sho wmemo_xml stored procedure SELECT SELECT Shell Appliance (SQL Server) SQL Server performs parsing, binding, authoriza tion Parallel execution plan generated 5. Parallel plan executes on compute nodes 6. Result returned to the user Compute Node (SQL Server) Plan Steps 4. Plan Steps MEMO containing candidate plans, histog rams, data types is generated Plan Steps 3. Return MEMO SQL optimizer generates execution alternatives Engine Service Control Node 1. Compute Node (SQL Server) Compute Node (SQL Server)

27 PDW AU3 新功能 MPP Cost Based Optimizer & DMS enhancements ½ Rack 10x faster query performance Cost-Based Optimization on MPP Engine Remove Data Type Conversions from DMS Entry Level Capacity for Massive Parallel Processing Upgrade Capability More Expressive Functions in PDW T-SQL T-SQL Enhancements configurations Seamless management of PDW Lower cost of Monitoring and Administration Extends integration to 3rd party with SAS connector Enhanced Integration Manageability Stored Procedures Tabular Data Stream Enhanced Support for SQL Server Applications Common Connectivity Native Communication Protocol Enriched Programmability Subset of SMP based SQL functionality International Language support Full International Language Support through increased Collation Types

28 AU3 Shell DB 自動 Cost Based 最佳化 Operator trees 簡單 query 語法 SELECT * from orders JOIN lineitem on (o_orderkey = l_orderkey) JOIN part on (l_partkey = p_partkey) WHERE p_name like '%smoke%'; PDW AU2 operator tree PDW AU3 operator tree (l_o = o_o) (l_pk = p_pk) shuffle P (p_pk) (l_pk) O (o_o) broadcast (l_o = o_o) LI (l_o) O (o_o) LI (l_o) P (p_pk)

29 PDW 整合 -- Informatica 架構 PDW Client Source Defn Target Defn Repository Source table Server Source Reader Target table TargetWriter Bulk Loader Bulk Loader used for Fast Loads ETL Fully Supported PDW Source and target within Informatica

30 目前支援的 BI 工具 AU3 T-SQL compatibility allows for common access for multi ple tools Current support on PDW drivers includes MicroStrategy SAP BusinessObjects Informatica Other tools have mixed experience Cognos support required : SET OPTION Core connectivity enhancements planned for the next 2 releases

31 PDW 整合 Hadoop Connector Config file BI-Directional Import/export interface Existing PDW Tools for loading / bulk loads SQOOP based adapter Landing Zone Node Bulk Data Loader Delimited File support PDW PDW agent HDFS dwsql

32 Importing HDFS 資料到 PDW 進階分 析 Application DBMS Power Programmers Admin BI Users SQOOP HADOOP Sensor/Blogs, Web RFID Docs Data Data SQL Server PDW Interactive BI/Data

33 Hadoop 透過 SQOOP 整合 PWD (export) SQOOP export with source (HDFS path) 1. & target (PDW DB & PDW table) 3. Copies incoming FTP data on Landing Serv Zone er 5. Hadoop 2. Read HDFS Connec data via tor mappers HDFS 4. Teln et Serv Invokes er DWLoader Compute Compute Node 1 Node 8 Landing Zone PDWconfiguration file Linux/ Window Hadoo s/pdw ControlCompute Node Nodes

34 PDW New Connector for Had oop SQOOP

35 TB+ 國內大型資料運用客戶 260 TB 台灣空照圖 ESRI 地理資訊整合運用 30T 大型資料倉儲 250~400 concurrent user 12 TB 行動數據資料倉儲 每日增加 40G, 自動化 Data loading 1.2 TB 通聯分析平台 150 億筆搜尋 ３５０筆 result set in 4 Sec 6 TB SQL Server 2008 on SAP 一天 250 萬 SAP dialog steps, 平均反應時間約 0.5 秒 2 TB of 5+ 1 cluster, High availability 網銀 for 7 X 24 access

36 Banking Core System in SQL Core Banking 分行 / 網銀 新光銀行 台灣銀行 富邦銀行 Data Warehouse 中華郵政 Bank of America First Premier Bank 36

37 Direct Edge from 4 hour 15 min

38 PDW 與 TERADATA 比較表 PDW High Value, Low Cost Solution Starting at $1.6M List Price And under $12K/TB Complete BI Solution TERADATA High TCO 2000 series starts at $32K/TB 6000 series starts at $57K/TB! Expensive projects, no/less service partners Incomplete BI Solution First class integration with Microsoft BI, ETL & MDM tools Integration with System Center No integrated BI tool Preferred BI & ETL Tools very expensive Likes to build inflexible self-made ETL processes Open and flexible: Vendor lock-in and inflexible Solution No proprietary hardware No Vendor lock-in Espouses Distributed Architecture 6000 series requires proprietary BYNET No Hardware choice Advocates Centralized DW Architecture 38

39 PDW 與 Exadata 比較表 PDW EXADATA Optimized Solution General Purpose Appliance High Value, Low Cost Solution Expensive Solution PDW is fully engineered for high scale DW Uses true MPP with balanced architecture Starting at $1.6M List Price And under $12K/TB Complete, Integrated BI Solution First class integration with Microsoft BI, ETL & MDM tools Compelling self-service BI Open and flexible: No proprietary hardware No Vendor lock-in Espouses Distributed Architecture Targets DW and OLTP but not ideal for both Database servers Use Shared resources and an unbalanced design. Result - performance issues at scale Full Rack X2-8 starts at $9.9M List Price And $35K per TB! Fragmented BI Solution Customers have to choose from a myriad of tools for BI and ETL Lacks compelling Self-service BI tools Vendor lock-in and inflexible Solution Locks customers to Sun hardware And expensive contracts 39

40 PDW 與 NETEZZA 比較表 PDW PDW is the EDW Hub: PDW is ideal as a EDW Hub Offers high concurrency and Mixed workloads Open and flexible: No proprietary hardware No Vendor lock-in Espouses Distributed Architecture High Value Solution Through integration with SQL Server, MS DW offers low cost solution Licensing discounts via EA and EAP IBM NETEZZA Netezza is a Data Mart Most of their customers use Netezza as a high performance Data Mart and rarely as a hub Netezza does not perform as well with mixed workloads Vendor lock-in Requires proprietary FPGA Card No Hardware choice Lock customer to IBM platform and services Total Solution is Not so cheap Despite low appliance cost, Netezza s total cost is not so cheap Add Expensive services from IBM Global Services! 40

41 PDW 與 GREENPLUM 比較表 PDW EMC GREENPLUM Complete BI Solution Incomplete BI Solution First class integration with Microsoft BI, ETL & MDM tools Integration with System Center No integrated BI tool Requires Analytical tools e.g. SAS or SAP BOBJ Also relies on 3rd party ETL tools High Concurrency: 1Poor Concurrency: PDW offers high concurrency And supports simultaneous data loading and querying High Quality Service Microsoft offers rich Partner ecosystem behind PDW. This includes GSIs like Accenture-Avanade We also offer a strong CoE Team from MCS and Premier Mission Critical Support to deliver high quality service to PDW customers Greenplum has real issues with concurrency It cannot run 100 queries simultaneously 1Unreliable Service Quality EMC is struggling to staff EDW projects with qualified personnel. Some GP customers are compained about the poor level of service Greenplum has a limited number of trained SI Partners 1. Source: Customer feedback obtained from market research conducted in APAC

42 SQL Server PDW Roadmap What is coming next? 2011 Q1 Shipped Q2 Appliance Update 1 Improved node manageability Better performance and reduced overhead OEM requests 2012 Q3 Shipp Appliance ed Q4 Update 2 Programmability Batches Control flow Variables Temp tables QDR infiniband switch Onboard Dell Q1 Shipp ed Appliance Q2 Update 3 Cost based optimizer Native SQL Server drivers, including JDBC Collations More expressive query language Data Movement Services performance SCOM pack Stored procedures (sub) Half-rack 3rd party integration (Informatica, MicroStrategy, Business Objects, HADOOP) Q3 Q4 V-Next Columnar store index Stored procedures Integrated Authentication PowerView integration Workload management LZ/BU redundancy Windows 8 SQL Server 2012 Hardware refresh

43 Resources Connect. Share. Discusss Microsoft Certification & Training Resources Resources for IT Professionals Resources for Developers / /

44