Windows Server Infrastructure for SQL Server

Transcription

1 Windows Server Infrastructure for SQL Server Michael Frandsen

2 Agenda SQL Server storage challenges The SAN legacy Traditional interconnects SMB past New old interconnects File Shares the new Black DIY Shared Storage Microsoft vnext HP vnext

3 Bio - Michael Frandsen I have worked in the IT industry for just over 22 years, 18 of these has been spent as a consultant. I have a close relationship with Microsoft R&D in Redmond, with the Windows team for 19 years, ever since the first beta of Windows NT 3.1, SQL server for 19 years, since the first version Microsoft did by themselves, v4.21a I am in various advisory positions in Redmond and am involved in vnext versions of Windows, Hyper-V, SQL Server and Office/SharePoint. Currently Windows Blue (Windows 8.1 & Windows Server 2012 R2) and SQL14 (SQL Server 2014) Specialty areas: Architecture & design High performance Storage Low Latency Kerberos Scalability (scale-up & scale-out) Consolidation (especially SQL Server) High-Availability VLDB Data Warehouse platforms BI platforms High Performance Computing (HPC) clusters Big Data platforms & architecture

4 Bio - Michael Frandsen

5 MentalNote Independent One-Man company with no affiliations Only selling consultancy deliverables No hardware sales No software sales Mission Excellence: Delivering knowledge and empowering clients through extensive and deep knowledge, always based on facts and objectiveness. This is achieved through a constant development of skills, by both internal research and partnerships with leading companies in the IT industry, such as Microsoft, HP and others, not only locally but mainly with these companies' Research and Development departments at their respective headquarters. Responsibility: Giving back to people, animals and places in need of support. Done with both manpower and funding of various causes, such as Cancer Research, Heart illness Animal Protection Terminally sick children, Orphans Developing Countries, Areas hit by natural disaster

6 SQL Server storage challenges Capacity Fast Shared Reliable

7 Performance The SAN legacy Because it s expensive it must be fast Price

8 The SAN legacy Shared storage or Direct Attached SAN File Server 2 x 8Gb/s Database Server 2 x 8Gb/s Mail Server 2 x 8Gb/s SAN 2 x 8Gb/s

9 The SAN legacy Widespread misconception

10 CACHE SQL SERVER WINDOWS CPU CORES MPIO Algorithm MPIO DSM WWN Zoning FC SWITCH Port Logic XOR Engine SCSI Controller The SAN legacy Complex stack FC HBA FC HBA A B A B A B CACHE STORAGE CONTROLLER A B A B DISK DISK LUN LUN DISK DISK SQL Server Read Ahead Rate CPU Feed Rate HBA Port Rate Switch Port Rate SP Port Rate LUN Read Rate Disk Feed Rate

11 SAN Bottleneck Typical SAN load: Low to medium I/O processor load (top - slim rectangles) Low cache load (Middle - big rectangles) Low disk spindle load (lower half - squares)

12 SAN Bottleneck Typical Data Warehouse / BI / VLDB SAN load: High I/O processor load maxed out (top - slim rectangles) High cache load (Middle - big rectangles) Low disk spindle load (lower half - squares)

13 SAN Bottleneck Ideal Data Warehouse / BI / VLDB SAN load: Low to medium I/O processor load (top - slim rectangles) Low to medium cache load (Middle - big rectangles) High disk spindle load (lower half - squares)

14 Traditional interconnects Fibre Channel Stalled at 8Gb/s for many years 16Gb/s FC still very exotic Strong movement towards FCoE (Fibre Channel over Ethernet) iscsi Started in low-end storage arrays Many still 1Gb/s 10Gb/E storage arrays typically have few ports compared to FC NAS NFS, SMB, etc.

15 File Share reliability Is this mission critical technology?

16 SMB Commands Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP_ANDX, TRANS2_SESSION_SETUP, LOGOFF_ANDX, PROCESS_EXIT, TREE_CONNECT, TREE_CONNECT_ANDX, TREE_DISCONNECT) File, directory and volume access (CHECK_DIRECTORY, CLOSE, CLOSE_PRINT_FILE, COPY, CREATE, CREATE_DIRECTORY, CREATE_NEW, CREATE_TEMPORARY, DELETE, DELETE_DIRECTORY, FIND_CLOSE, FIND_CLOSE2, FIND_UNIQUE, FLUSH, GET_PRINT_QUEUE, IOCTL, IOCTL_SECONDARY, LOCK_AND_READ, LOCK_BYTE_RANGE, LOCKING_ANDX, MOVE, NT_CANCEL, NT_CREATE_ANDX, NT_RENAME, NT_TRANSACT, NT_TRANSACT_CREATE, NT_TRANSACT_IOCTL, NT_TRANSACT_NOTIFY_CHANGE, NT_TRANSACT_QUERY_QUOTA, NT_TRANSACT_QUERY_SECURITY_DESC, NT_TRANSACT_RENAME, NT_TRANSACT_SECONDARY, NT_TRANSACT_SET_QUOTA, NT_TRANSACT_SET_SECURITY_DESC, OPEN, OPEN_ANDX, OPEN_PRINT_FILE, QUERY_INFORMATION, QUERY_INFORMATION_DISK, QUERY_INFORMATION2, READ, READ_ANDX, READ_BULK, 14 distinct READ_MPX, WRITE READ_RAW, RENAME, SEARCH, SEEK, SET_INFORMATION, SET_INFORMATION2, TRANS2_CREATE_DIRECTORY, TRANS2_FIND_FIRST2, TRANS2_FIND_NEXT2, TRANS2_FIND_NOTIFY_FIRST, TRANS2_FIND_NOTIFY_NEXT, TRANS2_FSCTL, TRANS2_GET_DFS_REFERRAL, operations?!?? TRANS2_IOCTL2, TRANS2_OPEN2, TRANS2_QUERY_FILE_INFORMATION, TRANS2_QUERY_FS_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_REPORT_DFS_INCONSISTENCY, TRANS2_SET_FILE_INFORMATION, TRANS2_SET_FS_INFORMATION, TRANS2_SET_PATH_INFORMATION, TRANSACTION, TRANSACTION_SECONDARY, TRANSACTION2, TRANSACTION2_SECONDARY, UNLOCK_BYTE_RANGE, WRITE, WRITE_AND_CLOSE, WRITE_AND_UNLOCK, WRITE_ANDX, WRITE_BULK, WRITE_BULK_DATA, WRITE_COMPLETE, WRITE_MPX, WRITE_MPX_SECONDARY, WRITE_PRINT_FILE, WRITE_RAW) Other (ECHO, TRANS_CALL_NMPIPE, TRANS_MAILSLOT_WRITE, TRANS_PEEK_NMPIPE, TRANS_QUERY_NMPIPE_INFO, TRANS_QUERY_NMPIPE_STATE, TRANS_RAW_READ_NMPIPE, TRANS_RAW_WRITE_NMPIPE, TRANS_READ_NMPIPE, TRANS_SET_NMPIPE_STATE, TRANS_TRANSACT_NMPIPE, TRANS_WAIT_NMPIPE, TRANS_WRITE_NMPIPE)

17 SMB Commands Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP, LOGOFF, TREE_CONNECT, TREE_DISCONNECT) File, directory and volume access (CANCEL, CHANGE_NOTIFY, CLOSE, CREATE, FLUSH, IOCTL, LOCK, QUERY_DIRECTORY, QUERY_INFO, READ, SET_INFO, WRITE) Other (ECHO, OPLOCK_BREAK) TCP is a required transport SMB2 no longer supports NetBIOS over IPX, NetBIOS over UDP or NetBEUI

18 SMB 2.1 Performance improvement Up to 1MB MTU to better utilize 10Gb/E! Disabled by default! Real benefit required app support Ex. Robocopy in W7 / 2K8R2 is multi-threaded Defaults to 8 threads, range 1-128

19 SQL Server SMB support < 2008 Using UNC path could be enabled with trace flag Not officially supported scenario No support for system databases No support for failover clustering 2008 R2 UNC path fully supported by default No support for system databases No support for failover clustering

20 Two things happened SQL Server 2012 Windows Server 2012

21 SQL Server 2012 UNC support expanded System Databases supported on SMB Failover Clustering supports SMB as shared storage and TempDB can now reside on NON-shared storage Mark Souza commented: Great Suggestion!

22 Windows Server 2012 InfiniBand Teaming SMB 3.0 RDMA Multichannel SMB Direct

23 New old interconnects InfiniBand characteristics Been around since 2001 Used mainly for HPC clusters and Super Computing High throughput RDMA capable Low latency Quality of service Failover Scalable

24 InfiniBand throughput

25 InfiniBand throughput Trends in I/O Interfaces with Servers

26 InfiniBand throughput Low-level Uni-directional Bandwidth Measurements

27 File Shares the new Black Why file shares? Massively increased stability Cleaned up protocol Transparent Failover between cluster nodes with no service outage! Massively increased functionality Multichannel RDMA and SMB Direct Massively decreased complexity No more MPIO, DSM, Zoning, HBA tuning, Fabric zoning etc.

28 New protocol - SMB 3.0 Which SMB protocol version is used Client / Server OS Windows 8 Windows Server 2012 Windows 7 Windows Server 2008 R2 Windows Vista Windows Server 2008 Previous versions of Windows Windows 8 Windows Server 2012 Windows 7 Windows Server 2008 R2 Windows Vista Windows Server 2008 Previous versions of Windows SMB 3.0 SMB 2.1 SMB 2.0 SMB 1.0 SMB 2.1 SMB 2.1 SMB 2.0 SMB 1.0 SMB 2.0 SMB 2.0 SMB 2.0 SMB 1.0 SMB 1.0 SMB 1.0 SMB 1.0 SMB 1.0

29 Transparent Failover SQL Server or Hyper-V Server Normal operation \\fs1\share Failover to Node B Connections and handles autorecovered; application IO continues with no errors \\fs1\share File Server Node A File Server Node B File Server Cluster

30 SMB Multichannel Single RSS-capable SMB Client RSS Multiple 1GbE s 1GbE SMB Client 1GbE Multiple in a team SMB Client Teaming Multiple RDMA s /IB SMB Client /IB Full Throughput Bandwidth aggregation with multiple s Multiple CPUs cores engaged when using Receive Side Scaling (RSS) Switch RSS SMB Server Switch 1GbE 1GbE SMB Server Switch 1GbE 1GbE Switch Teaming SMB Server Switch Switch /IB /IB SMB Server Switch /IB /IB SMB Multichannel implements end-to-end failure detection Leverages teaming if present, but does not require it Automatic Configuration SMB detects and uses multiple network paths

31 SMB Multichannel RSS SMB Client CPU utilization per core Switch RSS SMB Server Core 1 Core 2 Core 3 Core 4

32 SMB Multichannel RSS SMB Client CPU utilization per core Switch RSS SMB Server Core 1 Core 2 Core 3 Core 4

33 SMB Multichannel RSS SMB Client 1 RSS SMB Client 2 Switch Switch Switch Switch RSS RSS SMB Server 1 SMB Server 2

34 SMB Multichannel RSS SMB Client 1 RSS SMB Client 2 Switch Switch Switch Switch RSS RSS SMB Server 1 SMB Server 2

35 MB/sec SMB Multichannel Performance SMB Client Interface Scaling - Throughput x 2 x 3 x 4 x I/O Size

36 RDMA in SMB 3.0 SMB over TCP and RDMA User Client 1 Application Unchanged API Memory 4 RDM A Memory File Server 1. Application (Hyper-V, SQL Server) does not need to change. Kernel SMB Client SMB Server 2. SMB client makes the decision to use SMB Direct at run time TCP/ IP NDKPI 2 SMB Direct SMB Direct NDKPI TCP/ IP 3. NDKPI provides a much thinner layer than TCP/IP No longer flow anything via regular TCP/IP 3 RDMA RDMA 4. Remote Direct Memory Access performed by the network interfaces. Ethernet and/or InfiniBand

37 SMB Direct and SMB Multichannel SMB Client 1 SMB Client 2 R- 54GbIB R- 54GbIB R- R- Switch 54GbIB Switch 54GbIB Switch Switch R- 54GbIB R- 54GbIB R- R- SMB Server 1 SMB Server 2

38 SMB Direct and SMB Multichannel SMB Client 1 SMB Client 2 R- 54GbIB R- 54GbIB R- R- Switch 54GbIB Switch 54GbIB Switch Switch R- 54GbIB R- 54GbIB R- R- SMB Server 1 SMB Server 2

39 DIY Shared Storage New paradigm for SQL Server storage design Direct Attached Storage (DAS) Now with flexibility Converting DAS to shared storage Fast RAID controllers will be shared storage NAND Flash PCIe cards will be shared storage

40 New Paradigm designs Fusion Fusion Fusion PCIe Flash IO IO IO

41 New Paradigm designs File Server File Server NAND Flash Shared Storage Traditional SAN Shared Storage

42 IODuo IODuo New Paradigm designs SQL Server Cluster Node 1 HP DL980 SQL Server Cluster Node 2 HP DL980 Windows Server 2012 Windows Server 2012 HP IO Accelerator Flash SQL TempDB PCIe 2 x 56Gb/s 2 x 56Gb/s PCIe HP IO Accelerator Flash SQL TempDB 2 x 36 port Mellanox Infiniband Switch Windows Server 2012 Cluster Node 1 HP DL580 2 x 56Gb/s 2 x 56Gb/s Windows Server 2012 Cluster Node 2 HP DL580 4 x 8Gb/s 4 x 8Gb/s 3PAR 7000-series Shared Flash Storage SQL Data / Log

43 Demo

44 Demo

45 Demo

46 Demo

47 SQL Server Virtualization - storage challenges Capacity Fast Shared Reliable

48 Hyper-V v3.0 Only two goals: Adopt new technologies in the Win8 kernel Be the best hypervisor for SQL Server

49 Hyper-V v3.0 How do you become the best hypervisor for SQL Server?

50 Hyper-V v3.0 Microsofts initial idea up to November % 35% 36% 33% Servers by number of CPU Sockets 30% 25% 20% 21% 15% 10% 5% 0% 6% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 0%

51 Hyper-V v3.0 New insight for the Hyper-V Team SQL instances by number of CPU Sockets 40% 35% 36% 30% 25% 24% 20% 15% 10% 16% 12% 11% 5% 0% 1% 0% 0% 0% 0% 0% 0% 0% CPU s

52 Hyper-V Team idea of Physical to Virtual Before: 750 Servers with SQL Server 920 SQL Server Instances 200TB Storage After: Servers with Hypervisor and SQL Server 920 SQL Server Instances 200TB Storage

53 Real life consolidation on Physical servers Before: 750 Servers with SQL Server 920 SQL Server Instances 200TB Storage After: 6 Servers with SQL Server 12 SQL Server Instances 140TB Storage

54 Real life consolidation on Physical servers How did we achieve the Storage savings? Databases by type 63% 37% System User

55 Share of disk space Digging deeper Further storage re-claims could easily be done in databases 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Disk capacity waste in SQL environments (350TB) 83% 57% 17% Allocated disk space Allocated database space Used database space Total free space

56 Final specs of Hyper-V v3.0 Capability Hyper-V Server 2008 R2 Hyper-V Server 2012 Number of logical processors on host Maximum supported RAM on host 1 TB 4 TB Virtual CPUs supported per host Maximum virtual CPUs supported per virtual machine 4 64 Maximum RAM supported per virtual machine 64 GB 1 TB Maximum running virtual machines supported per host Guest NUMA No Yes Maximum failover cluster nodes supported Maximum number of virtual machines supported in failover clustering

57 Final specs of Hyper-V v3.0 So what about Storage? VMware tops out at 300,000 IOPS per VM A really good number A single Windows Server 2012 Hyper-V VM does: 985,000 IOPS

58 New Paradigm designs File Server File Server NAND Flash Shared Storage Traditional SAN Shared Storage

59 New things are happening SQL Server 2014 (SQL14) Windows Server 2012 R2 (Windows Blue)

60 Windows Server 2012 R2 RTM September 5 th 2013 Both Server and Client Win8.1 Hyper-V v IOPS -> IOPS Improved network performance IOPS/ Improved Storage Spaces Caching Tiered Storage -> IOPS/ 8K I/O from 950 MB/s -> MB/s

61 SQL Server 2014 Still in development Project Hekaton In-Memory OLTP Columnstore Index Clustered & Updateable Updated Always-On Improved reliability and scalability 8 replicas Completely New Query Engine For the first time control of IOPS with resource policies Buffer Pool Extension

62 Re-think hardware usage Storage Memory

63 Re-think hardware usage Storage Memory L5 Cache L4 Cache

64 Re-think hardware usage Storage Memory L2 RAM L1 RAM

65 SSD Buffer Pool Extensions What s being delivered: Usage of non-volatile drives (SSD) to extend buffer pool NUMA aware Large page and BUF array allocation Main benefits: BP Extension for SSDs Improve OLTP query performance with no application changes No risk of data loss (using clean pages only) Easy configuration optimized for OLTP workloads on commodity servers (32GB RAM) Scalability improvements for systems with >8 sockets

66 Data Buffer Pool Manager Query Tree Cmd Parser Command TDS TDS Query Plan Optimizer Query Execut or Result Sets SNI Relational Engine Results Protocol Layer Plan Cache Transaction Log Access Methods GetPage D Data Cache Cached Pages Data Files Write I/O Transaction Manager Storage Engine Buffer Manager Read I/O Buffer Pool

67 Data IOPS Offload to Storage Class Memory (SCM) Query Tree Cmd Parser Command TDS TDS Query Plan Optimizer Query Execut or Result Sets SNI Relational Engine Results Protocol Layer Plan Cache Transaction Log Access Methods GetPage Cached Pages D Data Cache L2 Buffer Pool Data Files Write I/O Transaction Manager Storage Engine Buffer Manager Read I/O Buffer L1 Buffer Pool Pool PCIe Flash Fusion Fusion Fusion (SCM) IO IO IO

68 Easy enablement

69 Troubleshooting options DMVs sys.dm_os_buffer_pool_extension_configuration sys.dm_os_buffer_descriptors XEvents sqlserver.buffer_pool_extension_pages_written sqlserver.buffer_pool_extension_pages_read sqlserver.buffer_pool_extension_pages_evicted sqlserver.buffer_pool_page_threshold_recalculated Performance Monitor Counters Extension page writes/sec Extension page reads/sec Extension outstanding IO counter Extension page evictions/sec Extension allocated pages Extension free pages Extension page unreferenced time Extension in use as percentage on buffer pool level

70 New things are happening in hardware

71 HP vnext Cluster-in-a-box HP X5000 DL580 G8 Ivy Bridge EX (4+ sockets) Dragonhawk SuperDome2 with x64 and Windows Moonshot Many servers in very little space at low power

72 Thank you Business Card: Above on LinkedIn