Oracle Database-Optimized Flash

Transcription

1 Oracle Database-Optimized Flash Setting New Standards for Database Performance Kothanda Umamageswaran Vice President, Exadata Development Gurmeet Goindi Technical Product Strategist

2 Flash Has Transformed Enterprise Storage Compared to Hard Disk Drives, Flash Is many orders of magnitude faster Has many orders of magnitude higher bandwidth Has extremely low latency Has wearing issues as it ages, but technology is catching up Is expensive, but the price gap is shrinking Every storage vendor has some flash based solution for your Database Q: Will my database realize the full benefit of flash technology? A: It will depend on how fast you can move the data from the flash to the database 2

3 Realizing Full Potential of Flash What s the right connectivity for flash SCSI attached SAS, SATA PCI Express attached What s the right storage architecture Server attached Storage attached Something else 3

4 PCI Express Vs SAS Connectivity PCI Express is orders of magnitude faster than SAS, and is getting faster PCI Express has the same characteristics as Flash High Throughput Low Latency PCIe has 13x throughput of SAS 4 8 Using legacy interconnects like SAS fundamentally bottlenecks flash drives SAS 6 Gbps SAS 12 Gbps PCIe 3.0 x4 PCIe 3.0 x8 Througput GB/s 4

5 SCSI Access Model SCSI was designed for tapes and HDDs HDDs are sequential whereas Flash devices are massively parallel Traditional IO stack is optimized for spinning media 512 Byte block size transfers Flash and databases do 4KB/8KB IOs Using legacy interfaces like SCSI fundamentally bottlenecks flash drives CPU 8 KB IO B B 512 B SCSI HBA SCSI 4 KB IO 5

6 PCI Express Flash with NVMe Interface Non Volatile Memory Express is a brand new ground up interface designed for flash NVMe is inherently parallel NVMe provides native atomic IO size affinity for databases NVMe IO stack massively reduces CPU utilization and latency CPU PCIe NVMe 2.2x NVMe is 2.2x Faster than SCSI PCI Express Flash with NVMe Interface is the right choice for your Database 6

7 How fast is PCIe NVMe Flash? Typical SATA HDD random read IOPS : 80 Typical SATA SSD random read 8k : 40,000 Typical PCIe NVMe SSD random read 8k : 250, X Usain Bolt s running speed: 28 mph Discovery Space Shuttle speed: 17,400 mph 622X Q: What s the best architecture to deliver this performance to my Database? 7

8 Where To Introduce A Flash Device SSD NAS/SAN PCI-E Flash Direct Attached Mount flash inside your server PCI-E or SATA/SAS Networked Storage Share device on the network (FC or 10GE) Popular implementations: Tiered Storage: Multiple tiers of disk drives (SSD, FC, SAS); data moves between these tiers SSD HDD SSD PCI-E Flash Hybrid Storage: Direct Attached flash on the storage controller and HDD in expansion shelves All Flash Arrays: All storage is some form of flash device either SSD or custom flash modules 8

9 Server Attached Flash PCI Express attached Flash I/O cards or NVMe drives Extremely fast, improves performance for currently active data Eliminates round trip between server and storage => Fastest I/O response Easy to install and configure, but usually not hot swappable SSD attached via SATA or SAS Slower than PCI Express Cheaper than PCI Express attached flash PCI-E Flash SSD 9

10 Server Attached Flash - Limitations Database limited to size of single server Storage limited by number of PCIe or disk slots in the server ANY component failure = loss of database access Firmware, Card, Driver, etc. Local second server with identical Flash recommended ($$$ x 2) MTBF of the Server decreases as the number of flash cards increases No Data Loss Requires Synchronous Log Shipping to standby Negates performance benefit of Flash writes Poor resource utilization Local storage results in islands of storage No IO resource management for prioritizing multiple databases NO HA 10

11 Tiered Or Hybrid Storage Hierarchy of storage tiers based on performance The Storage controller moves the data between these layers based on usage Effectiveness will depend on ability to detect the type of workload More resilient and scalable architecture compared to server attached flash Enables database HA and RAC Better storage resource utilization as it gets shared among multiple servers Enables database consolidation Server attached flash is usually faster then networked attached flash Storage Array 11

12 Tiered Or Hybrid Storage - Limitations The Storage Controller software determines which data is hot Has limited visibility to the content of the data Driven by usage patterns and not application needs Usage pattern changes as the database workload changes (OLTP Vs Reporting) makes the achieved performance less predictable Usually caches data based on yesterday s workload Storage Controllers have no Database awareness and will cache: Sequential read tables, or scans Backups, mirrored copies of blocks Limited throughput usually in low single digit GB/s 12

13 All Flash Arrays No spinning media, all SSD, or proprietary flash modules Best in class performance among storage based solutions Expensive, but new technology is driving the cost down Limited throughput usually in low single digit GB/s Most implementations don t scale very well Availability not at the same maturity level as traditional storage arrays. Are missing functionality that traditional arrays have added over the last two decades. Storage Array 13

14 Storage Arrays Fundamentally Bottleneck Flash Tiered, Hybrid or All Flash 10X Bottleneck! Even with 25x 10Gb/s Links Servers 25 GB/s Storage Array 250 GB/s Flash Bandwidth Flash Drives are much faster than SANs and LANs One PCI flash drive is 2X faster than a 10Gb Ethernet link 100 shared flash drives require Gb Ethernets between host and array! By 2016 PCI flash drives will be 4X faster than a 10Gb Ethernet link Shared Arrays throttle modern flash by an order of magnitude 14

15 Exadata Storage Eliminates the Flash Bottleneck Full Scale-Out, PCIe NVMe Flash, Application Offload, Database Aware Exadata Moves Queries to Storage, not Storage to Queries ONLY platform with shared storage benefits and NO FLASH BOTTLENECK State-of-the-art PCIe Flash with NVMe Scale out architecture avoids bottlenecks Adds flash capacity and performance by adding servers Database Aware Storage Metadata about IO present on the cell Flash on the Storage Server enables sharing unlike server cards SQL 40 Gb/s 40 Gb/s Smart Scan InfiniBand Switch 40 Gb/s 40 Gb/s 263 GB/s Scan Bandwidth PCIe 15

16 Oracle s Innovations in Flash 16

17 Oracle s Infrastructure Innovations in Flash Oracle Exadata V2: First to bring flash storage to the database market Oracle Exadata X3: Doubled flash capacity Oracle Exadata X4: 100GB/s throughput scans in a single rack Oracle Exadata X5: Lowest latency NVMe and increases scans to 140GB/s Oracle Exadata X5: Hot-pluggable NVMe server for the database Oracle Linux: First Linux vendor with production NVMe drivers 17

18 Oracle s Software Innovations in Flash Exadata Smart Flash Cache Exadata Smart Flash Log Exadata Smart Flash Cache Scan Awareness Exadata Smart File Initialization Exadata Smart Columnar Flash Cache Exadata Smart Flash Cache Space Resource Management 18

19 Exadata Smart Flash Cache Understands different types of I/Os from database Skips caching I/Os to backups, data pump I/O, archive logs, tablespace formatting Caches Control File Reads and Writes, file headers, data and index blocks Immediately adapts to changing workloads Unlike tiering that relies on historical statistics and is slow to move Tiering caches yesterday s hot data not today s Tiering uses large chunks (1MB) while cache responds faster with 64KB chunks Write-back flash cache Caches writes from the database not just reads RAC-aware from day one Doesn t need to mirror in flash for read intensive workloads Compare to: flash arrays that require flash cache in the server doubling cost 19

20 Exadata Smart Flash Log Outliers in log IO slow down lots of clients Log Writer Outliers from any one copy of mirror affect response time foreground Log Buffer foreground foreground Performance critical algorithms like space management and index splits are sensitive to log write latency client client client Legacy storage IO cannot differentiate redo log IO from others 20

21 Exadata Smart Flash Log Smart Flash Log uses flash as a parallel write cache to disk controller cache Whichever write completes first wins (disk or flash) Reduces response time and outliers log file parallel write histogram improves Greatly improves log file sync Uses almost no flash capacity (< 0.1%) Completely automatic and transparent Smart Logging - Off Smart Logging - On No Outliers 21

22 Exadata Smart Flash Cache Scan Awareness On a traditional cache, if you scan dataset larger than cache size Blocks 0,1,2,3 brought into cache, cache is full Block 20,21,22,23 say replaces 0,1,2,3 Repeat the same scan Block 0,1, 2, 3 will replace blocks 20,21,22,23 Block 20,21,22,23 will again replace block 0,1,2,3 Traditional caches churn with no actual benefit Some implementations call the insertion of new block in the middle scan resistant CACHE HOT Insert new block Churn COLD 22

23 Exadata Smart Flash Cache Scan Awareness Exadata Smart Flash Cache is scan resistant Ability to bring subset of the data into cache and not churn OLTP and DW scan blocks can co-exist Nested scans bring in repeated accesses Repeat, For each item in large table, scan small table Smart enough to pull the small table into flash since it is accessed repeatedly even though the size of large table alone is larger than flash cache No need to set KEEP attribute in data warehouses Happens automatically, no tuning or configuration needed CACHE HOT COLD 23

24 Exadata Smart File Initialization File Initialization Database Write Back Flash Cache Database Smart File Initialization Database Metadata Block IO Metadata Storage Cell Storage Cell Storage Cell I/Os in parallel I/Os to Flash Metadata Disks Flash Flash Disks Disks 24

25 Exadata Smart File Initialization Combine the benefits of previous Smart Initialization and Writeback Flash Cache Write file creation meta-data to writeback flash cache Tiny amount of flash space used to cache large portions of initialized data on disk File creation sped up by an order of magnitude Initialization I/Os to disk deferred or not performed if data loaded Create tablespace, file extensions, autoextend show benefit Redo log initialization included in Exadata Happens automatically, no tuning needed Database Metadata Storage Cell Metadata Flash Disks 25

26 Exadata Smart Columnar Flash Cache Hybrid Columnar Compression balances need for OLTP and Analytics As CPUs get faster want even faster scans Smart Flash Cache automatically transforms blocks from hybrid columnar to pure columnar for analytics during flashcache population Dual format representation for single row lookups Only selected columns read from flash during a query Up to 5x query speedup Completely automatic and transparent Compression Units select columna from table where Flash Cache Population Columns 26

27 Smart Flash Cache Space Resource Management FINANCE SALES SUPPLY CHAIN Flash Cache is a shared resource Database as a Service creates need for efficient resource sharing Specify minimum (flashcachemin) and maximum (flashcachelimit) sizes, or fixed allocations (flashcachesize), a database can use in the flash cache ALTER IORMPLAN - dbplan=((name=sales, flashcachesize=100g), - (name=finance,flashcachelimit=100g, flashcachemin=20g), - (name=schain, flashcachesize=200g)) Container database resource needs specified at the storage Pluggable database container resource limits expressed as percentages in the container database 27

28 Exadata Smart Flash Benefits Smart Flash Cache is database aware Smart Flash Logging avoids redo log outliers Smart Flash Cache Scan provides subset scanning and is table scan resistant Smart File Initialization creates a file by writing meta-data to flash cache Smart Columnar Flash Cache extends columnar benefit to storage Smart Flash Cache Space Resource Management provides granular control 28

29 Exadata X5-2 High Capacity (HC) 1.35 PB Disk per Full Rack Ultra-Fast PCIe Flash Cards 4x 1.6 TB PCIe Flash Cards per Server State-of-the-Art NVMe based PCIe flash New 12x High Capacity 8TB disk drives Smart Flash Cache gives performance of flash with capacity of disk Exadata storage offload - 16-cores/server * Full Rack : 8 DB servers, 14 storage servers Analytic Scans OLTP Reads (8K) OLTP Writes (8K) Flash OLTP Latency Rate per Rack* 140 GB/s 4.14 M IOPS 2.69 M IOPS M IOPS 29

30 New Exadata X5-2 Extreme Flash (EF) Industry s Highest I/O rates for OLTP and Analytics dramatically reduce cost per I/O State-of-the-art NVMe based PCIe flash Consistent, extremely low response times Exadata storage offload - 16-cores/server Ideal for I/O intensive workloads Deploy few servers to host huge I/O workloads 12.8 TB Ultra-Fast PCIe Flash per Server Rate per Rack* Analytic Scans 263 GB/s Random I/O (8K OLTP) 4.14 M IOPS Random 8K I/O Latency M IOPS * Standard Exadata Full Rack with 14 EF Storage Servers, 8 DB Servers 30

31 Copyright 2015, Oracle and/or its affiliates. All rights reserved. 31

32