Technical Paper Best Practices fr SAS n EMC VNX Unified Strage
Release Infrmatin Cntent Versin: 1.0 February 2014. Trademarks and Patents SAS Institute Inc., SAS Campus Drive, Cary, Nrth Carlina 27513. SAS and all ther SAS Institute Inc. prduct r service names are registered trademarks r trademarks f SAS Institute Inc. in the USA and ther cuntries. indicates USA registratin. Other brand and prduct names are registered trademarks r trademarks f their respective cmpanies. Statement f Usage This dcument is prvided fr infrmatinal purpses. This dcument may cntain appraches, techniques and ther infrmatin prprietary t SAS.
Cntents Intrductin... 1 Brief Overview f VNX Architecture... 1 Physical Strage Media Types... 1 Flash SLC versus emlc... 2 VNX Operating Envirnment... 2 VNX Architecture Decisins fr SAS... 2 Cnsideratins and Best Practices fr Using Thin Pls and FAST VP...3 Cnsideratins and Best Practices fr Using Dedicated LUNs... 5 Thrughput Testing...5 Cnclusin...5 References...5 Resurces...6 Cntact Infrmatin...6 i
Intrductin The EMC VNX Unified Strage System is a scalable, highly cnfigurable, and easy-t-manage strage subsystem answering the needs f strage perfrmance fr small t medium sized SAS wrklads. EMC VNX Unified Strage Systems can be cnfigured t cver a brad range f SAS wrklad types, frm thse that can benefit frm virtualizatin, tiered strage, and autmatin; t thse requiring mre dedicated resurces. The varius wrklads have specific strage needs and requirements fr maximum applicatin perfrmance. This technical paper will utline best practices fr architectural setup and tuning t maximize SAS Applicatin perfrmance with EMC VNX Unified strage. An verview f the strage system will be discussed first, including physical and virtual architecture, pling and virtualizatin, strage tiers and management. This will be fllwed by a list f practical recmmendatins fr implementatin with SAS. Brief Overview f VNX Architecture In cntrast t the previus array we cvered, the *EMC SYMMETRIX VMAX (listed belw this paragraph), the EMC VNX des nt prvide a scale-ut Virtual Matrix architecture t build a large hlistic, shared system. Instead, the VNX sprts a dual cntrller architecture. In dual cntrller systems, ne cntrller is usually active, with the ther passive. Servicing the small t mid-range market, the VNX is intended t act as a mre lcal array versus the enterprise serving VMAX. While still prviding enterprise strage perfrmance, the VNX systems tend t have mre lcal rles supprting prject-based and segregated-usage strage systems. Instead f a single scale up, strage as a service system, the VNX is mre scale-ut by adding additinal separate VNX Systems. VNX still supprts many f the ppular features intrduced in the VMAX, including strage virtualizatin, pl based strage, autmated strage tier management, etc. It als integrates slid-state strage fr applicatin usage, and fr FAST CACHE where that is applicable. The EMC white paper EMC Unified VNX Strage cvers best practices fr VNX strage and is highly recmmended fr reading - http://www.emc.cm/cllateral/sftware/white-papers/h10938-vnx-best-practices-wp.pdf. *The Best Practices fr SAS n EMC SYMMETRIX VMAX Strage cvers many f the riginal and still applicable pints arund pls, virtualizatin, tier plicies, and prmtin - http://supprt.sas.cm/resurces/papers/emc_sas_vmax_best_practices.pdf. Recently intrduced t the market are new VNX mdels, dubbed VNX2. There are substantial imprvements between VNX and the newer VNX2 mdels: The previus VNX mdels nly utilized Single Layer Cell Enterprise Flash Drives (SLC EFDs), the VNX2 als ffers Enterprise Multi-Layer Cell Enterprise Flash Drives (emlc EFDs). Mre n this later. Cntrller speed is 4 times faster n the VNX2, utilizing better prcessrs Lgical Blck Assignment mves n tiers were made at 1 GB chunks n VNX, VNX2 supprts a reduced chunk size f 256K. This can make tier migratins mre efficient and less disruptive. The strage prcessr cache is n lnger tunable. Memry allcatin amunts between Read and Write, cache page size, and cache watermarks are n lnger cnfigurable. The cache is nw self-learning and aut-tunes. The increased capacity f the VNX2 is ptimized fr up t 30 GB/sec fr large blck, sequential, IO. It als nw supprts up t 32 cres n virtual machines. Imprved flash drive handling has als been incrprated. The VNX supprted small t medium systems up t 150 drives in capacity. The VNX2 systems supprt medium systems up t 1500 drives in capacity, and nw ffer FAST (Fully Autmated Strage Tiering), and supprts a brader range f prtcls such as FC, FCE, and pnfs. Physical Strage Media Types The underlying physical strage in the VNX system cnsists f 3 basic media types, r strage basic tiers, listed belw frm highest perfrming t lwest perfrming drive technlgies in terms f perfrmance: 1
Tier 1 EMC Enterprise Flash Drives(EFD) available as Single Layer Cell EFD (SLC EFD), and enhanced Multi-Layer Cell EFD (emlc EFD) Tier 2 SAS Drives (SAS 10K and SAS 15K Drives) Tier 3 NL-SAS (SATA Drives Cnnected via a SAS Interface) Nte: ** Drive capacities may vary by installatin. Check with EMC representative fr yur cnfiguratin These strage tiers prvide the least (Tier 3 NL-SAS) t the highest (Tier 1 - FLASH) perfrmance and cst per Gigabyte. NL-SAS devices are large, slwer drives, with much higher respnse times than SAS r FLASH devices. Read Miss respnse times are in the 12ms range cmpared t Flash Drives at 1 ms r 15K FC devices at 6 ms. Since SAS System Applicatins have a large-blck, sequential rientatin, this paper will fcus n attributes t supprt thrughput riented perfrmance. Flash SLC versus emlc What shuld yu use fr SAS Wrklads? SLC r emlc Flash? Despite the chart abve, SLC EFDs have cnsiderably lwer write service times than emlc EFDs, and have a much lnger cell life (*see the link belw fr further reading). Given SAS prpensity fr writing large amunts f data, this culd be a cmparative pint fr yu. Hwever, emlc EFDs are nw utilizing mre advanced wear leveling techniques, and emplying better strategies fr garbage cllectin and reduced write amplificatin t give them lnger cell life. Their perfrmance has climbed significantly and has greatly reduced the gap between MLC and SLC flash. Still, a large part f the verall perfrmance in a flash strage system lies in the cntrller speed and handling. S the significant cst difference between SLC EFDs and emlc EFDS will prbably be ne f the deciding factrs n which f these flash technlgies t emply. If a prtin f yur wrklad is Very IO intensive, and requires the fastest pssible perfrmance, regardless f price, then cnsider using SLC EFDs. The use f these mre expensive devices may be prhibitive fr deplying t yur entire SAS data system, and emlc Flash and SAS spinning drives perfrm quite admirably fr the general wrklad. * http://www.cmputerweekly.cm/feature/mlc-vs-slc-which-flash-ssd-is-right-fr-yu VNX Operating Envirnment The VNX Operating Envirnment can be implemented as a File System r Blck-based system. Discussins in this paper will deal nly with VNX aspects running under VNX OE Blck implementatin. Readers are strngly encuraged t read pages 24 26 f the Best Practices fr SAS n EMC SYMMETRIX VMAX Strage. VNX Architecture Decisins fr SAS Many lng-term SAS Custmers have migrated thrugh years f strage technlgy, acrss increasingly cmplex strage systems architectures. Over the lng haul, there are standard paradigms that have nt changed in terms f successful strage architecture fr SAS. These paradigms still apply tday, regardless f what strage technlgy yu attempt t implement. The fllwing paper details thse cnsideratins: http://supprt.sas.cm/rnd/papers/sgf07/sgf2007-isubsystem.pdf. There are tw pertinent issues when using thin prvisining t virtualize strage fr SAS applicatins. The first is under-prvisining n spindle alne; the secnd is ver-subscriptin f the physical capacity. Thin prvisining is ften implemented n much larger, slwer drives t gain capacity cheaply. Ding this reduces the actual number f disks needed. Since SAS is I/O thrughput riented (see the paper nted in the previus paragraph), this can be detrimental. Thrughput aggregatin is attained by striping acrss many disks, and aggregating the thrughput f each disk in a single stripe when reading r writing. When fewer disks are invlved, lesser thrughput can be attained. This is detrimental t large SAS wrklads. If utilizing thin pls, use wide striping t attain higher thrughput. Anther gal f thin-prvisining virtualizatin typically includes prviding thin pls f strage that multiple applicatins (and hence varying wrklads and types) will share. The physical prvisining underneath the thin pls is nt sufficient t cver the defined lgical space n defined LUNs. Hence the term thin. This under-prvisining f the actual ttal physical space available, t the sum f the Lgical LUN spaces prvided is called ver-subscriptin. The paradigm banks n the fact that nt all virtual custmers will hit the thin pl at nce, with a wrklad that wuld simultaneusly demand all their defined lgical LUN extents. Unfrtunately that is exactly what SAS ad hc wrklad envirnments are easily capable f engendering. Granted this can happen in any type f array setup, thin r nt, the 2
thin pl cnstructin typically makes it mre vulnerable fr SAS if versubscriptin ccurs. Overhead, safety, and expansin space n the ld thick LUN definitin has been replaced by a pl that may be quickly becme versubscribed. Care and attentin must be paid t peak lad demand, and adequate cverage prvided if thin pl cnstructin is chsen. Otherwise, yu may wish t fllw the rute f nt using thin-pl prvisining. Sme large SAS shps, even when prvisining new virtualized strage arrays have chsen t create thin pls that were nt ver-subscribed, acrss mst r all the spindles in their array. This wrks well as lng as yur demand lad and perfrmance expected fit within the architecture yu create. Other SAS shps have very heavy IO wrklads, with strict perfrmance requirements. Sme f these custmers have had t bypass sme f the benefits f virtualized, thin-prvisined pls, and autmated management, in rder t guarantee fixed resurce perfrmance. In these instances nly dedicated LUNs were used, utside f thin pls. Depending n yur shp s strage expertise, it will require wrking clsely with yur EMC Strage Engineer fr desired perfrmance. Bth appraches will be cvered belw in Cnsideratins and Best Practices fr Using Thin Pls and FAST VP, and Cnsideratins and Best Practices fr Using Dedicated LUNs belw. Disclaimer - If yu are emplying a VNX system fr SAS, there are the general guidelines and best practices yu shuld cnsider. These guidelines have been frmulated based n experience with a wide variety f SAS custmer sites. They are a gd starting place, and d nt represent a guarantee f perfrmance fr yur wrklad, safety and recvery needs, r shp standards and ecnmics. Use prudent judgment and depart frm these guidelines where they make sense fr yur particular situatin. There is n ne-size-fits-all strage slutin fr SAS shps. Given the extreme flexibility, autmatin, virtualizatin, and pwer f the EMC /VNAX system, what are the best practices fr strage supprt n SAS Applicatins? This next sectin will give generally recmmended practices by SAS and EMC resulting frm multiple field experiences. Cnsideratins and Best Practices fr Using Thin Pls and FAST VP As a general starting pint, a single large thin pl shuld be used fr all devices t be presented as FAST VP thin pl LUNs t the hst server(s). This pl shuld be created frm the SAS tier, and becmes the bind pl in the FAST VP prcess. Separate strage grups r management pls, are created ut f the same pl f thin LUNs described abve. The strage grups are used t apply FAST VP plicies t differing wrklads within the SAS implementatin Utilize SAS and EFD Drives fr yur SASWORK, UTILLOC file systems. Use SAS and EFD drives fr yur Permanent SAS Data file systems that require gd perfrmance. Determine what prtin f yur wrklad requires the mre expensive perfrmance bst f flash. There are t trains f thugh, SAS Permanent Data versus SASWORK. Fr SAS user cmmunities emplying a typical large read at the beginning f jbs, utilizing SASWORK and UTILLOC in the middle f the jb-flw, and writing ut a smaller result set, flash heavily favrs SAS Permanent Data Strage. Aside frm that, SASWORK benefits well since it is heavily utilized. The SAS Utility space fr Threaded PROCEDURES benefits frm flash perfrmance as well. These file systems typically represent clse t a 50/50 read/write rati, and large-blck, sequential flash reads perfrm better than writes. Yu have t determine where yur ecnmic trade-ff pints are versus perfrmance gained n allcating flash devices. Even n a thin pl, yu may wish t dedicate resurces n the LUNs t attain perfrmance gals. In ther wrds set the thin pl acrss enugh spindles as t apprpriately prvisin the entire wrklad underneath it frm all the management pls. In general utilize emlc Flash Drives fr a gd cst-benefit/perfrmance rati fr any file systems residing n the Flash layer. If yur wrklad is extreme, and yu can ft the cst f SLC, cnsider using that instead. EFD drives shuld be targeted fr a maximum capacity lad f 70% f ttal space. This prvides ample space fr quick cell writes when pre-emptive garbage cllectin isn t effective enugh, and helps avid write cliffs resulting frm write-amplificatin. This dwn-frmatting f the file system has prven effective in maintaining effective verall flash write perfrmance under heavy cnditins. Spread EFD drives acrss all available busses. 3
If yu utilize NL-SAS devices at all, d s nly fr prject strage, backups, and wrk that can be dne with lw perfrmance and service levels. If yu plan n using FAST VP fr autmated tier management in terms f prmtin and demtin, the best perfrmance is achieved by the fllwing tier cnfiguratin: Tier 1 EFD Raid-5 (4+P) Tier 2 SAS Raid-5(4+1) IF Used at All - Tier 3 NL-SAS Raid-6(8+2P) Fr best perfrmance, cnsider using nly 2 Tier Pls emlc and SAS 15K (same drives & sizes fr each tier within a single management pl). A benefit f using FAST VP n VNX, even if yu dn t have plicies fr tier prmtin r demtin, is t prvide nging lad balancing f LUNs acrss the available drives based n slice temperature. Be sure t schedule any planned data-n-tier relcatins fr ff-hurs. Review yur FAST VP plicies, and cnsider manually setting them in a fixed mde. Yu may need t cnsider nt using FAST prmtin r demtin if it has an adverse effect n yur IO stream. Many custmers keep the prmtin r demtin activity as a manual activity (nn-autmated) fr tier prmtin. If availability is a high cncern, cnsider limiting management pl sizes t the maximum number f drives recmmended n initial pl creatin tables. D nt expand them abve this. This will help limit expsure t the MTBF rate n a very large drive stripe set. Outside availability, a large number f drives in a pl is highly desired because it can yield higher aggregate bandwidth. Utilize EMC Pwerpath t balance LUNs acrss as many adapters as pssible. When building a file system n a pl, 100 MB/sec thrughput is the recmmended minimum bandwidth, with 100 150 MB/sec/cre as an aggregate fr all SAS related file systems n the array. Never mix Randm and Sequential wrklads n the same pl f devices. D nt use Blck Cmpressin r Blck De-duplicatin fr SAS file systems. Build a minimum f 2 3 thin FAST VP Strage Management Pls. If yu wish t use FAST VP cnsider the fllwing initial pl allcatins: A SASWORK Pl fr SAS wrking space with the fllwing initially recmmended FAST Plicy tier spreads: 20 30 % Tier 1 - EFD Drives 100% - Tier 2 SAS 15K Drives 0% - Tier 3 NL-SAS A SASDATA Pl fr SAS Permanent Data space with the fllwing initially recmmended FAST Plicy tier spreads: 20 30 % Tier 1 EFD Drives 100% - Tier 2 SAS 15K Drives 30-60%** - Tier 3 NL-SAS A UTTILOC Pl fr SAS Threaded Utility space. This wuld segregate the Utility space frm the SASWORK file space. It wuld have the same initially recmmended FAST Plicy tier spreads as SASWORK: 20 30 % Tier 1 EFD Drives 100% - Tier 2 SAS 15K Drives 0% - Tier 3 NL-SAS Drives Nte that in the abve allcatins, NL-SAS is nt used fr SAS Wrking r Threaded Utility Space. It can be used fr SAS DATA in smaller SAS wrklads that d nt have a high thrughput requirement (e.g. >75 4
MB/sec). NL-SAS (SATA) is mre apprpriate fr SAS wrklads exhibiting a randm access perfrmance prfile (e.g. heavy INDEX usage, SAS/OLAP, and randm traversal f data). Cnsideratins and Best Practices fr Using Dedicated LUNs When IO perfrmance f very large systems becmes paramunt, it may be necessary t frsake the benefits f thin pl virtualizatin, and autmated tier management. In this case perfrmance is s impacted by wrklad, that the strage architecture must be streamlined in every way t supprt large blck, sequential, IO. In this case, the fllwing best practices and cnsideratins are ffered: Create standard thick (classic) LUNs acrss a large number f RAID Grups, n fixed tiers f strage. Dedicated LUN systems typically are striped everything. The LUNs are striped acrss all the disks in the array. D nt make these LUNs part f any thin pl. Treat them as thick LUNs in the same way as EMC Clariin Arrays did. Raid levels shuld be RAID5(4+1) fr SAS 15K Drives, RAID5( (4+P)fr EFD Drives. D nt mix drive sizes, types, r rtatinal speeds in the RAID Grups supprting cmmn LUNs. D nt mix randm and sequential wrklads n LUNs. Ensure LUN resurces meet minimum thrughput requirements fr yur wrklad: 100 MB/sec minimum per file system If yu are dedicating LUNs, yu likely need mre thrughput per file system. Aggregate thrughput frm SASWORK and Permanent SAS File systems shuld be >100 MB/sec per cre n yur server. D nt use NL-SAS Drives fr dedicated LUNs Thrughput Testing It is wise t perfrm system thrughput testing with new cnfiguratins. This ensures yu have a finite idea where the systems thrughput bundaries are. A thrughput prgram prvided by SAS Technical Supprt fr UNIX/LINUX systems is here: http://supprt.sas.cm/kb/51/660.html. The same paper fr Micrsft Windws is here: http://supprt.sas.cm/kb/51/659.html Cnclusin Mdern array technlgy has incrprated virtualizatin, enabling thin prvisining and ffering autmated strage tiering. These fferings are intended t cmbat high array csts by driving high utilizatin f resurces, and autmating much f the array Administratin. Unfrtunately the gal f driving high utilizatin is ften diametric t having sufficient thrughput resurces quickly available fr large-blck I/O applicatins such as SAS. When emplying such arrays, the advice in this paper, and the specific vendr papers listed in the Further Reading Sectin belw shuld be carefully cnsidered. In many instances the new technlgies can be utilized effectively, and in thers it must be mitigated with apprpriate architecture changes, and usage. Wrk with yur Strage Vendr t ensure yu are emplying their strage technlgy t its best affect, and prviding high perfrmance t yur SAS applicatins. There is n substitute fr careful planning, and testing, t ensure adequate perfrmance will be prvided. References Deplying EMC VNX Unified Strage Systems Fr Data Warehuse Applicatins, Best Practices fr Adptin and Deplyment, EMC Slutins Grup, December 2011. EMC Technical Ntes EMC Symmetrix V-MAX Best Practices. Technical Nte P/N 300-009-149 REV A02. Nvember 18, 2009. Cpyright 2009 EMC Crpratin All Rights Reserved. EMC VNX Unified Best Practices fr Perfrmance, Applied Best Practices Guide, VNX OE fr Blck 5.33, EMC Enterprise and Mid-range Systems Divisin, 2013. 5
Resurces http://supprt.sas.cm/kb/42/197.html - this link cntains many papers related t perfrmance mnitring, hardware architectures and tuning, and best practices fr SAS I/O Wrklads. Cntact Infrmatin Yur cmments and questins are valued and encuraged. Cntact the authrs at: Tny Brwn SAS Institute Inc. 15455 N. Dallas Parkway Dallas, TX 75001 United States Wrk Phne: +1(214) 977-3916 Fax: +1 (214) 977-3921 E-mail: tny.brwn@sas.cm Margaret Crevar SAS Institute Inc. 100 SAS Campus Dr Cary NC 27513-8617 United States Wrk Phne: +1 (919) 531-7095 Fax: +1 919 677-4444 E-mail: margaret.crevar@sas.cm Steven Bnuchi EMC Inc., US Central Divisin United States Wrk Phne: +1(913) 708-3265 E-mail: steven.bnuchi@emc.cm 6
T cntact yur lcal SAS ffice, please visit: sas.cm/ffices SAS and all ther SAS Institute Inc. prduct r service names are registered trademarks r trademarks f SAS Institute Inc. in the USA and ther cuntries. indicates USA registratin. Other brand and prduct names are trademarks f their respective cmpanies. Cpyright 2014, SAS Institute Inc. All rights reserved.