Linux on z/vm Memory Management

Transcription

1 Linux on z/vm Memory Management Rob van der Heij velocitysoftware.com IBM System z Technical Conference Brussels, 2009 Session LX45 Velocity Software, Inc Copyright 2009 Velocity Software, Inc. All Rights Reserved. Other products and company names mentioned herein may be trademarks of their respective owners.

2 Introduction Virtualization is said to be easy and automatic Our Linux application people got out-of-memory errors and claim they have not enough memory. But z/vm was not even paging a little bit. Can you explain that again? 2

3 Linux on z/vm experts know there are challenges It is not always easy or automatic Not intuitive - Some knobs seem to go reverse Expert guidance must be understood within context This is confusing for new installations Need a consistent overview of where we are Context of this presentation Building on real customer data and experience Running Linux on z/vm for almost 10 years Applies to Linux on z/vm less interested in LPAR Little interest in benchmarks or artificial lab workloads Introduction 3

4 Agenda CMS versus Linux z/vm Paging versus Linux Swap Two Layers of Memory Management Linux Page Cache Virtual Server Sizing Memory Tuning Options Enterprise Applications Performance data shown in the presentation was collected and processed with ESALPS. 4

5 Why can t I use my Linux Tools? Linux data is incomplete and incorrect Virtualization changes the rules of the game CPU Usage perceived by Linux can be very wrong Assumptions about used and available do not hold anymore z/vm performance impacts Linux behavior Need to combine Linux and z/vm performance data z/vm does not clone system administrators You may not have time to look when it happens Complex interactions make it hard to reproduce Multi-tier application involves multiple virtual servers Centralized data collection is easier to manage 5

6 CMS versus Linux Traditional z/vm installations are surprised by Linux CMS Applications take the resources they need CMS was designed to run in shared resource environment No extra resource usage just because it is there Virtual machine size does not increase memory requirement Linux on z/vm is different Designed to run in a dedicated resource environment Use all resources you can get would be wasted otherwise Same workload in a larger virtual machine uses more memory More expensive to run - not always better or faster In many cases performance will be worse not intuitive 6

7 z/vm Paging Technique to implement virtual memory Memory over commit Limited amount of real memory to hold working set Sum of virtual machine memory is more than real memory Disk space to make up for the difference Not everyone will want all their resources at the same time Over commit is good: only way to enforce sharing Hardware support makes paging transparent Virtual machine does not notice functional difference Paging does cause latency and slows things down z/vm Paging Challenges Which pages can be paged out to create available memory Which additional pages should also be paged in 7

8 General z/vm Paging Strategy Take memory resources away from idle users User resources reviewed during demand scans New transaction may have different requirements Users keep resources until transaction completes No demand for CPU resources for some time Virtual machine drops from queue z/vm Paging Less effective with Linux virtual machines Many Linux servers are never idle remain in queue Easy demand scans don t produce enough pages Most free pages produced in emergency scan Resources also taken from active virtual machines Expanded storage helps to reduce the damage LX44 Wed 15:15 8

9 z/vm Paging and Linux Swap Linux terminology swap is misleading No modern Operating System uses swapping anymore Both Linux and z/vm using demand paging techniques z/vm design is to accommodate some paging Many systems run fine under moderate paging load Some systems handle fairly high paging rates Expanded storage introduces paging hierarchy Virtual machine sizing prevents excessive usage Linux on dedicated hardware is not designed to swap Disk I/O for swap in Linux impacts performance Linux systems are typically not sized to swap to disk Swap serves as a rescue for process out-of-control 9

10 Two Layers of Memory Management Linux on z/vm: two layers of memory management Both need to make same kind of decisions Which pages to retain and which to select for page out Which additional pages to page in They do so mostly independent of each other Two managers is not better than one manager Linux real memory is in fact virtual memory on z/vm Both use Least Recently Used (LRU) strategy Local optimizations are not productive on global level Two layers of LRU interfere worst case scenario 10

11 Two Layers of Memory Management Two layers of management cause inefficiency z/vm retains data that Linux does not need now Data from processes that are idle Memory that is unused after program terminates Collaborative Memory Management CMM1 Linux uses memory unaware of real cost Use of excess memory to cache data and avoid I/O Competes with busy servers that need the memory Size the virtual machine to avoid excessive cache 11

12 Two Layers of Memory Management z/vm View of Virtual Machine Resources Grows from 133 MB to 378 MB Some virtual machine memory was paged out Screen: ESAUSPG Velocity Software-Test VSIVM4 ESAMON /16 1 of 2 User Storage Analysis CLASS * USER ROBLX1 <---Storage occupancy in pages---> Pages <Address UserID <---Main Storage---> <--Paging---> Moved <Pages R Time /Class Total >2GB <2GB Xstor DASD <2GB VirtDisk :17:00 ROBLX :16:00 ROBLX :15:00 ROBLX :14:00 ROBLX :13:00 ROBLX :12:00 ROBLX :11:00 ROBLX :10:00 ROBLX

13 Two Layers of Memory Management Linux View of Memory Resources Grows from 75 MB to 80 MB not like VM metrics Overall figure is factor 2 off Nothing reveals the increase with 250 MB Screen: ESAUCD2 Velocity Software-Test VSIVM4 ESAMON /16 1 of 2 LINUX UCD Memory Analysis Report NODE BROBLX1 < Storage sizes in MegaBytes -- <--Real Storage--> Over <-----SWAP Storage----> Time Node Total Avail Used head Total Avail Used MIN :17:00 broblx :16:00 broblx :15:00 broblx :14:00 broblx :13:00 broblx :12:00 broblx :11:00 broblx :10:00 broblx

14 Two Layers of Memory Management Two memory managers do not exchange information Results in less than optimal resource usage Used (MB) Res (MB) Memory Usage Linux Screen: ESALNXP Velocity Software-Test VSIVM4 ESAMON /16 03:10 1 of 4 LINUX VSI Process Statistics Report NODE BROBLX1 LIMIT <-Process Ident-> <------CPU Percents-----> Time Node Name ID PPID GRP Total sys user syst usrt :19:00 broblx1 vsi-agen *Totals* :18:00 broblx1 vsi-agen *Totals* :17:00 broblx1 vsi-agen *Totals* :16:00 broblx1 bash vsi-agen *Totals* :15:00 broblx1 vsi-agen *Totals* :14:00 broblx1 vsi-agen *Totals* :13:00 broblx1 bash sshd vsi-agen *Totals* Usage (MB) :11 03:13 03:15 03:17 The bash shell ran a program Already ended in next sample Linux freed memory z/vm retains the memory 14

15 Linux Page Cache Basic concept of Linux memory management Not some allocated area you could remove or resize Memory pages with corresponding disk location Program code loaded from disk (demand paging) Shared libraries loaded from disk (demand paging) Pages swapped out but not yet re-used (swap cache) Excessive memory will be used as page cache But not all in page cache is wasted resources Some amount of page cache must be kept available 15

16 Linux Page Cache Linux-2.4 approach was very simplistic Swap-out would require an I/O operation Keeping data in cache might save an I/O operation Strategy: drop data from cache to avoid swap-out Effect: unreferenced old data occupied memory Linux-2.6 introduced swappiness parameter Provide a way to swap out unreferenced old data Keep some page cache even if that causes swap-out Specified as percentage of total Linux memory Must be adjusted when virtual machine size is changed 16

17 Virtual Server Sizing Dedicated server footprint Large enough to handle peak requirements Standard server size Swap defined for Linux over commit No swapping Excess memory used as page cache Virtual server footprint* Sized just large enough for peak Sizing based on application requirements Swap defined for Linux over commit No need for swapping Little penalty for less page cache Easy win - may be good enough Goal is to avoid z/vm paging Excess memory dedicated server Linux memory requirements virtual server Excess memory Linux memory requirements footprint* is total requirement, not just virtual machine size 17

18 Virtual Server Sizing Squeeze the virtual machine Smaller than peak requirement Will be swapping during peaks Acceptable when using VDISK Add as much VDISK as you reduced the virtual machine size Replacement rather than extra Expect both to remain resident Swapping reduces requirements Linux will cache less to avoid swap Be aware of swappiness Total peak requirement goes down Not easy to predict must measure Excess VDISK can be taken out Need to do so if you want to reduce the footprint of the server Excess swap disk behaves LRU VDISK swap VDISK swap footprint virtual machine Excess memory Linux memory requirements footprint virtual machine Excess memory Linux memory requirements 18

19 Virtual Server Sizing Removing excessive VDISK Reduces virtual server footprint Allows for more servers without z/vm paging Still need unused swap space for Linux memory over commit When monitoring and alerting: use VDISK since it is almost for free Otherwise use real disk (slows down the server so users will call you) ESALPS: Review Swap Used over time in ESAUCD2 report footprint virtual machine VDISK swap Excess memory Linux memory requirements footprint savings virtual machine VDISK swap Excess memory Linux memory requirements 19

20 Virtual Server Sizing Important to remove excessive VDISK Linux prefers to use fresh blocks causes fragmentation Over time a large portion of VDISK will be previously used Linux does not care about the contents, no further references z/vm still preserves the data and pages it out to disk Does not slow you down yet, but uses paging capacity Eventually Linux will re-use the old VDISK blocks Forces z/vm to page in VDISK slows down Linux VDISK swapping Reduces available z/vm paging capacity Experiments suggest latest kernel may not need this anymore Watch this space ESALPS: Compare Swap Used in ESAUCD2 with resident and paged in ESAVDSK 20

21 Virtual Server Sizing Measuring VDISK fragmentation Linux uses 11.3 MB (2900 pages) z/vm retains 125 MB (32,000 pages) ESAUCD2 < Storage sizes in MegaBytes -- <--Real Storage--> Over <-----SWAP Storage----> Time Node Total Avail Used head Total Avail Used MIN :16:00 broblx Ready; T=0.01/ :16:19 ESAVDSK <--Size---> <--pages--> DASD X- AddSpc VDSK Resi- Lock- Page Store Time Owner Space Name Pages Blks dent ed Slots Blks :16:00 ROBLX1 VDISK$ROBLX1$$$0203$ K

22 Virtual Server Sizing Adding more servers eventually requires z/vm paging Good thing it saves memory resources Footprint varies over time Virtual machines take turns using real memory - sharing z/vm page-out when server goes idle CP must be able to see which server is idle z/vm page-in at transaction start Causes some latency for transactions Page-in Page-out VDISK swap footprint Linux memory requirements Excess memory virtual machine 22

23 Virtual Server Sizing Active server requires peak footprint to be resident Amount of memory needed to run workload without paging Virtual machine as well as swap VDISK Only visible when z/vm has some memory pressure Idle server requires idle footprint to be resident Memory required to run background tasks Virtual machine memory still referenced by Linux No swapping, VDISK can be paged out Multiple VDISKs to introduce hierarchy and locality of reference Not every workload shows usage patterns that require this Recent kernels use different strategy to allocate blocks Transition between peak and idle involves z/vm paging Balanced system Peak footprint requires taking resources away from idle servers Total of current peak and idle footprints must fit at all times 23

24 Virtual Server Sizing Transaction latency z/vm paging capacity Number of paging devices Efficiency of block paging Minus capacity used for other paging # of pages t slow # of pages t fast Virtual machine sizing Amount of pages to be paged in Difference between idle and peak size Large amounts may hurt even more Replenish the available list page-out needed Increased demand scans # of pages t # of pages t Total of active virtual machines must fit slow fast 24

25 Virtual Server Sizing Conclusion Total of peak and idle footprints must fit real memory Will change with new workload - Measure and adjust Server peak requirement determined by application Virtual machine memory Active swap in VDISK Server VDISK ratio is main control Active vdisk divided by total virtual machine t t VDISK ratio Idle footprint Latency Peak overhead Production Low Large Small Small Development High Small Large Large prod test 25

26 Virtual Server Sizing Suggested approach only works with VDISK swap Makes Linux swap during peak utilization Real I/O to disk would be too slow to do this Performance penalty of VDISK swap is minimal Causes some extra CPU usage during peak utilization Should not be a problem for low-utilized servers Confuses people with discrete server background It s not really swapping, it s just memory This is a Linux on z/vm thing only Do not confuse it with the Classical Linux Swap 26

27 Virtual Server Sizing Classical Linux Swap Disk Normally not used not even during peak utilization Primarily to encourage Linux to over commit memory May be used during extreme workload or things out of control Use VDISK for this too when you have alerts set up When not in use it is almost free When it does get used, review your workload sizing Set up an alert to detect when this gets used Refurbish the VDISK when problem has been fixed Using real disk will slow down Linux seriously when being used Users and managers will alert you when performance is bad 27

28 Cooperative Memory Management First attempt to couple memory managers CMM1 Builds on technique used by CMS Diagnose Instruction to release page Mapping to real page frame is removed page is not there Backing page slot in Expanded Storage or on DASD dropped Support as ballooning technique in Linux Balloon acquires real pages from Linux to give back to z/vm Effective as temporary reduction of Linux memory size Blocked for further use in Linux until balloon is deflated again Requires proper steering to set size of balloon 28

29 Cooperative Memory Management modprobe cmm /proc/sys/vm/cmm_pages size of the balloon lxrob1:/proc/sys/vm # free -m total used free shared buffers cached Mem: /+ buffers/cache: Swap: MB from free lxrob1:/proc/sys/vm # echo > cmm_pages lxrob1:/proc/sys/vm # free -m total used free shared buffers cached Mem: /+ buffers/cache: Swap: lxrob1:/proc/sys/vm # echo > cmm_pages lxrob1:/proc/sys/vm # free -m 200 MB from cached 200 MB from free total used free shared buffers cached Mem: /+ buffers/cache: Swap: lxrob1:/proc/sys/vm # cat cmm_pages

30 Cooperative Memory Management Benefits are as good as quality of steering IBM implemented controls in VMRM Published results do not show much benefit Flexible sizing of the balloon is realistic option today Preparation of virtual machine resizing (be reasonable) Scheduled resource reduction during day shift Many factors affect dynamic tuning of CMM1 Several of those involve Crystal Ball Technology Cost for inflating the balloon Do not ask Linux to do the impossible Do not make Linux give up cache without a reason Both Linux and z/vm performance data required 30

31 Setting of swappiness Linux memory management parameter /proc/sys/vm/swappiness Value ranges from 0 to 100 default at 60 Allow for swap-out to retain some page cache Popular in Linux on desktop usage Strong opinions about both extremes Low swappiness Don t swap out my application just to cache data High swappiness Get rid of rarely referenced anonymous data Tuning of the parameter is not obvious Objective is to retain some amount of page cache The swappiness control is rather vague Default of 60 is probably high for large servers 31

32 Setting of swappiness Kernel Compile Cache friendly scenario Low swappiness: Double amount of swapping Increased non-swap disk I/O Lazy write of temporary files Swap rate (pg/s) Kernel build in 100M swappiness 60 so si Sizing virtual machine is the best way to save resources 0 01:17 01:21 01:25 01:29 01:33 Kernel build in 100M swappiness 10 Real Memory 100MB 100MB 1GB so si swappiness Average Resident Runtime M MB MB 12 Swap rate (pg/s) :47 01:51 01:55 01:59 32

33 Use of drop_caches Kernel command introduced with Linux-2.6 Kernel action rather than a tuning parameter Instructs the kernel to drop all cached data and inodes Some suggest to use sync to clean dirty pages Kernel will load back all popular data after this Works only for Linux memory management z/vm is not aware that pages are now unused Likely paged-out since Linux does not reference them further CMM-1 could inform z/vm memory management But why not use just CMM-1 to free pages and return them Useful to establish some base level requirement Supposed to get you out of swap thrashing situation Not performance tuning but just a debugging aid 33

34 Using cpuplugd for memory cpuplugd is part of s390-tools package plug: is Linux speak for enabling something ( hot plug ) cpuplugd: background process for enabling virtual CPUs Can be used to control CMM-1 as well memunplug: to unplug memory, inflate the balloon memplug: to plug in some memory, deflate balloon Memory control disarmed in sample configuration # Per default this function is disabled, because this rule has to be # adjusted by the admin for each production system, depending on the # environment Objectives and target values are static settings UPDATE="60" CMM_MIN="0" CMM_MAX="8192" CMM_INC="256 34

35 Using cpuplugd for memory Steering is done with only Linux metrics # Memplug and memunplug can contain the following keywords: # - apcr: the amount of page cache reads # - freemem: the amount of free memory (in megabyte) # - swaprate: the number of swapin and swapout operations Steering specified as rules MEMPLUG = "swaprate > freemem+10 & freemem+10 < apcr" MEMUNPLUG = "swaprate > freemem Since apcr includes swap I/O, first rule can be simplified to MEMPLUG = swaprate > freemem+10 Balanced system has 2-10 MB free -> add memory when swapping 5 pg/s or more Suggested rule for unplug is incorrect Removing memory at swap rate of more than 10 MB/s is bad idea! More logical approach to remove memory when not swapping at all MEMUNPLUG = "swaprate = 0 35

36 Using cpuplugd for memory Linux Memory Tuning with cpuplugd Controls are very limited and simplistic Either too aggressive or too soft, high latency This has never been tried for real Static target values to represent z/vm resources Lack of feedback causes waste of resources When developer can t get sample right, should I try? Real Memory swappiness Average Resident Runtime 100MB M MB MB 13 1GB 0 360MB 12 25MB MB 15 After making the rules work Pivot around swapping 10 KB/s CMM_MAX to just avoid killing it Increment of 1 MB Update every 10 seconds Results depend a lot on the workload Probably not the best possible setting 36

37 Collaborative Memory Management Improvement of ballooning technique in CMM-1 New ESSA instruction in z9 and later hardware Collaborative Memory Management Assist (CMMA) SLES10 Support and fixes in SP2 Kernel option cmma=on (your defaults may vary) Support introduced in z/vm 5.3 MEMASSIST The Good News Addresses all concerns with CMM-1 Very little overhead because of hardware support Fully automatic tuning and self optimizing No external controls needed (no VMRM interaction) 37

38 Collaborative Memory Management Hardware support for virtual memory page state Linux state (simplified) Unused Linux does not care for contents anymore Volatile Contents is also on disk, may be taken away Stable Page is in use and must be retained by z/vm ESSA instruction to communicate state changes z/vm state Resident Contents is present in main memory Preserved Contents is paged out Zero Page full of binary zero Special program check when page is gone 38

39 Collaborative Memory Management No Blinkenlights implemented You can t tell whether it is working at all Response of QUERY MEMASSIST is not accurate Last minute option in Linux to disable CMMA CMMA is active only when both Linux and z/vm enable it Linux fix should go into the service stream eventually lxrob1:~ # modprobe vmcp lxrob1:~ # cat /proc/kallsyms grep cmma_flag dc000 B cmma_flag lxrob1:~ # vmcp d r6dc000 R006DC means CMMA active Default setting is probably surprising on older hardware 2. At z/vm IPL, the initial setting for MEMASSIST FOR ALL is ON, even if the assist is not installed on the machine. This allows a virtual machine to SET MEMASSIST ON, causing the assist to be simulated by z/vm for guest testing purposes. 39

40 Collaborative Memory Management No instrumentation available Linux does not keep statistics about state changes z/vm monitor does not reveal any counters about it HPMA might make existing metrics incorrect Documentation is limited and fragmented ESSA instruction not in the Principles of Operation z/vm implementation is Object Code Only Linux source code is just part of the story Future of the CMMA code in Linux is uncertain Affects algorithms in the common kernel source Probably too risky for Open Source community SLES11 ships only part of CMMA anymore Published performance benefits are not convincing Lab workload can be constructed to demonstrate function Without instrumentation you can t tell what it does for you 40

41 Collaborative Memory Management z/vm 5.3 does exploit Linux unused pages Without instrumentation - like searching for a black hole Large program in Linux terminated lots of pages available Workload in another server required more resident pages Screen: ESAUSPG Velocity Software-Test VSIVM4 1 of 2 User Storage Analysis <---Storage occupancy in pages---> UserID <---Main Storage---> <--Paging---> Time /Class Total >2GB <2GB Xstor DASD :07:00 ROBLX :06:00 ROBLX :05:00 ROBLX :04:00 ROBLX :03:00 ROBLX :02:00 ROBLX :01:00 ROBLX :00:00 ROBLX K pgs removed almost nothing paged Works! 41

42 Collaborative Memory Management z/vm 5.3 does not exploit Linux volatile memory Linux server with 1 GB of non-dirty page cache Memory pressure in z/vm causes page-out Screen: ESAUSPG Velocity Software-Test VSIVM4 1 of 2 User Storage Analysis <---Storage occupancy in pages---> UserID <---Main Storage---> <--Paging---> Time /Class Total >2GB <2GB Xstor DASD :01:00 ROBLX :00:00 ROBLX :59:00 ROBLX :58:00 ROBLX :57:00 ROBLX :56:00 ROBLX :55:00 ROBLX :54:00 ROBLX :53:00 ROBLX z/vm Paging Linux Volatile Memory 07:51 07:55 07:59 08:03 120K pgs removed All paged out 42

43 Collaborative Memory Management No full exploitation by z/vm memory management Publications phrase it as z/vm could use Existing Demand Scan strategy has not changed Pages are selected for removal just like before Virtual machine queue drop remains very important CP does skip the actual page-out when not necessary May result in reduced paging requirements Could avoid poor response due to paging very large virtual servers Probably a good thing for transition No fair treatment when only some servers enable CMMA Remaining CMMA-lite function could provide value Hard to tell without instrumentation 43

44 Enterprise Applications You think two memory managers is bad? Enterprise Applications add third layer of management Allocate a large (user configured) chunk of virtual memory Implement some kind of management strategy DB2 UDB uses buffers to cache data and index Oracle allocates SGA and PGA SAP allocates various memory pools for cache JVM allocates heap and uses garbage collection Some Java applications use one of the storage pool classes Implementing another layer of memory management Memory appears in-use for Linux memory management CMM will not be able to free up resources 44

45 Enterprise Applications Make one or more managers do nothing Configure buffers small enough that it does not swap Virtual machine sizing must match application configuration Accept z/vm paging for low-utilized servers Unfortunately most of these do not drop from queue Know your application to get this right Verify configuration with performance measurements Shared memory (Oracle SGA) resides in page cache Bug in Oracle makes it ignore the PGA target size JVM Garbage Collect actually touches the pages Very unpleasant when paged out by z/vm Do not install things applications that you do not need LX44 Wed 15:15 45

46 Linux on z/vm Memory Management is hard work Memory can be shared just like CPU can be shared Over commit of memory is what drives sharing Necessary to host low utilized servers in a cost effective way Virtual server sizing is the main tuning knob Virtual machine size and VDISK for Linux swap Ratio helps to prioritize the workload Work with application owners to get it right CMM-1 can provide some flexibility in adjustment Measure and monitor both z/vm and Linux metrics Can t make sense of one without the other Validate virtual server sizing Understand when workload grows beyond plan Summary 46

47 Visualization Techniques 47

48 Linux on z/vm Memory Management Big Thank You to our customers who let me work on their performance problems If you have performance problems, just drop me a note or catch me somewhere Rob van der Heij velocitysoftware.com IBM System z Technical Conference Brussels, 2009 Session LX45 Velocity Software, Inc Copyright 2009 Velocity Software, Inc. All Rights Reserved. Other products and company names mentioned herein may be trademarks of their respective owners.