Slaying the Virtual Memory Monster - Part II

1 of 8 04/19/2012 07:53 PM Slaying the Virtual Memory Monster - Part II Reed Robison 1 Oct 2007 4:46 PM 17 Someday I ll learn to write a simple blog post a couple of paragraphs about something cool and edgy. Not today. There is no way to cover Virtual Memory in brevity and still get anything out of it so... we re going to trudge through this. In my last post, we covered some important VM fundamentals. In this post, we re going to get a little more hands on and actionable. Let s talk about ways to diagnose and deal with VM issues. Because the failure characteristics are somewhat random in nature and there are several ways to tackle them, the first step is to profile the VM space on the device and try to figure out what you are up against. This is a good thing to do for a number of reasons: 1) You need know which processes are the biggest consumers of VM space reserved for loading DLLs 2) You need to know how much VM space is still available inside YOUR process 3) How you attack a VM problem depends on #1 and #2 above Profiling the Virtual Memory Situation When I first started debugging VM issues on Windows Mobile, I was surprised to find that we didn t release any decent low level memory tools in the public WM SDK. There are a couple of OEM tools we provide to device makers, but since they are not included in the public SDK, they won t help us much and I m not going to use or discuss them here. The good news is that the basic APIs you need to understand VM use are publically exposed within the ToolHelp API set. These APIs work across all versions of the WM platforms and for the most part this technique gets the job done. The best way to get started with this approach is via and old support tool we released way back in 2002 which remains available in our downloads -- called DumpMem. DumpMem simply uses the ToolHelp APIs to dump information about all the running processes into a text file and includes some basic heap data. The files are not particularly fun to work with (it s a big messy text file), but I ve seen several versions of home grown tools built around the underlying DumpMem code and parsing tools for the logs. I ve written several myself. With a little practice, you can learn to make sense of these files and gain a lot of insight from them. One concept I didn t cover in the last post was slots. A slot is a basic unit for maintaining virtual memory within the Windows Embedded CE kernel. Without going into great detail, I ll just say that each slot in the user address space is 32 MB and there are 32 slots set aside for the maximum number of running process. Two slots are special. Slot 0 is used for the current running process and Slot 1 is reserved exclusively for in-rom components that have been included as part of the device image. Slot 0 is represented by the area below 0x02000000 and since it s where the active process runs, that s what we re most interested in. For all practical purposes, that s our 32MB virtual address space area we have to make it all work. Virtual Memory in a perfect world Our emulators are a good place to start. They don t have the OEM extras -- code, drivers, startup software, etc., and thus, generally have more free VM than you ll find on commercially available devices. What are we looking for? 1) First we want to know where the lowest DLL is loaded in memory (regardless of process). Since we established that every DLL has to load lower than previous DLLs, we re basically looking to see how far that load position has gotten pushed down. All running processes that load DLLs can contribute to this position. This is often the biggest contributor in VM failure scenarios. When the DLL load position get pushed really low, it s more likely to bump into a process heap (which is growing upward). When that happens, LoadLibrary fails and you won t be able to load any more components in your process. This is also a very important metric to consider for all processes since the same low DLL load position has implications for all apps. 2) Next we want to identify the biggest free area between our heap (growing upward) and the lowest DLL load position in our process. We want to know this because it tells us how much room is left to load DLLs in our process and is often the largest single piece of contiguous memory left for the heap (important if you need to allocate a sizable chunk of memory without going to the large memory area like an image, etc). Step by Step Create a Memory Dump using DumpMem Let s start with DumpMem and a Hello World application. I m just going to let Visual Studio build a simple Hello World application and launch it on the WM6 Professional emulator. Then I ll use DumpMem to dump the process and we ll get messy.

2 of 8 04/19/2012 07:53 PM emulator. Then I ll use DumpMem to dump the process and we ll get messy. I simply select the process I want to dump (HelloWorld_1), choose Create File and dumpmem writes everything into a file called dumpmem.txt in the root directory of the device. You can choose any process in the list, but DumpMem only grabs extra heap information for the specific you select. We re not going to use heap data for this exercise, but sometimes, it s good to see what kind of allocations are in there. *DumpMem uses a fairly memory intensive API called CreateToolhelp32Snapshot to grab heap info about the selected process. If VM is already very tight on the device you are profiling, this API can cause DumpMem to fail. If this happens, you can change the call in the DumpMem source to pass in TH32CS_SNAPNOHEAPS flag. This won t give you a dump of the actual heap data, but it will allow you to get the process info with all the loaded modules which is usually all you need. The DumpMem log is broken into several sections: Platform Information o This is simply the platform, CPU type, and version number Virtual Memory Dump (of the selected process) o This is a map of the 32MB process space for the selected application. The format is address, size, state, type, use. Heap Dump of the selected process o The heap(s) break down every chunk of raw heap memory it contains and shows you the address, size of the chunk, and if it s in use (fixed or free) Processes and their modules o An enumeration of all the running processes, all the components each has loaded, and then for each component the name, load address, size, and module attributes We ll go into these sections in more detail later on, but first let s just focus on our first goal identifying the lowest DLL load position. To do this, skip over all the sections in the log file you just created to the last one Processes and their modules. What you find here is a list of all the running processes and more importantly all the components (DLLs, MUIs, etc.) they have loaded along with their load positions. We re mostly interested in components that are loaded into Slot 0 (below 0x02000000). One quick way to identify these components is by their attribute in the list. If the attribute does not include XIP we want to pay special attention because they will always be in this space and usually one of the lowest DLLs in memory (see bold items below). Another way might be to look at the load position of each component in something like Remote Process Viewer and see if it s using loading below 0x2000000. If the component has an XIP attribute, then it s a ROM based module and usually loaded into a special area in Slot 1 (outside the area we re worried about). Sometimes, when the XIP region gets full, these components will start getting pushed down into Slot 0. When that happens, they affect our VM space the same way as non-xip components. Anything below 0x02000000 that is not labeled XIP is what we re primarily concerned about. If I look at the dumpmem.txt file I just created, that section looks something like the list below (I ve trimmed off each module list after the first 3-4 items). Processes and their Modules Module File Attributes Legend RO - Read Only H - Hidden S - System A - Archive C - Compressed

3 of 8 04/19/2012 07:53 PM C - Compressed NK.EXE base address: C2000000 coredll.dll 03F4E000 96000 RO, H, S, XIP coredll.dll.0409.mui 7FFE0000 15000 RO, S, XIP hd.dll 88079000 4000 RO, H, S, XIP filesys.exe base address: 04000000 rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM vcefsd.dll 01990000 1C000 RO, H, S, XIP ramfmd.dll 01A80000 9000 RO, H, S, XIP device.exe base address: 06000000 rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP serdma.dll 01A10000 7000 RO, S, XIP... services.exe base address: 08000000 lap_pw.dll 01910000 18000 RO, S, C, RAM from ROM rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM obexsrvr.dll 02416000 A000 RO, H, S, XIP gwes.exe base address: 0A000000 kbdmouse.dll 01960000 9000 RO, H, S, XIP touch.dll 019B0000 A000 RO, H, S, XIP deviceemulator_lcd.dll 01B30000 2D000 RO, H, S, XIP shell32.exe base address: 0C000000 wlmhssearchbarcode.dll 01890000 8000 RO, S, C, RAM from ROM wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM wlmuiframework.dll 018E0000 19000 RO, S, C, RAM from ROM wlmtodayscreen.dll 01900000 6000 RO, S, C, RAM from ROM siminit.dll 02090000 6000 RO, S, XIP EmulatorStub.exe base address: 0E000000 coredll.dll 03F4E000 96000 RO, H, S, XIP coredll.dll.0409.mui 7FFE0000 15000 RO, S, XIP poutlook.exe base address: 10000000 ossvcs.dll 0300E000 42000 RO, S, XIP oleaut32.dll 03050000 2F000 RO, H, S, XIP ole32.dll 0307F000 2C000 RO, H, S, XIP connmgr.exe base address: 12000000 rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM ril.dll 02096000 11000 RO, S, XIP cspvoice.dll 02125000 10000 RO, S, XIP ConManClient2.exe base address: 14000000 devicedma.dll 01860000 B000 A

4 of 8 04/19/2012 07:53 PM devicedma.dll 01860000 B000 A edbgtl.dll 01870000 14000 A coredll.dll 03F4E000 96000 RO, H, S, XIP HelloWorld_1.exe base address: 16000000 mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP netcfagl2_0.dll 0218C000 3B000 RO, H, S, XIP mscoree.dll 021C7000 10000 RO, H, S, XIP fexplore.exe base address: 18000000 ossvcs.dll 0300E000 42000 RO, S, XIP oleaut32.dll 03050000 2F000 RO, H, S, XIP ole32.dll 0307F000 2C000 RO, H, S, XIP dumpmem.exe base address: 1A000000 toolhelp.dll 02E5F000 4000 RO, H, S, XIP ossvcs.dll 0300E000 42000 RO, S, XIP oleaut32.dll 03050000 2F000 RO, H, S, XIP Step by Step Build a list of the DLLs loaded below 0x02000000 If you make a list of all the DLLs in every process that have been loaded under 0x02000000 and order them by load position in memory, you get a pretty good idea of what s going on with the low DLL load position and which processes have contributed to pushing it down to its current position. Our current list of all DLLs loaded under 0x02000000 looks like this: Component load position size -- attributes devicedma.dll 01860000 B000 A (lowest DLL in memory) edbgtl.dll 01870000 14000 A wlmhssearchbarcode.dll 01890000 8000 RO, S, C, RAM from ROM wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM wlmuiframework.dll 018E0000 19000 RO, S, C, RAM from ROM wlmtodayscreen.dll 01900000 6000 RO, S, C, RAM from ROM lap_pw.dll 01910000 18000 RO, S, C, RAM from ROM rsaenh.dll 01930000 2B000 RO, H, S, C, RAM from ROM kbdmouse.dll 01960000 9000 RO, H, S, XIP touch.dll 019B0000 A000 RO, H, S, XIP serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP serdma.dll 01A10000 7000 RO, S, XIP s3c2410x_wavedev.dll 01A20000 9000 RO, H, S, XIP pwrbtn2410.dll 01A90000 4000 RO, H, S, XIP pcmcia.dll 01AA0000 A000 RO, H, S, XIP pcc_smdk2410.dll 01AB0000 9000 RO, H, S, XIP pcc_serv.dll 01AC0000 F000 RO, H, S, XIP nullcam.dll 01AD0000 C000 RO, H, S, XIP nleddrvr.dll 01AE0000 4000 RO, H, S, XIP irsir.dll 01B00000 6000 RO, H, S, XIP emulserv.dll 01B10000 5000 RO, H, S, XIP dmatrans.dll 01B20000 5000 RO, H, S, XIP battdrvr.dll 01BF0000 5000 RO, H, S, XIP backlight.dll 01C00000 4000 RO, H, S, XIP asyncmac.dll 01C20000 6000 RO, H, S, XIP scard.dll 01C40000 B000 RO, H, S, XIP mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP Step by Step Analyze the memory data as it applies to your process Low DLL load position This is a pretty small list of DLLs using a simple environment (it s the WM 6.0 SDK refresh Pro emulator). You can see the lowest DLL (devicedma.dll) loaded at 0x01860000 (~24MB). You may also notice that if you add the DLL size to a load position, there may be a small gap before the next DLL loads in memory. In WM5, we had to load components on a 64K boundary. This means even if you loaded two DLLs that were 10K in size, they would eat up 128K of VM. In WM6, we use a 4K boundary which makes more efficient use of memory. From our data, it appears that the next DLL loaded into memory (from any process) is going

5 of 8 04/19/2012 07:53 PM From our data, it appears that the next DLL loaded into memory (from any process) is going to find a home right under 0x01860000. That s where the lowest module is loaded right now and the next one has to load underneath it. This answers our first question about where the low DLL load position is. As you can see, we re already a long way down from 32MB since this is ~24 MB in the VM map. The low DLL load mark has already been reduced by 8MB. What pushed it down that far? You could go back through the list of component under 0x02000000 and find the processes that loaded them. That would tell you which processes are the biggest culprits. If you did this, you would see that shell32.exe is responsible for loading several Windows Mobile Live components, but even so the largest one is only about 240K. The real culprits are all the system components and drivers that have been pushed down into Slot 0 because slot 1 was full. This is very typical of commercial devices. That s what pushed down the low DLL load position by 8MB. This number is highly variable for retail devices and something you typically don t have a lot of control over. It s also why if you have a big application that requires a lot of VM space to load DLLs, you need to be especially careful to profile your target devices. What about our heap space? To get an understanding of available heap space left, I usually look at the first section of the dumpmem.txt file called Virtual Memory Dump (left column below). This is a simple outline of your memory space and how it s being used. If I identify the heap area between where my EXE is loaded and before the first DLLs appear in this VM Map, I can find the largest available free memory area left in my process. Before we look at the virtual memory dump, let s clarify one point. You will notice that all the memory positions seem to be offset from 0x16000000. You may recall that the base address for HelloWorld_1 was 0x16000000 (where it lives when it s not the active process in slot 0). Just subtract 16000000 from all the memory positions if it s easier to think about it in the context of Slot 0. Here we go Virtual Memory Dump of HelloWorld_1.exe 16000000 10000 F NA *The first little block of memory is 10000 in size. This is the reserved position at the bottom of each process and will always be here for any process. Next comes our EXE 16010000 2000 R NA Image 16012000 1000 C RW Image 16013000 1000 R NA Image 16014000 1000 C RO Image 16015000 1000 R NA Image 16016000 1000 C RO Image 16017000 1000 R NA Image *The memory from 16010000-16017000 is labeled as image. This is the bottom of the VM position and where our EXE is loaded. The lowest section labeled Image is always your EXE. Next comes our Heap and Stack memory 16018000 8000 F NA 16020000 1000 C RW Private 16021000 F000 F NA 16030000 D000 R NA Private 1603D000 3000 C RW Private 16040000 5000 C RW Private 16045000 2A000 R NA Private 1606F000 1000 F NA 16070000 5000 C RW Private 16075000 2A000 R NA Private 1609F000 1000 F NA 160A0000 D000 C RW Private 160AD000 22000 R NA Private 160CF000 1000 F NA 160D0000 20000 C RW Private 160F0000 F000 R NA Private 160FF000 1000 F NA 16100000 1000 C RW Private 16101000 2E000 R NA Private 1612F000 1000 F NA 16130000 F000 R NA Private 1613F000 1000 C RW Private 16140000 10000 C RW Private 16150000 F000 R NA Private 1615F000 1000 C RW Private *The area from 1601800-1615F000 is dynamic memory being used by the process heap, stack, etc. Notice there are a few free areas marked F but most of it is Private memory which has been committed or reserved. This is where the majority of your dynamic memory allocations occur and your heap expands upward. 16160000 1B10000 F NA *At the top of your heap area you will usually find a big free chunk of memory. This is largest contiguous free chunk of memory available inside your VM space (Slot 0). In this case, it s 1B10000 or ~27MB. Next (at the top of the VM area working its way down) you will find any

6 of 8 04/19/2012 07:53 PM Next (at the top of the VM area working its way down) you will find any DLLs loaded in our process. The first one shows up at 0x17C70000 17C70000 1000 R NA Image 17C71000 7000 C RO Image 17C78000 1000 R NA Image 17C79000 3000 C RO Image 17C7C000 18000 R NA Image 17C94000 1E000 C RO Image 17CB2000 1000 R NA Image 17CB3000 14000 C RO Image 17CC7000 1000 R NA Image 17CC8000 12000 C RO Image 17CDA000 5000 R NA Image 17CDF000 3000 C RO Image 17CE2000 4000 R NA Image 17CE6000 1000 C RO Image 17CE7000 5000 R NA Image 17CEC000 6000 C RO Image 17CF2000 4000 R NA Image 17CF6000 1000 C RO Image 17CF7000 9000 R NA Image 17D00000 4000 C RO Image 17D04000 4000 R NA Image 17D08000 2000 C RO Image 17D0A000 2000 R NA Image 17D0C000 B000 C RO Image 17D17000 1000 R NA Image 17D18000 2000 C RO Image 17D1A000 1000 R NA Image 17D1B000 1000 C RO Image 17D1C000 1000 R NA Image 17D1D000 1000 C RO Image 17D1E000 3000 R NA Image 17D21000 1000 C RO Image 17D22000 1000 R NA Image 17D23000 2000 C RO Image 17D25000 2000 R NA Image 17D27000 5000 C RW Image 17D2C000 8000 R NA Image 17D34000 EC000 F NA 17E20000 14000 R NA Image 17E34000 2000 C RW Image 17E36000 1A000 R NA Image 17E50000 1000 C RW Image 17E51000 53000 R NA Image 17EA4000 1000 C RW Image 17EA5000 58000 R NA Image 17EFD000 1000 C RW Image 17EFE000 8000 R NA Image 17F06000 5000 C RW Image 17F0B000 10000 R NA Image 17F1B000 1000 C RW Image 17F1C000 1A000 R NA Image 17F36000 1000 C RW Image 17F37000 E000 R NA Image 17F45000 1000 C RW Image 17F46000 1C000 R NA Image 17F62000 1000 C RW Image 17F63000 5000 R NA Image 17F68000 1000 C RW Image 17F69000 3000 R NA Image 17F6C000 2000 C RW Image 17F6E000 32000 R NA Image 17FA0000 40000 F NA 17FE0000 1C000 R NA Image 17FFC000 1000 C RW Image 17FFD000 3000 R NA Image *Memory from 17C70000 up is labeled as image. This is memory used by DLLs remember they load from the top of the VM space and work their way down. All I care about at this point is the lowest position used in my process. If I map address lowest DLL loaded at 17C70000 back into Slot 0 (subtracting the base address), I get 1C70000 which can be easily identified as mscoree2_0.dll in the module section of HelloWorld.exe. From this information, we achieve our second goal. We know that there is a large free chunk of VM space (0x01B10000 bytes or- ~27MB), between our EXE and the lowest DLL in our process. The lowest DLL loaded into our process is mscoree2_0.dll at 0x01C70000. Making sense of it all Let s now regroup on our objectives and put it all in context. Our goals were: 1) Understanding the lowest DLL load position in all processes:

7 of 8 04/19/2012 07:53 PM 1) Understanding the lowest DLL load position in all processes: >>> devicedma.dll at position 0x01860000 (~24MB) 2) Identifying the largest chunk of available VM and lowest DLL in OUR process >>> we found 0x1b10000 bytes FREE right above our heap and we located mscoree2_0.dll at 1C70000 giving us about 27MB of free space to work with. This was simply example and done in a very clean environment so we re still in a pretty healthy situation. Goal #2 tells us we have plenty of space to grow (heap) and plenty of room to load more DLLs. Goal #1: The lowest DLL load position in OUR process is 0x1C70000 so you might expect the next DLL to load right under it. Considering #1 though, the next DLL would actually load significantly lower just below 0x01860000. Remember that when our process loads another component, it has to find a position under every other component that has already been loaded by any process. This, in effect, restricts our heap range a bit. Where we had a giant contiguous chunk of almost 27MB worth of VM space loading a component now would split up that area by positioning itself around 23-24MB in the map. Our heap can still grow upward and claim most of this available space, but it s no longer single contiguous chunk. This becomes important when you have an app that might need to make a big allocation (e.g. manipulating an image, media file, etc.). I say "most" of this space because if you are not running with full privileges (e.g. - untrusted on a 2-tier phone), then you are limited to available VM space under the low DLL load position -- even if there is plenty of free space up there! This is a really simple exercise, but you can imagine how complex this can get with a lot of applications running especially if they load a bunch of other components. While there may only be 32 processes running, don t forget that services.exe and driver.exe can become quite full with dlls that load on boot. This can contribute to pushing down that low DLL load position. Once the DLL position gets pushed WAY down, the available area left to grow your heaps can get quite constrained. If you start running into problems with VM issues or need to get an idea of what is going on, you can use the above techniques to get an idea of what you have to work with (and why it might be failing). Where s the smoking gun? We already noted that when applications experience VM related failures, there s not always a smoking gun. The failures can be appear quite random. Let s talk about some of the most common failures and contributors. Components that fail to load - The most common failure occurs when an application attempts to load a dependant component either on startup or through a call to LoadLibrary. There has to be enough space between your heap and the low DLL load position to succeed, so when the DLL load position gets pushed too far down, LoadLibrary will fail. At this point, an application can fail to start or a manual call to LoadLibrary will return an error. The actual affect on an application can vary depending on how it handles this failure. Memory allocations that fail - When a heap can no longer expand, memory becomes too fragmented, or there is insufficient space for a large contiguous allocation-- memory allocations can begin fail. Few developers write to code to adequately check for memory failures so this very often results in unhandled exceptions for the application (e.g. crash). Dynamic failures - It s pretty easy to take responsibility for problems in your own application failure to start, exceptions, etc., but it s even more complex that. VM problems commonly extend into other processes because of the DLL loading and memory rules. How does this happen? Because every process can contribute to the low DLL load position, this means that the order in which they start can make a big difference. Suppose the low DLL load position has been pushed down to the point where only 10MB of space is left to load more DLLs. What happens then if you have three application A, B, and C that all load 5MB worth of unique components. If you load A and B, then C fails. If you load B and C, then A would fail. Part of the challenge today is that VM health is very much a little ecosystem with many contributors. The OEM shares responsible for delivering a device that doesn t eat up all the VM space before you install the first app. The developer shares responsible for not taking more than their share of resources. The end user shares responsible for all the apps they install. Microsoft shares responsibility for mitigating this problem at a platform level so nobody has think about it. In the not so distant future, we will likely adopt changes in the platform (like we did with CE6) that expand the VM range so that this isn t an issue anymore. Right now, we have to work with what we have. Tactics developers use to address VM problems Profile the VM space first. You cannot solve this problem unless you know what is going on with your memory. Static linking is your friend. When you statically link code, it becomes part of thee EXE which means it loads in the bottom of the VM space (the best possible area). You don t have to worry about DLL load rules and you don t impact the VM of other processes. You may end up using a little more RAM if multiple apps use the same code, but the trade-off is probably worth the headaches you can avoid. Plus, you avoid extra signing costs by eliminating DLLs. Start order matters. While more of a band-aid than a solution, controlling the load order of processes on a device can help you work around VM issues. For this to work, you really need a good understanding of the memory profile of the device as well as which processes are contributing to the problem. The first app gets the most DLL space, the second gets the slightly less, etc. By controlling which apps start first and/or disabling services or drivers you don t need you can sometimes avoid VM problems. Merge small DLLs. Windows Mobile 5.0 had to load components on a 64K boundary which meant that every a small DLL used a minimum of 64K. If you have a bunch of small DLLs, it may make sense to merge them to save space. Windows Mobile 6 loads

8 of 8 04/19/2012 07:53 PM small DLLs, it may make sense to merge them to save space. on a 4K boundary which makes much better use of space. Move resources to resource-only DLLs (dll s with no entry point). Resource only DLLs are loaded up in slot 63 and don t use up your valuable VM space in slot 0. For large memory allocations, consider using VirtualAlloc and the Large Memory Area (aka High Memory Area). Allocations of 2MB or greater use space well outside of slot 0 and leave your process with more room to work. Trim your code. Make it smaller and lighter and save on VM space In Summary I ve rambled long enough about the Virtual Memory Monster. Somewhere this is a blog post turned into whitepaper. Hopefully those of you who read this now have a better understanding of the creature and some skills you can use to battle it. Like most nasty, bad-guy type, it s more misunderstood than evil. Hopefully you never run into it, but if you do you have some tools under your belt. If you guys want to dig into this in more detail, let me know. I could blog through a real world problem or two to help you get the hang of it. Some of you will undoubtedly want to know more about how NETCF optimizes memory inside of this limited area. You will notice that if you load a NETCF assembly, it doesn t use up DLL space like native DLLs. NETCF goes to great lengths to make good use of memory transparently. Since there are already some great posts on this topic, I ll simply reference them here: http://blogs.msdn.com/stevenpr/archive/2005/12/12/502908.aspx http://blogs.msdn.com/mikezintel/archive/2004/12/08/278153.aspx Next up. Power Management or maybe IE Mobile? Both of these areas seem to be generating a lot of interesting support activity right now. Cheers, Reed