Slaying the Virtual Memory Monster - Part II

Transcription

1 1 of 8 04/19/ :53 PM Slaying the Virtual Memory Monster - Part II Reed Robison 1 Oct :46 PM 17 Someday I ll learn to write a simple blog post a couple of paragraphs about something cool and edgy. Not today. There is no way to cover Virtual Memory in brevity and still get anything out of it so... we re going to trudge through this. In my last post, we covered some important VM fundamentals. In this post, we re going to get a little more hands on and actionable. Let s talk about ways to diagnose and deal with VM issues. Because the failure characteristics are somewhat random in nature and there are several ways to tackle them, the first step is to profile the VM space on the device and try to figure out what you are up against. This is a good thing to do for a number of reasons: 1) You need know which processes are the biggest consumers of VM space reserved for loading DLLs 2) You need to know how much VM space is still available inside YOUR process 3) How you attack a VM problem depends on #1 and #2 above Profiling the Virtual Memory Situation When I first started debugging VM issues on Windows Mobile, I was surprised to find that we didn t release any decent low level memory tools in the public WM SDK. There are a couple of OEM tools we provide to device makers, but since they are not included in the public SDK, they won t help us much and I m not going to use or discuss them here. The good news is that the basic APIs you need to understand VM use are publically exposed within the ToolHelp API set. These APIs work across all versions of the WM platforms and for the most part this technique gets the job done. The best way to get started with this approach is via and old support tool we released way back in 2002 which remains available in our downloads -- called DumpMem. DumpMem simply uses the ToolHelp APIs to dump information about all the running processes into a text file and includes some basic heap data. The files are not particularly fun to work with (it s a big messy text file), but I ve seen several versions of home grown tools built around the underlying DumpMem code and parsing tools for the logs. I ve written several myself. With a little practice, you can learn to make sense of these files and gain a lot of insight from them. One concept I didn t cover in the last post was slots. A slot is a basic unit for maintaining virtual memory within the Windows Embedded CE kernel. Without going into great detail, I ll just say that each slot in the user address space is 32 MB and there are 32 slots set aside for the maximum number of running process. Two slots are special. Slot 0 is used for the current running process and Slot 1 is reserved exclusively for in-rom components that have been included as part of the device image. Slot 0 is represented by the area below 0x and since it s where the active process runs, that s what we re most interested in. For all practical purposes, that s our 32MB virtual address space area we have to make it all work. Virtual Memory in a perfect world Our emulators are a good place to start. They don t have the OEM extras -- code, drivers, startup software, etc., and thus, generally have more free VM than you ll find on commercially available devices. What are we looking for? 1) First we want to know where the lowest DLL is loaded in memory (regardless of process). Since we established that every DLL has to load lower than previous DLLs, we re basically looking to see how far that load position has gotten pushed down. All running processes that load DLLs can contribute to this position. This is often the biggest contributor in VM failure scenarios. When the DLL load position get pushed really low, it s more likely to bump into a process heap (which is growing upward). When that happens, LoadLibrary fails and you won t be able to load any more components in your process. This is also a very important metric to consider for all processes since the same low DLL load position has implications for all apps. 2) Next we want to identify the biggest free area between our heap (growing upward) and the lowest DLL load position in our process. We want to know this because it tells us how much room is left to load DLLs in our process and is often the largest single piece of contiguous memory left for the heap (important if you need to allocate a sizable chunk of memory without going to the large memory area like an image, etc). Step by Step Create a Memory Dump using DumpMem Let s start with DumpMem and a Hello World application. I m just going to let Visual Studio build a simple Hello World application and launch it on the WM6 Professional emulator. Then I ll use DumpMem to dump the process and we ll get messy.

2 2 of 8 04/19/ :53 PM emulator. Then I ll use DumpMem to dump the process and we ll get messy. I simply select the process I want to dump (HelloWorld_1), choose Create File and dumpmem writes everything into a file called dumpmem.txt in the root directory of the device. You can choose any process in the list, but DumpMem only grabs extra heap information for the specific you select. We re not going to use heap data for this exercise, but sometimes, it s good to see what kind of allocations are in there. *DumpMem uses a fairly memory intensive API called CreateToolhelp32Snapshot to grab heap info about the selected process. If VM is already very tight on the device you are profiling, this API can cause DumpMem to fail. If this happens, you can change the call in the DumpMem source to pass in TH32CS_SNAPNOHEAPS flag. This won t give you a dump of the actual heap data, but it will allow you to get the process info with all the loaded modules which is usually all you need. The DumpMem log is broken into several sections: Platform Information o This is simply the platform, CPU type, and version number Virtual Memory Dump (of the selected process) o This is a map of the 32MB process space for the selected application. The format is address, size, state, type, use. Heap Dump of the selected process o The heap(s) break down every chunk of raw heap memory it contains and shows you the address, size of the chunk, and if it s in use (fixed or free) Processes and their modules o An enumeration of all the running processes, all the components each has loaded, and then for each component the name, load address, size, and module attributes We ll go into these sections in more detail later on, but first let s just focus on our first goal identifying the lowest DLL load position. To do this, skip over all the sections in the log file you just created to the last one Processes and their modules. What you find here is a list of all the running processes and more importantly all the components (DLLs, MUIs, etc.) they have loaded along with their load positions. We re mostly interested in components that are loaded into Slot 0 (below 0x ). One quick way to identify these components is by their attribute in the list. If the attribute does not include XIP we want to pay special attention because they will always be in this space and usually one of the lowest DLLs in memory (see bold items below). Another way might be to look at the load position of each component in something like Remote Process Viewer and see if it s using loading below 0x If the component has an XIP attribute, then it s a ROM based module and usually loaded into a special area in Slot 1 (outside the area we re worried about). Sometimes, when the XIP region gets full, these components will start getting pushed down into Slot 0. When that happens, they affect our VM space the same way as non-xip components. Anything below 0x that is not labeled XIP is what we re primarily concerned about. If I look at the dumpmem.txt file I just created, that section looks something like the list below (I ve trimmed off each module list after the first 3-4 items). Processes and their Modules Module File Attributes Legend RO - Read Only H - Hidden S - System A - Archive C - Compressed

3 3 of 8 04/19/ :53 PM C - Compressed NK.EXE base address: C coredll.dll 03F4E RO, H, S, XIP coredll.dll.0409.mui 7FFE RO, S, XIP hd.dll RO, H, S, XIP filesys.exe base address: rsaenh.dll B000 RO, H, S, C, RAM from ROM vcefsd.dll C000 RO, H, S, XIP ramfmd.dll 01A RO, H, S, XIP device.exe base address: rsaenh.dll B000 RO, H, S, C, RAM from ROM serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP serdma.dll 01A RO, S, XIP... services.exe base address: lap_pw.dll RO, S, C, RAM from ROM rsaenh.dll B000 RO, H, S, C, RAM from ROM obexsrvr.dll A000 RO, H, S, XIP gwes.exe base address: 0A kbdmouse.dll RO, H, S, XIP touch.dll 019B0000 A000 RO, H, S, XIP deviceemulator_lcd.dll 01B D000 RO, H, S, XIP shell32.exe base address: 0C wlmhssearchbarcode.dll RO, S, C, RAM from ROM wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM wlmuiframework.dll 018E RO, S, C, RAM from ROM wlmtodayscreen.dll RO, S, C, RAM from ROM siminit.dll RO, S, XIP EmulatorStub.exe base address: 0E coredll.dll 03F4E RO, H, S, XIP coredll.dll.0409.mui 7FFE RO, S, XIP poutlook.exe base address: ossvcs.dll 0300E RO, S, XIP oleaut32.dll F000 RO, H, S, XIP ole32.dll 0307F000 2C000 RO, H, S, XIP connmgr.exe base address: rsaenh.dll B000 RO, H, S, C, RAM from ROM ril.dll RO, S, XIP cspvoice.dll RO, S, XIP ConManClient2.exe base address: devicedma.dll B000 A

4 4 of 8 04/19/ :53 PM devicedma.dll B000 A edbgtl.dll A coredll.dll 03F4E RO, H, S, XIP HelloWorld_1.exe base address: mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP netcfagl2_0.dll 0218C000 3B000 RO, H, S, XIP mscoree.dll 021C RO, H, S, XIP fexplore.exe base address: ossvcs.dll 0300E RO, S, XIP oleaut32.dll F000 RO, H, S, XIP ole32.dll 0307F000 2C000 RO, H, S, XIP dumpmem.exe base address: 1A toolhelp.dll 02E5F RO, H, S, XIP ossvcs.dll 0300E RO, S, XIP oleaut32.dll F000 RO, H, S, XIP Step by Step Build a list of the DLLs loaded below 0x If you make a list of all the DLLs in every process that have been loaded under 0x and order them by load position in memory, you get a pretty good idea of what s going on with the low DLL load position and which processes have contributed to pushing it down to its current position. Our current list of all DLLs loaded under 0x looks like this: Component load position size -- attributes devicedma.dll B000 A (lowest DLL in memory) edbgtl.dll A wlmhssearchbarcode.dll RO, S, C, RAM from ROM wlmshared.dll 018A0000 3C000 RO, S, C, RAM from ROM wlmuiframework.dll 018E RO, S, C, RAM from ROM wlmtodayscreen.dll RO, S, C, RAM from ROM lap_pw.dll RO, S, C, RAM from ROM rsaenh.dll B000 RO, H, S, C, RAM from ROM kbdmouse.dll RO, H, S, XIP touch.dll 019B0000 A000 RO, H, S, XIP serial_smdk2410.dll 019E0000 A000 RO, H, S, XIP serdma.dll 01A RO, S, XIP s3c2410x_wavedev.dll 01A RO, H, S, XIP pwrbtn2410.dll 01A RO, H, S, XIP pcmcia.dll 01AA0000 A000 RO, H, S, XIP pcc_smdk2410.dll 01AB RO, H, S, XIP pcc_serv.dll 01AC0000 F000 RO, H, S, XIP nullcam.dll 01AD0000 C000 RO, H, S, XIP nleddrvr.dll 01AE RO, H, S, XIP irsir.dll 01B RO, H, S, XIP emulserv.dll 01B RO, H, S, XIP dmatrans.dll 01B RO, H, S, XIP battdrvr.dll 01BF RO, H, S, XIP backlight.dll 01C RO, H, S, XIP asyncmac.dll 01C RO, H, S, XIP scard.dll 01C40000 B000 RO, H, S, XIP mscoree2_0.dll 01C70000 C4000 RO, H, S, XIP Step by Step Analyze the memory data as it applies to your process Low DLL load position This is a pretty small list of DLLs using a simple environment (it s the WM 6.0 SDK refresh Pro emulator). You can see the lowest DLL (devicedma.dll) loaded at 0x (~24MB). You may also notice that if you add the DLL size to a load position, there may be a small gap before the next DLL loads in memory. In WM5, we had to load components on a 64K boundary. This means even if you loaded two DLLs that were 10K in size, they would eat up 128K of VM. In WM6, we use a 4K boundary which makes more efficient use of memory. From our data, it appears that the next DLL loaded into memory (from any process) is going

5 5 of 8 04/19/ :53 PM From our data, it appears that the next DLL loaded into memory (from any process) is going to find a home right under 0x That s where the lowest module is loaded right now and the next one has to load underneath it. This answers our first question about where the low DLL load position is. As you can see, we re already a long way down from 32MB since this is ~24 MB in the VM map. The low DLL load mark has already been reduced by 8MB. What pushed it down that far? You could go back through the list of component under 0x and find the processes that loaded them. That would tell you which processes are the biggest culprits. If you did this, you would see that shell32.exe is responsible for loading several Windows Mobile Live components, but even so the largest one is only about 240K. The real culprits are all the system components and drivers that have been pushed down into Slot 0 because slot 1 was full. This is very typical of commercial devices. That s what pushed down the low DLL load position by 8MB. This number is highly variable for retail devices and something you typically don t have a lot of control over. It s also why if you have a big application that requires a lot of VM space to load DLLs, you need to be especially careful to profile your target devices. What about our heap space? To get an understanding of available heap space left, I usually look at the first section of the dumpmem.txt file called Virtual Memory Dump (left column below). This is a simple outline of your memory space and how it s being used. If I identify the heap area between where my EXE is loaded and before the first DLLs appear in this VM Map, I can find the largest available free memory area left in my process. Before we look at the virtual memory dump, let s clarify one point. You will notice that all the memory positions seem to be offset from 0x You may recall that the base address for HelloWorld_1 was 0x (where it lives when it s not the active process in slot 0). Just subtract from all the memory positions if it s easier to think about it in the context of Slot 0. Here we go Virtual Memory Dump of HelloWorld_1.exe F NA *The first little block of memory is in size. This is the reserved position at the bottom of each process and will always be here for any process. Next comes our EXE R NA Image C RW Image R NA Image C RO Image R NA Image C RO Image R NA Image *The memory from is labeled as image. This is the bottom of the VM position and where our EXE is loaded. The lowest section labeled Image is always your EXE. Next comes our Heap and Stack memory F NA C RW Private F000 F NA D000 R NA Private 1603D C RW Private C RW Private A000 R NA Private 1606F F NA C RW Private A000 R NA Private 1609F F NA 160A0000 D000 C RW Private 160AD R NA Private 160CF F NA 160D C RW Private 160F0000 F000 R NA Private 160FF F NA C RW Private E000 R NA Private 1612F F NA F000 R NA Private 1613F C RW Private C RW Private F000 R NA Private 1615F C RW Private *The area from F000 is dynamic memory being used by the process heap, stack, etc. Notice there are a few free areas marked F but most of it is Private memory which has been committed or reserved. This is where the majority of your dynamic memory allocations occur and your heap expands upward B10000 F NA *At the top of your heap area you will usually find a big free chunk of memory. This is largest contiguous free chunk of memory available inside your VM space (Slot 0). In this case, it s 1B10000 or ~27MB. Next (at the top of the VM area working its way down) you will find any

6 6 of 8 04/19/ :53 PM Next (at the top of the VM area working its way down) you will find any DLLs loaded in our process. The first one shows up at 0x17C C R NA Image 17C C RO Image 17C R NA Image 17C C RO Image 17C7C R NA Image 17C E000 C RO Image 17CB R NA Image 17CB C RO Image 17CC R NA Image 17CC C RO Image 17CDA R NA Image 17CDF C RO Image 17CE R NA Image 17CE C RO Image 17CE R NA Image 17CEC C RO Image 17CF R NA Image 17CF C RO Image 17CF R NA Image 17D C RO Image 17D R NA Image 17D C RO Image 17D0A R NA Image 17D0C000 B000 C RO Image 17D R NA Image 17D C RO Image 17D1A R NA Image 17D1B C RO Image 17D1C R NA Image 17D1D C RO Image 17D1E R NA Image 17D C RO Image 17D R NA Image 17D C RO Image 17D R NA Image 17D C RW Image 17D2C R NA Image 17D34000 EC000 F NA 17E R NA Image 17E C RW Image 17E A000 R NA Image 17E C RW Image 17E R NA Image 17EA C RW Image 17EA R NA Image 17EFD C RW Image 17EFE R NA Image 17F C RW Image 17F0B R NA Image 17F1B C RW Image 17F1C000 1A000 R NA Image 17F C RW Image 17F37000 E000 R NA Image 17F C RW Image 17F C000 R NA Image 17F C RW Image 17F R NA Image 17F C RW Image 17F R NA Image 17F6C C RW Image 17F6E R NA Image 17FA F NA 17FE0000 1C000 R NA Image 17FFC C RW Image 17FFD R NA Image *Memory from 17C70000 up is labeled as image. This is memory used by DLLs remember they load from the top of the VM space and work their way down. All I care about at this point is the lowest position used in my process. If I map address lowest DLL loaded at 17C70000 back into Slot 0 (subtracting the base address), I get 1C70000 which can be easily identified as mscoree2_0.dll in the module section of HelloWorld.exe. From this information, we achieve our second goal. We know that there is a large free chunk of VM space (0x01B10000 bytes or- ~27MB), between our EXE and the lowest DLL in our process. The lowest DLL loaded into our process is mscoree2_0.dll at 0x01C Making sense of it all Let s now regroup on our objectives and put it all in context. Our goals were: 1) Understanding the lowest DLL load position in all processes:

7 7 of 8 04/19/ :53 PM 1) Understanding the lowest DLL load position in all processes: >>> devicedma.dll at position 0x (~24MB) 2) Identifying the largest chunk of available VM and lowest DLL in OUR process >>> we found 0x1b10000 bytes FREE right above our heap and we located mscoree2_0.dll at 1C70000 giving us about 27MB of free space to work with. This was simply example and done in a very clean environment so we re still in a pretty healthy situation. Goal #2 tells us we have plenty of space to grow (heap) and plenty of room to load more DLLs. Goal #1: The lowest DLL load position in OUR process is 0x1C70000 so you might expect the next DLL to load right under it. Considering #1 though, the next DLL would actually load significantly lower just below 0x Remember that when our process loads another component, it has to find a position under every other component that has already been loaded by any process. This, in effect, restricts our heap range a bit. Where we had a giant contiguous chunk of almost 27MB worth of VM space loading a component now would split up that area by positioning itself around 23-24MB in the map. Our heap can still grow upward and claim most of this available space, but it s no longer single contiguous chunk. This becomes important when you have an app that might need to make a big allocation (e.g. manipulating an image, media file, etc.). I say "most" of this space because if you are not running with full privileges (e.g. - untrusted on a 2-tier phone), then you are limited to available VM space under the low DLL load position -- even if there is plenty of free space up there! This is a really simple exercise, but you can imagine how complex this can get with a lot of applications running especially if they load a bunch of other components. While there may only be 32 processes running, don t forget that services.exe and driver.exe can become quite full with dlls that load on boot. This can contribute to pushing down that low DLL load position. Once the DLL position gets pushed WAY down, the available area left to grow your heaps can get quite constrained. If you start running into problems with VM issues or need to get an idea of what is going on, you can use the above techniques to get an idea of what you have to work with (and why it might be failing). Where s the smoking gun? We already noted that when applications experience VM related failures, there s not always a smoking gun. The failures can be appear quite random. Let s talk about some of the most common failures and contributors. Components that fail to load - The most common failure occurs when an application attempts to load a dependant component either on startup or through a call to LoadLibrary. There has to be enough space between your heap and the low DLL load position to succeed, so when the DLL load position gets pushed too far down, LoadLibrary will fail. At this point, an application can fail to start or a manual call to LoadLibrary will return an error. The actual affect on an application can vary depending on how it handles this failure. Memory allocations that fail - When a heap can no longer expand, memory becomes too fragmented, or there is insufficient space for a large contiguous allocation-- memory allocations can begin fail. Few developers write to code to adequately check for memory failures so this very often results in unhandled exceptions for the application (e.g. crash). Dynamic failures - It s pretty easy to take responsibility for problems in your own application failure to start, exceptions, etc., but it s even more complex that. VM problems commonly extend into other processes because of the DLL loading and memory rules. How does this happen? Because every process can contribute to the low DLL load position, this means that the order in which they start can make a big difference. Suppose the low DLL load position has been pushed down to the point where only 10MB of space is left to load more DLLs. What happens then if you have three application A, B, and C that all load 5MB worth of unique components. If you load A and B, then C fails. If you load B and C, then A would fail. Part of the challenge today is that VM health is very much a little ecosystem with many contributors. The OEM shares responsible for delivering a device that doesn t eat up all the VM space before you install the first app. The developer shares responsible for not taking more than their share of resources. The end user shares responsible for all the apps they install. Microsoft shares responsibility for mitigating this problem at a platform level so nobody has think about it. In the not so distant future, we will likely adopt changes in the platform (like we did with CE6) that expand the VM range so that this isn t an issue anymore. Right now, we have to work with what we have. Tactics developers use to address VM problems Profile the VM space first. You cannot solve this problem unless you know what is going on with your memory. Static linking is your friend. When you statically link code, it becomes part of thee EXE which means it loads in the bottom of the VM space (the best possible area). You don t have to worry about DLL load rules and you don t impact the VM of other processes. You may end up using a little more RAM if multiple apps use the same code, but the trade-off is probably worth the headaches you can avoid. Plus, you avoid extra signing costs by eliminating DLLs. Start order matters. While more of a band-aid than a solution, controlling the load order of processes on a device can help you work around VM issues. For this to work, you really need a good understanding of the memory profile of the device as well as which processes are contributing to the problem. The first app gets the most DLL space, the second gets the slightly less, etc. By controlling which apps start first and/or disabling services or drivers you don t need you can sometimes avoid VM problems. Merge small DLLs. Windows Mobile 5.0 had to load components on a 64K boundary which meant that every a small DLL used a minimum of 64K. If you have a bunch of small DLLs, it may make sense to merge them to save space. Windows Mobile 6 loads

8 8 of 8 04/19/ :53 PM small DLLs, it may make sense to merge them to save space. on a 4K boundary which makes much better use of space. Move resources to resource-only DLLs (dll s with no entry point). Resource only DLLs are loaded up in slot 63 and don t use up your valuable VM space in slot 0. For large memory allocations, consider using VirtualAlloc and the Large Memory Area (aka High Memory Area). Allocations of 2MB or greater use space well outside of slot 0 and leave your process with more room to work. Trim your code. Make it smaller and lighter and save on VM space In Summary I ve rambled long enough about the Virtual Memory Monster. Somewhere this is a blog post turned into whitepaper. Hopefully those of you who read this now have a better understanding of the creature and some skills you can use to battle it. Like most nasty, bad-guy type, it s more misunderstood than evil. Hopefully you never run into it, but if you do you have some tools under your belt. If you guys want to dig into this in more detail, let me know. I could blog through a real world problem or two to help you get the hang of it. Some of you will undoubtedly want to know more about how NETCF optimizes memory inside of this limited area. You will notice that if you load a NETCF assembly, it doesn t use up DLL space like native DLLs. NETCF goes to great lengths to make good use of memory transparently. Since there are already some great posts on this topic, I ll simply reference them here: Next up. Power Management or maybe IE Mobile? Both of these areas seem to be generating a lot of interesting support activity right now. Cheers, Reed