GENERAL... 2 SETTINGS AND CONFIGURATION... 7 USING THE LOGGING CAPABILITY... 9 LOGON SPEED...

Citrix Presentation Server FAQ FAQ DATASHEET Troubleshooting Frequently Asked Questions Table of Contents GENERAL... 2 SETTINGS AND CONFIGURATION... 7 USING THE LOGGING CAPABILITY... 9 LOGON SPEED... 12 www.citrix.com

General The Profile management (Pm) service does not appear to be doing anything? This may very well be one of the more commonly experienced symptoms. The purpose of this document is to guide you to identifying the root cause of problems you may encounter. This entire document is focused on drilling into these items and how to go a step deeper in understanding what is happening in your environment. As a quick summary, here is a check list to get you moving for some quick checks and verifications that everything is setup correctly. 1. Is the Group Policy setting (or INI setting) enabled o By default, Pm is not enabled to process profiles o ACTION: Enable Pm via setting in GPO or INI 2. Is logon an Admin Account (by default not managed) o ACTION: Use different account or change configuration to process administrators. It is recommended to have at least one admin account not managed for troubleshooting 3. Is user in a managed group o By default everyone except admin accounts are managed but this can restrict to only users in a specific domain group (and to include admin accounts) 4. Is the defined directory in the user store being created o Default user store location is Windows folder in home directory o Lack of this folder and any data in it suggests Pm is not doing anything o ACTION: Check APPINIT_DLL (see below) o ACTION: Verify above configuration for users/admins o ACTION: Make sure a HOME directory is defined or an alternative location is defined (e.g. unique per user UNC path with appropriate user permissions) 5. Check APPINIT_DLL XP and WS2003 ONLY o Vista and WS2008 do not use or need this o Registry value under key: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows o Value "AppInit_DLLs": entry "upmhook.dll" o Make sure the DLLs are in the Profile management installation directory (upmlogonmonitor.dll and upmhook.dll) 6. Check log file is the configuration being found o %SYSTEMROOT%\SYSTEM32\LogFiles\UserProfileManager o ACTION: verify UPM finding its configuration (and from where) in the log file (the log file will indicate settings found and from where) 7. Run gpresult (and generate RSOP in Group Policy Management Console) o ACTION: Is GPO being applied and is it the right one 8. Enable all logging (in GPO) o Same log file but now captures user level details o Captures everything happening with User/Pm 9. The profile cannot be unloaded correctly check for the following logfile entries 2

o CRegistryHive::Load: RegLoadKey of < > to < _logoff> failed with: The process cannot access the file because it is being used by another process. o ProcessLogoff: The user's registry hive cannot be loaded. Probable cause: an application or service did not close its handles properly. Consider using UPHClean. Aborting! o ACTION: Install UPHClean (on Windows XP and Windows Server 2003). UPHClean is already in Vista and Windows Server 2008. How to differentiate between Citrix profile manager issue and Windows profile issue? To see if it is a windows problem enable UserEnv Debug level: http://support.microsoft.com/kb/221833. Unfortunately, this is only available on Windows XP and Server 2003. Additional info can be extracted from the Profile management logfile. In the windows event log and in the Pm log I get the following critical error message: " GetUserStorePath: Home directory not found! Profile management first looks for the variables %HOMESHARE% and %HOMEPATH%. If that fails, it will query AD directly. The latter happens during logon, since Pm is notified before user environment variables are in place. By default Pm saves the user's profile in a subdirectory called WINDOWS within the home directory (this can be changed to any other directory or UNC path within the configuration). The error message indicates this directory cannot be found. Change the group policy "Path to the user settings" to a path which exists and is writeable for the user. Further information on how to set the %HOMESHARE% variable can be found in the following Microsoft knowledge base articles: If using cached credentials the variable %HOMESHARE% is not set (KB329407) If the domain controller's name cannot be resolved (KB892227) If hotfix KB818078 for Windows XP is installed (KB824898) If you are using a roaming profile (check with "set home") (KB841844, W2k, eg. KB833884, XP) What exactly is happening during logon and logoff? Please refer to the logon and logoff chart for a detailed logic: http://support.citrix.com/article/ctx119466 The NTUSER.DAT file is not getting its settings copied back to the user s central store? The other files and folders are being copied, just not the NTUSER.DAT. The log file has the following entries: 3

2009-01-08;13:42:35.585;ERROR;XD;Joe;1;5252;CRegistryHive::Load: RegLoadKey of <C:\Documents and Settings\Joe\NTUSER.DAT> to <S-1-5-21-2978719075-2539376396-840234952-1234_logoff> failed with: The process cannot access the file because it is being used by another process. 2009-01-08;13:42:35.585;ERROR;XD;Joe;1;5252;ProcessLogoff: The user's registry hive cannot be loaded. Probable cause: an application or service did not close its handles properly. Consider using UPHClean. Aborting! There is a process (probably a service) that keeps a handle to the user s registry hive open even after the profile is being unloaded by Windows. This prevents HKCU from being unmounted from the registry. The user s NTUSER.DAT file stays opened exclusively by the system and Pm cannot access the file. There are two possible solutions: [easy] Install UPHClean from Microsoft. Please note: This is only required on XP / Server 2003 since Vista / Server 2008 come with UPHClean already built in. http://support.microsoft.com/?scid=kb%3ben-us%3b837115&x=11&y=13 [difficult] Identify the culprit that keeps the registry handle open. This part is easy with UPHClean: it will log all relevant information to the event log. The hard part is to get the bug fixed in a service that may even be part of the OS. Using UPHClean for issues where the NTUSER.DAT is in use. Microsoft s UPHClean monitors the computer while user profiles are unloaded and forces the release of registry handles that are still open. Windows can then unload the user profile correctly. UPHClean can be used without problems in conjunction with Profile management and is recommended by Microsoft for Terminal Services and desktop operating systems. If you encounter issues because of open handles, Citrix recommends installing UPHClean. For detailed information see http://support.microsoft.com/kb/837115 The user s certificates do not appear to be captured in the Citrix user profile? In order to save personal certificates which have been imported into the certificate store during a session, the following directory must be added to the list of directories to be synchronized: Application Data\Microsoft\SystemCertificates\My The component Application Data in this path depends on the language of the operating system. If a policy is used to configure multi-language systems, the aforementioned path must be included in the list once for each language Example: on an English Windows XP and Windows Server 2003 system the path is Application Data\Microsoft\SystemCertificates\My, on a German system it is Anwendungsdaten\Microsoft\SystemCertificates\My. 4

If you do not alter the default configuration of Profile management, these certificates are automatically synchronized. What if the Profile management service crashes? If the service crashes, the normal windows mechanisms will be leveraged. As the user got a local profile, after logoff the local profile will remain on the machine. If the policy to remove local profiles after n days (assume n=1) is set (applies only on Vista systems), the profile will be deleted after one day (if the user did not log on again and the Profile management service has not been restarted). If the Profile management service has been restarted, it will check whether a newer Citrix user profile exists in the user store. If so, the local copy will be synched with the user store. I need to contact Citrix Support. What information should I collect? Please include the following files and information for Citrix Support to allow them to help you in the best and fastest way: All log files from Profile management (under "%SystemRoot%\System32\Log files\userprofilemanager" o Be sure you have all log settings activated. A log file from the affected machine should contain at least the following information: o Start of the service o Reading of the configuration by the service o One full logon process of the affected user o The activity the user is doing to cause the issue o One full logoff process of the affected user The resultant set of policies (RSOP) for the machine and user affected. This can be generated by the Group Policy Management Console (GPMC). What operating system, language and version is installed on the affected system If available: the Userenv debug file The INI file from the profile directory that contains all the settings The Provisioning Server INI file if applicable The more input you can provide, the better Citrix Support can assist you. How do I perform a session dump? Profile management allows you to dump its current internal state to a file. This is helpful in situations where you can isolate an issue to a specific point within a session but there is no visible output in the log file. Writing the log file is triggered by the creation of an (empty) file called $$UPM_log$$.txt. Pm detects the creation of the trigger file by reading the NTFS change journal. After writing the internal data log file the trigger file is deleted. Do the following to produce the dump file: 5

Create a file called $$upm_log$$.txt in the root of the drive on which the user profiles are located (typically C:). Profile management will automatically dump its internal data state to the file UserProfileManagerInternalData.log in the log directory and delete the trigger file $$upm_log$$.txt. Add the dump file to the other files sent to Citrix support. To facilitate debugging, Pm can be forced to write the state of the internal lists to the file UserProfileManagerInternalData.log in the Pm log directory. If the internal data log file already exists, it will be overwritten. 6

Settings and Configuration It appears as if some GPO settings unrelated to Profile management are not being applied? Cause is typically improperly excluding the following group policy settings in registry: Software\Policies Software\Microsoft\Windows\CurrentVersion\Policies Software\Microsoft\Windows\CurrentVersion\Group Policy These settings need to be included or excluded as a group. References: http://forums.citrix.com/thread.jspa?threadid=241514&tstart=0 http://blogs.sepago.de/helge/2009/02/25/citrix-user-profile-manager-howregistry-exclusion-lists-can-mess-up-group-policy-processing/ This was a known issue in the v2 release. And only the first two items had been configured in the default configuration. Citrix now recommends not including any of these settings in the exclusion list and allow them to remain in the profile s registry. I ve updated the configuration in the GPO / INI file and the settings are not being applied? Changes to GPOs are normally applied to the computer during the periodic background refresh. Pm gets notified of changes and rereads its configuration (from group policy and the INI defaults file). Note that Pm gets notified only if policies were actually changed. If you modified the INI defaults file and want Pm to reread its configuration, do either of the following: Run: gpupdate /force (recommended) Restart the Pm service [NOTE] To do both steps is currently the only way to guarantee to force a new location for the cache file. Both should work but in the second case it could be that Pm uses more CPU and disk time (as the file journal for all users has to be tracked for changes during the restart). What variables may I use when setting "Path to the user settings" or other configuration paths? 7

Since the Profile management service runs before the user logs on, only system environment variables and AD attributes may be used. There are two user environment variables that are allowed by exception. This would be %USERNAME% and %USERDOMAIN%. Basically any variable that is set before a user logs onto a system may be leveraged in the configuration. 8

Using the Logging Capability Everything appears to be configured properly but nothing seems to happen when the users logon. Nothing seems to ever be written to the log file? On Windows XP and Windows Server 2003, the DLLs that monitor the logon events are registered here: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs Verify that there is an entry for upmhook.dll. If not, another something may have deleted the entry (another application install etc). Repair or reinstall Pm will solve this issue. Also check whether 8.3 names are being generated. The AppInit_Dlls hooking mechanism supports only 8.3 paths. This is a legacy "feature" of Windows. The PM installer takes the full path to the hook DLL and creates the 8.3 version which it then appends to the AppInit_Dlls registry value. But this mechanism fails if the system is configured not to generate 8.3 names at all. The result is that the installer will use long path elements instead of short ones, causing Windows' AppInit_Dlls functionality to fail. Easiest option is to reinstall into a path which the name is 8.3 compliant. Additionally you can verify that if the account logging in is an administrator, the configuration has defined that administrators are processed by Pm. Also verify if a group is defined for Pm processing, the users who appear to be ignored by Pm do in fact belong to that group. Another possibility is that Pm is deactivated (default setting). Activate the appropriate setting in the GPO or INI file to enable Pm. E.g. in the INI: [General Settings] ServiceActive=1 Suggested steps for leveraging the log file. As a rule of thumb the first thing to do is check the log file of Profile management. In most cases you should be able to find out what the issue is by just consulting the log file. The log file is a good starting point for troubleshooting. The following is a step by step guide for troubleshooting. Enable logging to the maximum (see chapter Activating the logging ) Check the log file for errors. These can be found by searching for the string ERROR. Check the log file for warnings. These can be found by searching for the string WARNING. Check the log file after having run the command gpupdate /force on the machine on which the error occurs. Review which settings are set and from where the configuration has been read (group policy or INI file) Check if the location of the user store was determined correctly. Check if all information from Active Directory was read correctly Always check the time stamps. Is there an action lasting too long? 9

What is the format of the log file? Each line of the log file consists of several fields, all separated by semicolons: FAQ DATASHEET Field Date Time Severity Domain User name Session ID Thread ID Function and Description Description Date of the log entry Time of the log entry including milliseconds Either INFORMATION, WARNING or ERROR The Domain of the user (where applicable) The name of the user (where applicable) The session ID (where applicable) The ID of the thread outputting this line The name of the User Profile Manager function currently executing and the message to be logged What if nothing obvious appears in the log file? If you could not identify the reason you are experiencing issues using the log file, the following steps may help. Check if you have the latest version of Profile management installed. This can be done by checking the version info of the UserProfileManager.exe file (right click in Explorer->Properties->Version) o If you are not using the latest version please upgrade Maybe someone else already encountered your problem. Check the support forum at Citrix http://support.citrix.com/forums/forum.jspa?forumid=185. Enable User Environment Debug logging. See http://support.microsoft.com/kb/221833/en-us/ for further info. o Analyze the output file. Details to analyze can be found here http://technet.microsoft.com/en-us/library/cc786775.aspx. Always try to reproduce the issue you are observing on a clean machine. This machine should have the same operating system as the system you are encountering the issue(s) on. Only the operating system itself and Profile management should be installed. Check if the issue persists. o If not, add the software products that are installed on the affected machine one by one and try to reproduce the issue after each step. The output of the Userenv debug log can be extensive but it will provide you with a lot of information of the logon process of your system. How to capture log files when using Provisioning Server shared images. In Profile management v2.1, the ability to define a path for the log File to any local drive is available. Thus pointing to a persistent drive within a shared image can now be used to store the log file. 10

When using XenDesktop, another way to get the logs is to create an identical desktop group, place one VM in there and point it to the same VDisk. Only let the one test user and the administrator access this VM. Modify the desktop group Advanced Properties, set Idle Pool count to 1 for all times of the day (this will stop power management turning off when you need it most and set Logoff Behavior to do nothing (this will prevent the machine restarting and resetting the image to a pristine state). The user should be able to logon, logoff and then the admin should be able to go in (straight from the XenCenter or VMware console) and pull off the logs. The only time this might be an issue is if the PVS write-cache is stored locally, in which case, the Admin may have to use the same PC as the user. It is unlikely this will work if the write cache is stored in RAM. If the PVS write cache is stored in the PVS server, then this shouldn t be an issue. If this PVS write cache is not on the PVS server, you may have to create a copy of the VDisk and assign this copy to the new VM which we added to the new desktop group, and change the write cache on this copied VDisk to be stored on the PVS server. If the issue is occurring at logon (profile is current on network but not loading correctly) the admin should be able to just map to the \\vmhostname\c$ and pull off the logs before the user logs off the session. Obviously this would only give us a partial log but if the problem is at logon, then hopefully this may be enough to isolate the error. Another useful tip It is often useful when troubleshooting XD VM issues (in shared mode), to add a local disk to the XenServer VM (or VMware, HyperV VM!),be it only a small one around 1 Gb. Then use this procedure with the PVS Vdisk in Private Mode to redirect the Windows Event Logs to this physical disk. It will usually show up as E:\ but on previously diskless VM you will need to format it and bring it online whilst booting from the VDisk in Private mode. Remember also that part of the PVS imaging process is optimizing the disk for imaging, restricts EV logs to 64 Kb, you may wish to find the appropriate reg. entry and increase this if capturing logs over a long period. How to redirect the Event Viewer logs. 1. Open REGEDIT (or another Registry editor program) and navigate to the key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\eventlog 2. Open the subkey that contains the event log you want to move. On most machines, you'll be able to choose between Application, Security and System. 3. Each key contains a value named File (type REG_EXPAND_SZ), which contains the pathname and filename to the log file. By default this is %SystemRoot%\system32\config\.Evt. You can provide a new pathname and filename here, but you should use the.evt file extension. 4. Close the Registry and restart the computer. You can also use Policy Preferences in Group Policy to changes these registry entries if you prefer. You can redirect most types of logs in this way (not all but most!). 11

Logon Speed The user logon time still seems to take too long. What can I do to identify the issue? If this is the user s first logon, please note that as opposed to a roaming profile that does not write the new profile data to the network location until logoff, Profile management does so at initial logon. Thus the initial logon will take longer compared to a roaming profile BUT the logoff will be faster (again compared to a roaming profile on first use). Pm might help improve this but it really depends on what is in the profile and what is consuming the time to load it. If the profile is being weighted down by bloat (e.g. there are many extra settings, files and folders in the profile that are not really needed but are being captured and stored) then Pm may very well help improve this significantly. E.g. with Pm you can granularly define what is in and what is excluded from a profile. Thus by managing the size, the time it takes to copy down is optimized and improved. Here is a blog on that topic. http://community.citrix.com/x/a4aaag Also with Pm activated and logging activated to the maximum, additional information is captured into the logfile. So not only the profile loading process (NTUSER.DAT and files) but also times querying AD, getting correct paths, etc. is logged with time stamps. In many cases you can identify a bottleneck this by having a look at the Pm log file. At the end of the loading process of the profile, Pm summarizes this. Here s an example: 2008-10-12,15:44:46.552,INFORMATION,SEPAGO,Joe,2,DispatchLogonLogoff: ----- ----- Finished logon processing successfully in [s]: <2.28>. Tip: Always search for -------- to get starting and end points of logon / logoff events with accumulated information. Per the log file showing 2.28 seconds to process the logon, this would be a great performing environment (this example was only processing the Registry and not files). If the time captured was similar to Finished logon processing successfully in [s]: <20.00>. (or more) you should have a look at the single timestamps in the log file and identify if a big gap exists between two entries. That is presumably the point you can start to find out what is going wrong. If there is no such gap, have a look if the profile size maybe is very big. A 2GB profile may just take a while to load One additional note: If configuring the Pm setting Delete locally cached profiles on logoff to disabled a significantly gain in logon speed would be achieved on those machines. However this does not make sense on all machines (e.g. XenApp where you would not want to cache all user profiles unless you had ample local disk storage in other words a lot of local disk storage). 12

www.citrix.com About Citrix Citrix Systems, Inc. (NASDAQ: CTXS) is the global leader and the most trusted name in Application Delivery. More than 230,000 organizations worldwide rely on Citrix to deliver any application to users anywhere with the best performance, highest security and lowest cost. Citrix customers include 100 percent of the Fortune 100 companies and 99 percent of the Fortune Global 500, as well as hundreds of thousands of small businesses and prosumers. Citrix has approximately 8,000 partners in more than 100 countries. Annual revenue in 2007 was $1.4 billion. 2009 Citrix Systems, Inc. All rights reserved. Citrix, Citrix XenApp, Citrix Presentation Server and Citrix XenDesktop TM are registered trademarks of Citrix Systems, Inc. in the United States and other countries. Microsoft, Windows, and Sharepoint are registered trademarks of Microsoft Corporation in the United States and other countries. All other trademarks and registered trademarks are the property of their respective owners.