Troubleshooting Citrix MetaFrame Procedures

Troubleshooting Citrix MetaFrame Procedures Document name Troubleshooting a Citrix MetaFrame environment v1.0.doc Author Marcel van As Last Revision Date 28 February 2006 Edited and released by: www.dabcc.com

Index Index... 2 Introduction... 3 Intended Audience... 3 Basic flow for troubleshooting... 4 Troubleshoot Secure Gateway... 5 Troubleshoot Server and Services... 7 Troubleshoot Citrix Services... 9 Troubleshoot Terminal Services... 12 Troubleshoot printing problems... 14 Troubleshoot ICA Session Performance... 16 Troubleshoot Server Performance... 18 Troubleshoot Session Sharing Problems... 20 www.dabcc.com Page 2 of 21

Introduction Trying to track down and resolve technical problems on a Citrix MetaFrame Presentation Server environment could be one of the most frustrating experiences. To help you through the process, this document explains the necessary steps you need to do for effective troubleshooting on a Citrix environment. In addition, this document addresses some common problems. This document is version 1.0, so if you have questions and/or comments then please send them to: troubleshoot@van-as.tk Intended Audience This procedure is intended for system engineers or administrators responsible for maintaining and supporting a Citrix MetaFrame Presentation Server environment. The described procedures are based on the assumption that you have: An understanding of server based computing concepts Knowledge of Citrix MetaFrame Presentation Server Demonstrated knowledge of at least Windows 2000 Server www.dabcc.com Page 3 of 21

Basic flow for troubleshooting The basic flowchart for troubleshooting a Citrix MetaFrame environment and Citrix Secure Gateway is shown in Figure 1 below. Each of the circled numbers is described on the pages after the flowchart. All actions highlighted in blue are described in other paragraphs. www.dabcc.com Page 4 of 21

Troubleshoot Secure Gateway The flowchart for troubleshooting the Citrix Secure Gateway is shown in Figure 2 below. Each of the circled numbers is described on the pages after the flowchart. 1. CSG service running - The secure gateway runs as a service Secure Gateway Service. Launch the services MMC on the corresponding server and check if the Secure Gateway Service is running. 2. Start CSG service - Through the services MMC the Secure Gateway Service must be start. 3. Start CSG diagnostics - Launch the Citrix Secure Gateway diagnostics tool. Watch the output to find the problem area. www.dabcc.com Page 5 of 21

4. Reinstall or renew certificates - When the CSG diagnostics identifies the SSL Certificates are invalid reinstall the certificates or renew the certificated when they are expired. In case private certificates are used verify the root certificate is installed and not expired. 5. Check if the IIS service is running - When the IIS service is not running the Web Interface is not accessible and the logon agent is not working. Launch the IIS manager MMC and check to see if the IIS service is running and if all the websites, particularly the Web Interface, are online. 6. Start IIS service - Start the IIS services if not running. Enable the websites of not running within IIS. 7. Check if the XML services are running - Check all the MetaFrame servers to see if the Citrix XML Service is running 8. Start XML Service - Start the Citrix XML Service if not running on the Citrix MetaFrame server. Start the IIS service and enable the website (default website) containing the Web Interface. 9. Launch the WIAdmin - By launching the Citrix Web Interface Console we can view and or change the secure gateway settings and Web Interface settings. 10. Consult IIS specialist - If you reach this state you have IIS running but the website is not functional. This could be caused by port conflicts or other faulty configured IIS setting. This document does not provide information for troubleshooting IIS. Consult Microsoft or IIS Specialist. 11. Reconfigure the Web Interface - Reconfigure the Web Interface farm settings and provide the correct Citrix servers for the farm. Reconfigure the Web Interface server side firewall settings and provide the correct secure gateway server and the STA servers. 12. Check if IIS is running on the STA servers - Launch the IIS manager MMC and check to see if the IIS service is running and if all the websites are online. 13. Start IIS Service - Start the IIS services if not running on the STA servers. Enable the websites if not running within IIS 14. Check if IIS Script path is accessible - Check if the default website containing the scripts path and if its accessible and has scripts only execute rights. 15. Set ACL s - Set ACL s on the scripts path and set scripts only execute permissions. 16. Consult Citrix specialist - Consult Citrix or a Citrix specialist for troubleshooting the secure gateway configuration. www.dabcc.com Page 6 of 21

Troubleshoot Server and Services The flowchart for troubleshooting the server and standard services is shown in Figure 3 below. Each of the circled numbers is described on the pages after the flowchart. All actions highlighted in blue are described in other paragraphs. Check server & services Start 1 Server Online No Power on server Yes 2 3 Server accesible No Check Network 4 Yes 10 All services running No Start Services Success Finished 5 Yes 11 Failed Logon using RDP Success Check Citrix Services Yes Citrix related service 6 Failed No 12 13 Success Logon on console Success Check Terminal Services Check Eventlog Known Errors Fix errors 7 Failed 8 Unknown Errors Failed Administrative logon Success Debug logon proces Rebuild server 9 Failed Member of Domain Success Failed Join Domain 1. Server online Check if server is powered on. 2. Server accessible - Check if server is online in network. 3. Check Network - Check if network cables are plugged in. Check TCP/IP settings and change if necessary. www.dabcc.com Page 7 of 21

4. All services running - Use services MMC to check if all services are running. 5. Logon using RDP - Logon as a user to check if it is a Citrix problem or not. 6. Logon on Console - Logon as a normal user on the console. 7. Administrative logon - Logon as a administrator on the console 8. Debug logon process - Set the following registry key on the server to debug the logon process: Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon Value: UserEnvDebugLevel Type: REG_DWORD Data: 10002 (Hex) The data value of 10002 will enable verbose logging to a file on the server. Once you set this value, reboot your server and check for a userenv.log file in the %System Root%\Debug\UserMode\ folder. Remember to turn this off when you re done troubleshooting it, since each user logon can easily add 100KB to the size of this log file. 9. Member of domain - Check if the server is still a member of the domain and if the server can successfully contact the domain controllers. 10. Start Services - Start the services which are not started 11. Citrix Related Service - Is the failing service a Citrix Related Service such as the Citrix XTE server or the Independent Management Architecture 12. Check Event Log - Check the Event Log for warning or error messages. Lookup the Event ID s to check if it s a known error. 13. Fix error - If the error message is a known error fix it according the given resolution. www.dabcc.com Page 8 of 21

Troubleshoot Citrix Services The flowchart for troubleshooting the Citrix Services is shown in Figure 4 below. Each of the circled numbers is described on the pages after the flowchart. 1. Application or server problem - Find out if it s a server or application problem. If it s a server problem no Citrix application will start. If it s an application problem some applications can start with Citrix. 2. IMA Service running - Check with the services MMC if the service Independent Management Architecture is running and no errors for the IMA service exists in the server Event Log. 3. Restart IMA service - If the service Independent Management Architecture is not running start the service www.dabcc.com Page 9 of 21

4. IMA directory in path - Check the PATH environment to see if the directories <ProgramFiles>\Citrix\System32 and- <ProgramFiles>\Citrix\System32\IMA\Subsystems are in the path Set PATH Change the path environment and add the directories <ProgramFiles>\Citrix\System32 -and- <ProgramFiles>\Citrix\System32\IMA\Subsystems to the path environment 5. Datastore online - Check if the data store is online. Start the Citrix Management Console and on the Resource Manager Watcher Tab check if Data store Connection Failure is present and in error state. Or check this from the MS SQL Enterprise Manager and see if the CITRIXIMA database is online. 6. Bring data store online - Bring the data store online by restoring the database or rebuild the entire SQL server. As a temporary option fail over to the replicated database. To use the replicated database edit <ProgramFiles>\Citrix\Independent Management Architecture\MF20.dsn and change the server entry to the server hosting the replicated database. Execute the following command: dsmaint config /user:sa /pwd:citrix /dsn: <ProgramFiles>\Citrix\Independent Management Architecture\MF20.dsn 7. Datastore online - Clean the localhost cache by executing the command: dsmaint /recreatelhc Start the IMA service 8. Config IMA DSN - From within the ODBC administrator create a new file DSN with the connection to the CITRIXIMA database. Execute the following command: dsmaint config /user:sa /pwd:citrix /dsn: <path_to new_dsn_file>\<new_dsn_file> 9. ICA Port enabled - Telnet to port 1494 on the server and check if the ICA port is enabled and has a listener on it. 10. Enable ICA Port - Start the Citrix connection configuration and enable the ICA-TPC protocol 11. DSCheck - Run dscheck.exe from the command prompt and watch for errors. 12. DSCheck /Clean - Run dscheck.exe /clean from the command prompt to clean the data store. 13. Restore data store - Restore the data store from backup www.dabcc.com Page 10 of 21

14. Check Published Application - Check if the published application is enabled and the application location and startup options are correct. 15. Install / Repair Application - From the add/remove software control panel repair the application. Or (re)install the application from within the Citrix Installation Manager 16. Check application client options - Check the application client options from within the Citrix management console. Check if the client options like screen, colors and encryption are correct and the same as the other applications. 17. Set application client options - Set the application client options according the standard 640x480@16bit with basic encryption and sound enabled. www.dabcc.com Page 11 of 21

Troubleshoot Terminal Services The flowchart for troubleshooting the terminal services is shown in Figure 5 below. Each of the circled numbers is described on the pages after the flowchart. 1. Logon Enabled - Check if logons are enabled by executing the command change logon /query or from the MetaFrame settings of the server properties within the Citrix management console. 2. Enable Logons - Enable logons by executing the command change logon /enabled or by checking the option enable logons to this server from the MetaFrame settings of the server properties within the Citrix management console. www.dabcc.com Page 12 of 21

3. User TS Enabled - Check if the user has the properties Allow logon to terminal server enabled in active directory. 4. Enable Allow TS logon - Enable the options Allow logon to terminal server for the user in Active Directory. 5. Application Mode Enabled - Launch the Terminal Services Configuration MMC and check if Terminal Server Mode is set to Application Server. 6. Enable Application Mode - Launch the add/remove software control panel. Click the Add/Remove Windows Components button and click next. Change the Terminal Services option to Application server mode. 7. TS Licenses Available - Check if a Terminal Server License Server is available, activated and licenses are installed. 8. Install Licenses - Install a Terminal Server License Server and install the Terminal Server Licenses. 9. Administrative Logon - Try to logon as a administrator using a RDP session 10. Debug usrlogon process Set the following registry key on the server to debug the logon process: Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon Value: UserEnvDebugLevel Type: REG_DWORD Data: 10002 (Hex) The data value of 10002 will enable verbose logging to a file on the server. Once you set this value, reboot your server and check for a userenv.log file in the %System- Root%\Debug\UserMode\ folder. Remember to turn this off when you re done troubleshooting it, since each user logon can easily add 100KB to the size of this log file. 11. Check Event Log - Check the Event Log for warning or error messages. Lookup the Event ID s to check if it s a known error. 12. Fix errors - If the error message is a known error fix it according the given resolution. www.dabcc.com Page 13 of 21

Troubleshoot printing problems The flowchart for troubleshooting printing problems is shown in Figure 6 below. Each of the circled numbers is described on the pages after the flowchart. 1. Printers available on Client - Are the desired printers and drivers installed on the client. 2. Install Printers on Client - Install the printers and drivers on the client machine. 3. Client Printer Mapping Enabled - Check in the Citrix Connection Configuration Client Settings if the windows client printer mapping is enabled. 4. Enable Client Printer Mapping - Enable the Windows Client printer mapping in the Citrix Connection Configuration tool. 5. Events 1106 and 1107 in server Event Log - Check the Application Event Log on the Citrix servers for 1106 and 1107 errors from source MetaFrameEvents and category Printer Management. The event description will tell which printer failed for auto-creation and what driver should be used. www.dabcc.com Page 14 of 21

6. Printer Driver installed on Server - In the Citrix Management Console - Printer Management - Drivers, check whether the driver is available in the farm and installed on all servers. 7. Install Printer Driver on Server - If the Printer Driver is available in the Citrix Farm on a server select the server with the driver and replicate the driver to the servers with the missing driver. Depending on the load and number of servers this can take up a few hours before replication is completed. If the driver is not in the Farm install the Windows 2000 version, NOT a version-2 (NT) driver, of the printer driver on a server and replicate the driver from that server to the other servers. 8. Create Printer Mapping - When driver names are not exactly the same on the client and the server, printer creation will fail. In the Citrix Management Console create a printer mapping with the client driver name with the corresponding server driver. Open the 1106 and 1107 error in the application Event Log and copy the client driver name. Paste the client driver name in the Citrix management console and select the corresponding or compatible server driver. Examples: Client: HP Laserjet 4000 PCL Server: HP Laserjet 4000 Series PCL Client: HP Laserjet 4/4M Server: HP Laserjet 4 Client: HP Laserjet 1100 Server: HP Laserjet Series II Client: Lexmark 2205 Server: HP Laserjet 4 Note: If you are receiving a lot of 1106 and 1107 errors then you might be interested in DABCC Advanced Print Manager (www.dabcc.com/apm) www.dabcc.com Page 15 of 21

Troubleshoot ICA Session Performance 1. Session Active - Start the Citrix Connection Center by double clicking on the ICA symbol in the notification area (clock). Check if ICA session is still active. 2. Reconnect Session - Reconnect the session by launching the application again or from the Citrix Web Interface with the reconnect button (if available). www.dabcc.com Page 16 of 21

3. Noticeable Traffic - Check the properties of the ICA session. Check if the session generates data to and from the Citrix server. 4. Network Active - Check if the network is still active. Refresh application on the Web Interface. Log off from the Web Interface and logon in the Web Interface again. 5. Normal Resource Usage - Application looks unresponsive. Check if application consumes too many resources like memory and CPU or causing the CPU queue to fill up. If abnormal resource usage is detected logoff or kill user process on Citrix server. 6. Force logoff after 15min- Disconnect the users session and try reconnecting. If the application is still unresponsive, let the application run for 15 minutes. Watch resource usage of the application during this period. If the application is still unresponsive, logoff the user session or kill the user process. 7. Fix Network - Take appropriate actions to repair the network. 8. Kill Application - Kill the application if it is using too many resources which can affect other users performance. Try logging of the user session first, otherwise kill the application or reset the session. 9. Launch new application - Launch a new application and watch if it is launched within the same session or a new session is created. 10. Original app unresponsive - If the original application is unresponsive a new session is created by launching a new application. Profile corruption, networking issues, database timeouts and macro s (like VBA script) are often the cause of hanging or unresponsive applications. 11. Force logoff after 45 minutes - If normal resource usage is normal, let the unresponsive application run for least 45 minutes. Watch resource usage of the application during this period. If the application is still unresponsive, logoff the user session or kill the user process. 12. New App normal Speed - Is the new application running at expected or reasonable speed. 13. Original App problem - The original application is having a problem. The problem is often caused by networking issues, database timeouts or internal application faults. Make note of the last actions made in the application like selections and queries. 14. Contact application support - Contact application support to troubleshoot the application. 15. Other users affected - Do other users have the same problem? If so, make note of the other users and contact Citrix administrators / Citrix support to troubleshoot server performance. 16. Shadow Session - If possible try to shadow the users session. 17. Expected behavior restored - Is the expected behavior of the application restored by shadowing the users session? 18. Reduce color quality - Does reducing the color quality (256 or 16 colors) on the users desktop increase the session performance? 19. Increase Session Memory / Update ICA Client - Increase session memory for the Citrix farm in the Citrix management console. If possible update the ICA client to at least a version 8 Client. If the problem happens often with the same application, the application is suffering on a GDI memory leak. www.dabcc.com Page 17 of 21

Troubleshoot Server Performance www.dabcc.com Page 18 of 21

1. Monitor Server Performance - Monitor the following performance counters Object Counter Instance Description Terminal Services Active Sessions Total user session connected Process Working Set _ Total Real physical memory used Memory Pages Input/Sec Memory read from pagefile (1 page = 4kB) Memory Pages Output/Sec Memroy written to pagefile (1 page = 4kB) Memory Available MBytes Estimate memory free Memory Free System Page Table Entries Free System PTE s Paging File %Usage _Total Pagefile usage Processor % Processor Time All CPU usage Instances System Processor Queue Length Number of processes waiting Cache Copy Read Hits % Memory efficiency of system file cache 2. Run Away User Processes - Check with the task manager for run away processes. 3. Kill Process - Kill the run away process by logging of the user holding the process or by terminating the process. 4. Memory Usage / User process - Check with the task manager the memory usage and VM size per process. 5. Kill Process - Kill the process if it is consuming too many memory and causing excessive paging. 6. Memory Efficiency - Check the performance counter Cache Copy Read Hits %. Constant values below 90% indicates a problem with kernel memory. 7. CPU Queue - With performance monitor measure System Processor Queue Length. This counter represent the number of processes waiting to be processed. A rule of thumb is a maximum of 2 processes per processor. 8. Reduce Processes - Reduce the number of processes by stopping some services or by logging of users from the system. 9. Network Queue - Check the Output Queue Length of the network interface with the performance monitor. If the output queue length is any value above 0 it indicates a problem with the network or network interface. 10. Troubleshoot Network - Troubleshoot the network connections, network load, network interfaces or other network components. 11. Disk Queue - Check the performance counter LogicalDisk Current Disk Queue Length. 12. Troubleshoot Disk Subsystem - Check the server for failing hard drives. Put pagefile on separate hard drive with it s own scsi or raid controller. 13. 3rd Level Support - Escalate the problem to a 3 rd level support group. 14. Restart Spooler - Restart the spooler service to free up some memory. 15. Stop Kernel Processes -Stop some kernel processes by stopping some services like NetIQ, SNMP, HP/Compaq monitoring agents. This will free up some kernel memory and creates a shorter process list. 16. Reduce Users - Logoff some of the users or add more servers. This will increase the performance. www.dabcc.com Page 19 of 21

Troubleshoot Session Sharing Problems Session Sharing Start 1 Session active and responsive No Troubleshoot ICA Session Performance Yes 2 Application on Same Server No 3 Publish on Server Yes 4 Same Application Appearance No 5 Set Application Appearance Yes 6 Same Client Options No 7 Set Client Options Yes 8 ICA Client Version < 8 9 Update ICA Client => 8 10 3rd Level Support 1. Session active and responsive - Is the first session still active and responsive? 2. Application on Same Server - Is the new application installed and published on the same server? 3. Publish on Server - If possible Install and Publish the application on the other servers. 4. Same Application Appearance - In the Citrix Management Console check if the applications have the same application appearance. 5. Set Application Appearance - Set the application appearance the same for all applications. www.dabcc.com Page 20 of 21

6. Same Client Options - Check the client options in the Citrix Management Console. Sound and Encryption options should be set the same for all applications. 7. Set Client Options - On the client options on each published application. Set the sound and encryption options the same for all applications. 8. ICA Client Version - Check the ICA Client Version on the users desktop. There should be at least version 8 installed. 9. Update ICA Client - Update the ICA Client to at least version 8 to improve the ICA session sharing feature. 10. 3rd Level Support - Escalate the problem to a 3 rd level support group. www.dabcc.com Page 21 of 21