BEST PRACTICES FOR MANAGING SIEBEL APPLICATION PERFORMANCE

BEST PRACTICES FOR MANAGING SIEBEL APPLICATION PERFORMANCE KERI McCLELLAN SABRE HOLDINGS Overview In 2004 the Siebel application encountered numerous performance issues. Hear a real world explanation of how the Sabre improved the application performance, achieved stability and manages/maintains performance and stability. Some of the actions taken include changes/improvements to desktop settings, application monitoring and use of monitoring tools, tight partnership with Siebel, code review and corrections and various database improvements. The presentation is focused on the business steps that were taken to improve internal customer satisfaction. This discussion will not be a technical track. Some technical details will be presented, but deeper questions will be taken offline. Session Objectives Four objectives have been identified for this session. Provide specifics of how Sabre manages application performance. Describe some of the application performance issues Sabre has experienced. Discuss the role of Siebel and their partners in Sabre's application success. Open discussion among session attendees about their personal application performance experiences. Introduction Sabre Holdings (TSG) employs approximately 9,000 resources in 45 countries. The largest population is located in Southlake, Texas where the corporate headquarters are located. The second largest population of employees is located in Montevideo, Uruguay. Sabre has established its Global Customer Support Center there and has hired over 700 Uruguayans. In 2006, revenue reached $2.8 billion. Sabre is committed to connecting people to the world s greatest travel possibilities. This is achieved by providing support to travelers, travel agencies, corporations and travel supplier with our products in 3 major lines of business, Sabre Travel Network, Sabre Airline Solutions and Travelocity. Siebel Deployment @ Sabre Holdings Sabre has two Siebel deployments, icrm and IMS. icrm is Sabre s internal CRM tool. This deployment went live in November, 2002. Release 1 took approximately 12 months to develop subsequent to the RFP process. Currently icrm has over 1,500 users. It is used to support airline customers and travel agency customers in a B2B service model. The Siebel application functionality is very broadly configured and takes advantage of many Siebel modules/offerings. For example, icrm has enabled Call Center, Contracts, Marketing, Campaigns, email Marketing, Partner Portal, eservice, Disconnected client, and SAP Connector. The IMS deployment went live in March, 2005. It is completely separate hardware and configuration. Release 1 took approximately 90 days from contract signing to go live. Approximately 1,500 users access IMS and the service model is different than icrm. This deployment is a B2C service model supporting Travelocity consumers, i.e. travelers. Siebel is very narrowly configured within this application. It is only limited to 20 Call Center views. This deployment has not encountered any challenging performance issues. COLLABORATE 07 Copyright 2007 by Keri McClellan Page 1

Performance Experience A primary difficulty that this team has experienced is defining the problem. One dimension of the problem is referred to as intermittent slow performance. This represents a comment that we have heard from users stating The system was slow today, but it was OK yesterday. Additionally, users have reported that they were working in the Siebel application and suddenly their session was disconnected. The team calls this situation a dropped connection. Another very common complaint from users was that the system would just hang or freeze during their Siebel session requiring them to cancel the task and establish a new Internet Explorer session and Siebel login. When executing queries from within Siebel, users would wait a long period (30 minutes to an hour or more) for the results and then find that no records were returned. In an attempt to clearly articulate the performance problem, user interviews were conducted from the executive level down. The result was literally a kitchen sink of complaints. For example, users indicated the need to reduce duplicates and improve data quality, raised issues around email processing and formats, complained that the tool was too complex, feel that management is not using the tool and why should they. Pinpointing when problems began is also challenging. Records indicate that about 6 months after go live users sporadically reported some issues. By this point, the icrm application had expanded to almost 800 users and was still growing rapidly. The production support resources attempted to resolve user issues; however, the lack of definition made troubleshooting almost impossible. No process existed for users to report issues on a consistent basis. Often times the resources were consumed with the development projects and support for the future releases, leaving limited resources to focus on the production environment. After the Release 5 in April of 2004, the application hit a wall. The user base had increased to over 1,200 and the team was receiving daily and hourly complaints from many users across the enterprise. Guiding Principles It was quickly determined that the resolution for Siebel application performance would not be an overnight solution. A comprehensive resolution plan was created with corrective actions from different angles. From a strategy perspective, a task force was created that included participation from Sabre, Siebel and other outsourcing partners. The task force met twice a week and eliminated all barriers to progress. The task force was governed by CXO-level leaders, who often got involved in the detailed discussion around performance and corrective actions. Key stakeholders/sponsors were identified to present the voice of the end user. Additionally, users were offered incentives to share their performance issues at an extremely low level of detail. The detail was necessary and the task force recognized the burden it would place on the users. A process was established to measure application performance and report to the task force and executive leadership on a daily basis how the application measured up. Every component of the technology framework was reviewed and a list of improvement recommendations created. User Desktop Environment After working with individual users regarding their performance problems, it was determined that the desktop was a contributing factor to the performance issues. At Sabre the users all receive a standard desktop configuration; however, the desktop is not locked down. Users had installed custom toolbars, pop up blockers, anti-spy ware utilities etc. all of which could impact Siebel. They were instructed to limit their personalization of Internet Explorer. Siebel Technical Support provided a desktop utility that when executed on the desktop would remove all traces of Active X controls and clean the registry. A desktop script was created that incorporated the utility from Siebel and additionally set all the Siebel URL s as trusted sites, enabled Active X control reference in order to download.jsp and Siebel IEOP files, installed and/or configured Java correctly, clean out the users temporary internet files and Siebel related cookies. To ensure consistent usage and application of this desktop script, it is pushed to all Siebel users upon start up of their computer. COLLABORATE 07 Copyright 2007 by Keri McClellan Page 2

Infrastructure Changes Upon reviewing the Siebel architecture, several changes were identified. The Siebel gateway server was moved to an application server that utilized a higher speed processor than it had previously been installed on. Due to the growing number of attachments, mostly due to the volume of emails being exchanged, supplemental storage needs were required. Connectivity to a storage array network (SAN) was implemented to fulfill this need. Over the course of time, the application servers had been installed, patched, configuration changes, etc. The only sure way to eliminate any doubts about the hardware and guarantee the accuracy of the server configuration was to perform a complete reinstall of the servers, from the operating system upwards. The web servers were installed in a DMZ environment that included a firewall between the web and application servers. They were moved out of the DMZ, eliminating the firewall and allowing the web and application servers to communicate across the same subnet. Overall, Sabre s network capacity was upgraded between headquarters and the data center located in Tulsa, OK., benefiting not only Siebel, but all applications. Hardware Upgrades The web servers and the database servers supporting the icrm environment were shared with other internal applications. Additional servers were acquired to allow for icrm to run on a dedicated set of servers. The new database server was an upgrade to an 8 CPU 32 MB RAM from a 4 CPU device. icrm did have dedicated application servers, but two new 4 CPU servers were added to the environment allowing for multi-threading capabilities. Application Changes Siebel conducted multiple expert services reviews. The findings indicated that the environment was only moderately customized; however, due to the depth and breadth of the application configuration, the environment itself was very complex. Well over 50 recommendations were made by Siebel. Regarding configuration, some of the correctives include: modify joins for custom foreign keys, set force active property to False, add primary fields for multi value links, always create a PickMap for fields with a defined PickList and set foreign keys, static PickLists should list the field defining the PickList, dynamic PickLists should list the fields where foreign keys must be updated in addition to any other fields that must be updated, correct fields that reference a non-existent join by removing them or specifying an appropriate join. Scripting changes/corrections were made to address the following issues: active debugging statements, use of active methods in server script, using NextRecord with DeleteRecord in loops, shared global values, non-indexed search and sort specifications, review routines that perform queries to ensure that they check for the existence of a valid record before performing any actions on the record(s) returned by the query. The most measurable improvement was the reduction in the number of object manager crashes. At the point when performance was at the worst, 15-20 object manager (OM) crashes were experienced per week. Within the environment, each crash would disconnect at least 20 users; therefore, the user community experienced to 300-400 disconnections per week. After the application changes were implemented, OM crashes decreased to 1-2 per week, i.e. 20 to 40 disconnections per week. One scripting change made the greatest improvement in decreasing the OM crashes. Custom code usage was executing any time an object (activity, service request, opportunity, contact, etc) was created, updated or deleted. A tremendous amount of effort was involved to remove the scripting, but the improvement was well worth the effort. The BMC Patrol suite was acquired in order to to bring better visibility. The Siebel Knowledge Module collects, processes and present metrics in dashboard-format GUI rather than scrubbing through log files. Additionally BMC Patrol performs monitoring and can perform recovery actions. COLLABORATE 07 Copyright 2007 by Keri McClellan Page 3

Application Upgrade Siebel strongly recommended that Sabre upgrade from Siebel 7.5.3 to 7.7 due to the improved performance and stability that 7.7 users were reporting. The elimination of Siebel s proprietary load balancer, Resonate, was considered to be an improvement because it had limited capabilities and very high implementation and maintenance complexity. The change to cost-based database optimization (CBO) from rules-based optimization (RBO) improved performance of the database; however, there were certainly growing pains associated with this change in database processing. Many of the day to day application queries that worked with RBO no longer executed with CBO. The application would function in regards to business processes, i.e. creating and updating records, but normal query functionality was greatly impacted. It took almost 3 months working very closely with sophisticated Siebel application dba resources to tune the most frequently run user queries. In parallel with the database optimization change, Oracle was upgraded from 8i to 9i. Query-Specific Changes One factor that was contributing to poor performance on user queries was the Automatic Trailing Wildcard indicator in the call center configuration file. The indicator was set to True causing all queries to default to a trailing wildcard, i.e. asterisk. Once this was changed to False user began seeing improvement to query performance. Many of the more common queries were developed into public queries that could be shared between user groups. This process allowed the team to identify the Where clause within the query variable and index the database columns accordingly. Although users had been trained on how to perform ad hoc queries, re-current training was necessary to minimize the number of poor performing queries being submitted. At the database level, the Oracle Profiler was leveraged to cancel queries that are running too long. It will limit the query time to 4 minutes. A recent enhancement has been the implementation of a Cancel Query button which the user can invoke should an inefficient query be submitted. Governance Changes The team established a Production Scorecard to visually describe the overall health and stability of the production environment. The scorecard is presented to sr. management recording the daily result on the performance metrics including the number of unplanned outages, number of OM crashes, CPU and memory utilization, network latency and number of user issues being reported. Production Support Changes The icrm team implemented Siebel for their own use. This allowed for better tracking of user issues, requests for enhancements and exchanges between the team and end users regarding performance concerns. As a result of leveraging Siebel for use within the team, service goals are managed more closely and the team is more responsive. Load and performance testing had previously been nonexistent and the performance activities. Mercury s Load Runner application was implemented for testing on all future releases. User feedback had indicated that some of the application functionality simply did not meet their usability needs. A Usability release was implemented the focused on improvements for two high-volume user groups. This release specifically addressed the requirements of the business process flow and the users were thrilled with the improved usability. Once a significant portion of the application performance issues had been address, Siebel was engaged to conduct an End User Survey Diagnostic (EUSD). This survey established a benchmark for measuring the icrm user feedback against other Siebel clients. Future surveys are planned as well. COLLABORATE 07 Copyright 2007 by Keri McClellan Page 4

Partnership with Siebel Throughout the performance improvement efforts, the Sabre account had a red status with Siebel. This categorization provided direct access to Siebel executive resources. It was many months before the status was downgraded. During this effort Siebel had several opportunities to demonstrate the Follow the Sun approach to support. It was an efficient process and the transition from desk to desk went smoothly. Onsite TAM support made it easy to demonstrate the production performance issues in real time. Siebel was able to engage expert services rapidly, i.e. within 24 hours, in many cases. Future Direction Even though tremendous progress has been made, there are still enhancements being planned. The email response capabilities should address some of the email usability issues that the users still encounter. Siebel v8 specifically includes some functionality around performance monitoring that Sabre will want to take advantage of. Sabre s customer service center is currently leveraging CTI integration between Avaya and Siebel, and there are additional efficiencies to be gained by implementing chat capabilities within the CTI layer. CTI diagnostics and reporting will also be enhanced. The database layer will be migrating to Unicode and Oracle 10g in order to support the multi byte character languages. COLLABORATE 07 Copyright 2007 by Keri McClellan Page 5