1
Sticky Session Setup and Troubleshooting Day, Date, 2004 time p.m. ET Teleconference Access: US & Canada: 888-259-4812 Teleconference Access: North America: xxxx Toll Number: 706-679-4880 International: xxxx International Password: Toll-Free: Advisor See DOC#1148600.1 Password: advisorpg Current Advisor Webcast Schedule DOC#740966.1 Archived Advisor Webcasts DOC#740964.1 Upcoming live webcasts and recent recordings: July 12, 2011: Customizing the Workflows in Oracle EBS Procurement July 12, 2011: PeopleTools: True Excel Templates for Oracle BI Publisher July 12, 2011: Making the EBS Upgrade from 11.5.10 Easier Part II July 13, 2011: Manufacturing: An Introduction: JDE- VCP Integration July 13, 2011: JD Edwards World: Working with Italian Withholding Vouchers July 13, 2011: Oracle EM JDE Application Pack for EnterpriseOne July 13, 2011: HCM: Plan Design Maintenance for Open Enrollment 2
Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 3
<Insert Picture Here> Sticky Session Setup and Troubleshooting Bill Holt Senior Technical Support Engineer Tim Bower Senior Principal Coe Engineer
While this presentation is intended as a guide to PeopleSoft customers regarding best practices for setting up and troubleshooting session affinity, it does lie outside the scope of GSC support. Contacting your load balancer vendor will still be the recommendation. 5
Program Agenda Our understanding of PIA Architecture Sticky Sessions what are they? Sticky session setup Oracle recommendations/pros and cons Symptoms of improper sticky session setup and how to troubleshoot Sticky Session Case Study Q & A <Insert Picture Here> 6
The Big Picture Application Server(s) Web Client http(s) Web Server Cluster Jolt Tuxedo Application Database Windows Client Process Scheduler 7
Web Server (WEB) Performs serialization and de-serialization. Puts together the presentation to be displayed on the browser. Maintains within various structures in Java heap memory and the user's browser. User cookies - PSToken Cookies. PeopleSoft web applications are java servlet-based and execute within in a Java heap. There is a practical limit on the size of a java heap. Java is horizontally scalable Scaling the web tier across multiple web servers requires some form of web server clustering 1. HTTP server (RPS) with proxy plugin. OR - 2. Load balancer. OR - 3. Some combination of 1 and 2 e.g. a load balancer routing requests to multiple HTTP servers running proxy plugins to multiple web instances. 8
Session Sticky What Is it? In a cluster, once a session is established to a PSFT web server through the load balancer, every request made for that user session needs to go to the same web instance. To accomplish this, load balancers or proxy servers need to be set up with session sticky or session affinity. Also known as session affinity or session persistence 9
Sticky Session Setup Methods Pros and Cons Method Description Pros Cons Source IP based Sticky rule Compares the requestors IP address to an IP address table: adds it if it does not exist, if one exists, routes subsequent requests to the the same web server for that IP address. IP Address table is a finite size and IP addressed can be dropped off even if they are active. IP addresses are not always unique to a workstation (outside firewall; AOL for example). This can create an uneven load. Issues with logoff commands crossing applications. NOT recommended for use with PeopleSoft SSL ID Similar to source IP method except the load balancer routes based on a unique SSL session ID to maintain the sticky rules between the Server and Client. Can be used in instances where SSL connections are maintained end-to-end, i.e. browser to load balancer to PIA Unpredictable how long the SSL ID will remain (for example, one version of IE expires every 30 minutes). Logoff issues. NOT recommended for use with PeopleSoft Active Cookie Insert Rule Load balancer issues its own cookie for routing decisions (for PeopleSoft cookies). No issues with IP address table size or unique IP addresses. No issues with proxied requests (e.g logout commands) This is the recommended method for use with PeopleSoft Problem with running SSL as the cookies are coming through encrypted. However, SSL Acceleration or Proxy is a feature that can be implemented when we need to run SSL connections from user's browsers. This is implemented in An expansion card that encrypts and decrypts SSL. Session Replication BEA - replicate all session information across all webservers. Lost capacity. Network pipes between servers must be really good. This is currently unsupported, as no testing has been done in PeopleTools 8.49 or 8.50. 10
Configuring Peoplesoft for Session Stickiness Web Profile Sticky Setup 11
Configuring Peoplesoft for Session Stickiness WebLogic Cookie Setup in weblogic.xml File 12
For More Information E-PIA: PeopleSoft and Load Balancers (Doc ID 653998.1) NOTE: The configuration settings outlined in this document are NOT suggestions, they are REQUIREMENTS 13
Symptoms of PIA Session Sticky Problems Users try to save a component and end up on a search page Users get a signon screen when trying to look at reports Left-hand navigation on PIA page displays properly, but content area shows "Page Cannot Be Displayed" "You must have cookies enabled" errors 14
To Determine if Session Stickiness May Be a Problem 1. Test without load balancer If the problem does not appear without the load balancer, the reported problem is likely caused by the load balancer 2. Test with one webserver running behind load balancer. This is a good test, but results may be unreliable since intermittent problem may still appear with just one webserver because various load balancer configuration options may still point at the load balancer 3. Collect browser headers while replicating problem iehttpheaders, LIVE HTTP HEADERS, Fiddler, and etc. If you see that the PORTAL-PSJESSIONID webserver cookie value is randomly changing, your load balancer is not maintaining session stickiness. 4. Turn on WebLogic extended access logging in both webservers behind the load balancer For instructions on turning on extended access logging, please see Document 644602.1. This is the most time consuming option and may impact performance in production, but it may provide the most valuable information. The extended logging collects all cookies in the same domain that are forwarded from browser to webserver for all end users logging into PeopleSoft. These include PeopleSoft cookies, WebLogic cookies, 3rd party cookies, and any custom cookies. 15
For More Information E-PIA: How To Identify Session Stickiness Problem On Load Balancer? (Doc ID 1307344.1) My Oracle Support PeopleTools Community https://communities.oracle.com/portal/server.pt/commu nity/peopletools_-_psft/476 16
Sticky Session Troubleshooting Case Study A walkthrough of a simple case study Using a simple web cluster of two WebLogic instances with a reverse proxy server Loss of session sticky is forced by shutting down a WebLogic instance while a user is navigating the tabs on a PeopleSoft component. 17
The Test Architecture PIA1 Host: pcoe-vm03.us.oracle.com WebLogic Port: 8000 Extended HTTP logging configured as per KM Doc ID 649428.1 PIA2 Host: pcoe-vm03.us.oracle.com WebLogic Port: 8080 Extended HTTP logging configured as per KM Doc ID 649428.1 18
The Test Architecture - continued Reverse Proxy Host: pcoe-vm14.us.oracle.com Apache HTTP Port: 80 Debug logging configured as per KM Doc ID 1272310.1 Browser Firefox 5.0, capturing header information using the Mozilla Live HTTP Headers plugin. https://addons.mozilla.org/en-us/firefox/addon/live-http-headers/ 19
Sticky Session Walkthrough - 1 When we first login to the PeopleSoft system, this is what we see on the browser: 20
Sticky Session Walkthrough - 2 On the PIA1 server, we can see that a CRM- PSJESSSIONID cookie has been set 2011-07-06 21:38:01 10.147.112.88 GET 0.372 200 8579 /psp/ps/employee/crm/h/?tab=default.. SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!- 21
Sticky Session Walkthrough - 3 From Live HTTP Headers on the browser, we can see the CRM_PSJSESSIONID cookie being set. POST /psp/ps/?&cmd=login&languagecd=eng HTTP/1.1 Host: pcoe-vm14.us.oracle.com User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0 HTTP/1.1 302 Moved Temporarily Date: Thu, 07 Jul 2011 00:36:58 GMT Server: Apache/2.2.3 (Oracle) Location: http://pcoe-vm14.us.oracle.com/psp/ps/employee/crm/h/?tab=default Expires: Thu, 01 Dec 1994 16:00:00 GMT Set-Cookie: CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; 22
Sticky Session Walkthrough - 4 After signing on, we navigate to the search page for the WEB_PROFILE component, enter a value and click the Search button 23
Sticky Session Walkthrough - 5 On PIA1, we can see the POST with the value of CRM- PSJSESSIONID unchanged 2011-07-06 21:38:27 10.147.112.88 POST 0.052 200 13339 /psc/ps/employee/crm/c/web_profile.web_profile.gbl. CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; 24
Sticky Session Walkthrough - 5 Same POST from the browser. The value for CRM-PSJSESSIONID doesn t change. http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl POST /psc/ps/employee/crm/c/web_profile.web_profile.gbl HTTP/1.1 Host: pcoe-vm14.us.oracle.com.. CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks. HTTP/1.1 200 OK Date: Thu, 07 Jul 2011 00:37:25 GMT Server: Apache/2.2.3 (Oracle) Content-Length: 13339 Expires: Thu, 01 Dec 1994 16:00:00 GMT IgnorePortalRegisteredURL: 1 PortalRegisteredURL: http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl UsesPortalRelativeURL: true Content-Encoding: gzip X-Powered-By: Servlet/2.5 JSP/2.1 Set-Cookie: PS_TOKENEXPIRE=7_Jul_2011_01:38:27_GMT; domain=.us.oracle.com; path=/ Connection: close Content-Type: text/html; CHARSET=UTF-8 25
Sticky Session Walkthrough - 5 Once the WEB_PROFILE component comes up, we click on the Virtual Addressing tab and see this page. 26
Sticky Session Walkthrough - 6 On PIA1, we can see the POST, and see that the CRM-PSJSESSIONID cookie remains unchanged: 2011-07-06 21:38:27 10.147.112.88 POST 0.0040 200 13339 /psc/ps/employee/crm/c/web_profile.web_profile.gbl "s_nr=1309192578226;.. SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; ExpirePage=http://pcoe-vm14.us.oracle.com/psp/ps/; PS_LOGINLIST=http://pcoe-vm14.us.oracle.com/ps; PS_TOKENEXPIRE=7_Jul_2011_01:38:27_GMT; PS_TOKEN=nAAAAAQDAgEBAAAAvAIAAAAAAAAsAAAABABTaGRyAk4AZQg4AC4AMQAwABRxi9kLgMBtsBnrox/+9CXbtuJX2VwA AAAFAFNkYXRhUHicS2VgYGBhZmJkBNJ7mBjAgCWAIZiBzZXBj8GdgQ/EdmMIYYhncGYIYjAxYjBgMARCXSBtDibNoKQeUNQISBqD2 WZAtiFQ3BiIGQAEDQop; http%3a%2f%2fpcoevm14.us.oracle.com%2fpsp%2fps%2femployee%2fcrm%2frefresh=list:%7c%7c%7c%7c%7c ; HPTabName=DEFAULT" 27
Sticky Session Walkthrough - 7 On the browser, we can see the same POST http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl POST /psc/ps/employee/crm/c/web_profile.web_profile.gbl HTTP/1.1 Host: pcoe-vm14.us.oracle.com SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; HTTP/1.1 200 OK Date: Thu, 07 Jul 2011 00:37:25 GMT Server: Apache/2.2.3 (Oracle) Content-Length: 13339 Content-Encoding: gzip Set-Cookie: PS_TOKENEXPIRE=7_Jul_2011_01:38:27_GMT; domain=.us.oracle.com; path=/ X-Powered-By: Servlet/2.5 JSP/2.1 Connection: close Content-Type: text/html; CHARSET=UTF-8 28
Sticky Session Walkthrough - 7 Shutdown the PIA1 web instance and click on another tab 29
Sticky Session Walkthrough - 8 But instead of what we expect, we get a search page 30
Sticky Session Walkthrough - 10 There s no more logging for PIA1 since it s down, but the PIA2 log shows a POST with our old value for CRM-PSJSESSIONID 2011-07-06 21:39:38 10.147.112.88 POST 1.321 200 6643 /psc/ps/employee/crm/c/web_profile.web_profile.gbl "s_nr=1309192578226; SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; 31
Sticky Session Walkthrough - 10 In the browser s log we see two POSTs. First one with the original value for CRM-PSJSESSIONID http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl POST /psc/ps/employee/crm/c/web_profile.web_profile.gbl HTTP/1.1 SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; HTTP/1.1 200 OK Date: Thu, 07 Jul 2011 00:37:28 GMT Server: Apache/2.2.3 (Oracle) Content-Length: 11716 Expires: Thu, 01 Dec 1994 16:00:00 GMT IgnorePortalRegisteredURL: 1 PortalRegisteredURL: http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl UsesPortalRelativeURL: true Content-Encoding: gzip X-Powered-By: Servlet/2.5 JSP/2.1 Set-Cookie: PS_TOKENEXPIRE=7_Jul_2011_01:38:29_GMT; domain=.us.oracle.com; path=/ Connection: close Content-Type: text/html; CHARSET=UTF-8 32
Sticky Session Walkthrough - 11 And a second POST that sets a new value for CRM-PSJSESSIONID http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl POST /psc/ps/employee/crm/c/web_profile.web_profile.gbl HTTP/1.1... SignOnDefault=PS; CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678; HTTP/1.1 200 OK Date: Thu, 07 Jul 2011 00:38:35 GMT Server: Apache/2.2.3 (Oracle) Content-Length: 6643 Expires: Thu, 01 Dec 1994 16:00:00 GMT IgnorePortalRegisteredURL: 1 PortalRegisteredURL: http://pcoe-vm14.us.oracle.com/psc/ps/employee/crm/c/web_profile.web_profile.gbl UsesPortalRelativeURL: true Content-Encoding: gzip X-Powered-By: Servlet/2.5 JSP/2.1 Set-Cookie: CRM-PSJSESSIONID=S22ZTVTZhQBSqqk811vXNWphJCFm2Gn2!832911604; domain=.us.oracle.com; path=/; HttpOnly Set-Cookie: PS_TOKENEXPIRE=7_Jul_2011_01:39:37_GMT; domain=.us.oracle.com; path=/ Set-Cookie: SignOnDefault=PS; domain=.us.oracle.com; expires=thursday, 14-Jul-2011 01:39:38 GMT; path=/ Connection: close Content-Type: text/html; CHARSET=UTF-8 33
Sticky Session Walkthrough - 12 PIA_servlets0.log.0 on PIA2 provides another clue 7/6/11 9:39:37 PM EDT 1-2145221434 unknown SEVERE psft.pt8.jb.icservice serializerequest..... Number of tools' cookies 4 CRM-PSJSESSIONID: lhpftvny92vsmp8gwvhhhk8ryhxdjgks. Request-ClusterInfo: true Total Number of header entries - exclude Cookie header: 15 state params in request: ICStateNum: 3 ICElementNum: 0 no states available on a post request 34
Sticky Session Walkthrough - 13 The Apache WebLogic Plugin Log shows exactly what happened Wed Jul 6 20:38:35 2011 <1694130999911512> ================New Request: [POST /psc/ps/employee/crm/c/web_profile.web_profile.gbl HTTP/1.1] =================.. Wed Jul 6 20:38:35 2011 <1694130999911512> Found cookie from cookie header: CRM- PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678 Wed Jul 6 20:38:35 2011 <1694130999911512> Parsing cookie CRM-PSJSESSIONID=lHpfTVNY92vSmp8GWvhhhK8ryhxdJGks!-701468678.... Wed Jul 6 20:38:35 2011 <1694130999911512> Connect failed: 111 Wed Jul 6 20:38:35 2011 <1694130999911512> *******Exception type [NO_RESOURCES] (apr_socket_connect call failed ) raised at line 1918 of../nsapi/url.cpp.... Wed Jul 6 20:38:35 2011 <1694130999911512> Remote Host 10.147.112.75 Remote Port 8080 35
For More Information E-PIA: Troubleshooting and Setup Recommendations Session Sticky/Load Balancing Issues (Doc ID 1312563.1) 36
Bottom Line Make sure ALL configurations are applied as recommended in E-PIA: PeopleSoft and Load Balancers (Doc ID 653998.1) The ONLY recommended method for setting up load balancer session affinity is Active Cookie Insert Rule 37
38
THANK YOU 39
40