TUT5860 Troubleshooting and Optimizing NetIQ Access Manager

Transcription

1 TUT5860 Troubleshooting and Optimizing NetIQ Access Manager #BrainShare #NetIQ5860

2 Agenda General Networking troubleshooting tools Access Manager troubleshooting tools Access Manager protected resource flow Access Manager log settings and log files Case study Additional reading 2

3 Networking Tools Ethtool (-S, -K TSO) netstat -patune connection and stat info tcpdump/wireshark/tshark (SSL private key) netcat general ip/icmp/tcp/udp stats under /proc/net/snmp ipsysctl TCP settings Network layout (firewall blocking data, SSL terminators; Load Balancers redirecting ports; masquerading) 3

4 Generic NetIQ Access Manager Troubleshooting Tools (cont.) Certificates and keystores openssl s_client -connect idpcluster.lab.novell.com:8443 CONNECTED( ) depth=1 /OU=Organizational CA/O=linuxlab5_tree verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/cn=idpcluster.lab.novell.com i:/ou=organizational CA/O=linuxlab5_tree 1 s:/ou=organizational CA/O=linuxlab5_tree i:/ou=organizational CA/O=linuxlab5_tree keytool -list -keystore /var/opt/novell/novlwww/devman.keystore -v Your keystore contains 1 entry Alias name: tomcatcreation date: 13-Dec-2013 Entry type: keyentrycertificate chain length: 2 Certificate[1]:Owner: O=novell, OU=accessManager, CN=linuxlab5 Issuer: O=linuxlab5_tree, OU=Organizational CA :Certificate[2]: Owner: O=linuxlab5_tree, OU=Organizational CA Issuer: O=linuxlab5_tree, OU=Organizational CA : 4

5 Generic NetIQ Access Manager Troubleshooting Tools (cont.) HTTP Request generator Curl ( HTTPRequester FF plugin D71B8B5632BC BD0D 1FAAB4AD8 6

6 Generic NetIQ Access Manager Troubleshooting Tools (cont.) IDP config 'Logging' TAB configuration 7

7 Generic NetIQ Access Manager Troubleshooting Tools (cont.) AC general logs from 'Auditing' TAB 8

8 Generic NetIQ Access Manager Troubleshooting Tools (cont.) Performance analysis tools on dependencies (LDAP performance on edirectory) HTTP common or extended logs (Web server performance) X-MAG FP4 header timestamp (DebugHeaders on) 9

9 Generic NetIQ Access Manager Troubleshooting Tools (cont.) Statistic logging (Auditing Device Health Device Statistics) 11

10 Access Gateway Overview Identity Server Identity Store User Accesses protected resource 2. User is redirected to Identity Server and is presented with an http login form requesting their username and password 3. The Identity Server verifies the username and password against the Identity Store 4. Once the user's identity is validated, the Access Gateway retrieves the user's common name and password 5. The Access Gateway injects the username and password into the authentication header and allows access to the encrypted Web content Access Gateway Apache or IIS web server configured to accept header-based authentication 13

11 Access Manager Advanced Overview Existing Session with Web Single-Sign-On and Access Gateway Cluster Assume User already had active session on AG2. Identity Server 5, 57 6 Identity Store 1. User Accesses protected resource on AG1 and the browser presents AG1 a valid Access Gateway session cookie created earlier by AG2 for this user session. 2. AG1 doesn't have a session for the user so it asks the other AGs in the Group to see if they have a session for the user 3. AG2 responds claiming ownership for the user session AG2 4. AG1 asks AG2 for the policy and user data required for the user to access the protected resource 2, 2, 3, 2, 3, 24, AG2 requests policy and user data from the Identity Server (if it isn't cached) 6. The Identity Server gets the user data from the Identity Store (if it isn't cached) Web Browser 1 AG1 9 Access Gateway Group (Load Balanced by L4 Switch) Web Servers 7. Identity Server responds to AG2 with the policy and user data 8. AG2 responds to AG1 with the policy and user data Assume authentication headers used for SSO to origin web servers 9. AG1 processes the policy and user data and allows access to the protected resource 14

12 AG Architecture 17

13 AG Tools Advanced Logging Options Advanced options Adds custom NAM level logging to error_log for each request /var/log/novell-apache2/error_log Can directly modify httpd.conf with these lines at the end and restart /etc/opt/novell/apache2/conf/ directory 18

14 AG Tools : X-MAG Headers Short descriptions about the processing path. 20

15 AG Tools - Server Status (use w3m) Gives web based real time statistics (load, CPU, connections, balancer) from httpd & AG module State (waiting, writing, keepalive, closing, etc) and number of free (idle) network slots that can serve Server generation Uptime Traffic data 21

16 AG Troubleshooting Logs /var/log/novell-apache2/rcnovell-apache2.out Apache startup messages (N/A for Windows) /var/opt/novell/amlogging/logs/ags_error.log NAM specific Apache startup messages and configuration updates /var/log/novell-apache2/access_log or extended_log Uses CommonLog or ExtendedLog module from apache 22

17 AG Logs - error_log /var/log/novell-apache2/error_log Httpd logs for GET/Response traffic from browsers here. Most of the logs will be here. General apache errors so can Google for generic issues Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#: AMEVENTID#994: Requ: GET service:nam32vm-pxy-srvc ( :2551-> :443) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994: validatecookie:local user. Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994: Restricted URL Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994: matched PR:rewriter-pr Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994: Contract-valid contract(secure/name/password/uri - >secure/name/password/uri) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994: balancer cookie is ZNPCQ =a1b14cc2; Path=/; Domain=.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] proxy: HTTP: fam 2 socket created to connect to Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] proxy: HTTP: connection complete to :80 ( ) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: connected from :46079 to :80 Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: sending request to webserver Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: received response from server Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: received status 200 from server Nov 14 19:35:19 mag32app-vm httpd[31648]: [warn] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F FF0: AMEVENTID#994:status:200 GET < fb0aeb56e9b4636e881ca59cdc4> X-Mag: <45B6586EB94FC2A7;ca59cdc4;994;usrLkup- >0;usrBase->0;LocUsr;rewriter-pr;Contract-valid->0;aclEvalTout->0;EvalACL->11;Allow->11;aud->11;nam32vm-pxy-srvc;default;SH;FP2- >11;WS=a1b14cc2;default;$custom_urs-ff-rewriter;FP4->17;C005;> [ :2551-> :443]service:nam32vm-pxy-srvc (185:3) - 23

18 AG Logs - httpheaders /var/log/novell-apache2/httpheaders HTTP headers output from browser <-> Proxy and Proxy <-> Web server Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from client for id 995:185: Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Host: nam32app-vm.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Connection: Keep-Alive Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Cookie: IPCZQX03bafce9af= fb0aeb56e9b4636e881ca59cdc4; ZNPCQ =a1b14cc2 Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers to webserver for id 995:185: Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: GET /neil/phpinfo.png HTTP/1.1 Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: Host: ncsles11ws.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: If-None-Match: "a e-38c234eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: If-Modified-Since: Wed, 22 Aug :23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: Via: 1.1 nam32app-vm.lab.novell.com (Access Gateway-ag- 45B6586EB94FC2A7-995) Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-For: Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-Host: ncsles11ws.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-Server: nam32app-vm.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Server: Apache/2.2.3 (Linux/SUSE) Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Last-Modified: Wed, 22 Aug :23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:ETag: "1aed0-12e-4eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Accept-Ranges: bytes Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Content-Length: 302 Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Content-Type: image/x-icon Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Last-Modified: Wed, 22 Aug :23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: ETag: "1aed0-12e-4eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Accept-Ranges: bytes Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Content-Length: 302 Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Content-Type: image/x-icon Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: X-Mag: 45B6586EB94FC2A7;ca59cdc4;995;usrLkup- >0;usrBase->0;LocUsr;_public_;publicURL->0;nam32vm-pxy-srvc;default;SH;FP2->35;WS=a1b14cc2;FP4->37; Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Via: 1.1 nam32app-vm.lab.novell.com (Access Gateway-ag- 45B6586EB94FC2A7-995) 24

19 AG Logs - soapmessages /var/log/novell-apache2/soapmessages Logs AG <-> ESP SOAP traffic (authentication and policy evaluation) Nov 14 19:35:11 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV=" verb="add" correlationid="990"><addentry seconds="130" owner="true" key=" fb0aeb56e9b4636e881ca59cdc4"><sessiondata UserRole="k9" UserName="public" pid="0" SPSessionID=""><contracts/></SessionData></AddEntry></CookieBrokerRequest></SOAP-ENV:Body></SOAP- ENV:Envelope> Nov 14 19:35:11 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV=" correlationid="990"><addentryresponse key=" fb0aeb56e9b4636e881ca59cdc4" status="ok"/></cookiebrokerresponse></soap-env:body></soap-env:envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <ns1:envelope><ns1:body><nidpsetsession softexpire="194" refreshcache="false" pid="kpo-ps)znt.qz6kpzeocrgiv]" id="3a2eda43d25c3d43620d2f ff0" hardexpire="299" XLibid=" fb0aeb56e9b4636e881ca59cdc4"><store type="ldap"><dn>cn=ncashell,o=novell</dn></store><authentications><contracts><contractset="true">secure/name/password/uri</ contract></contracts></authentications><roles><role>secnamepwdauthrole</role><role>authenticated</role></roles></nidpset Session></ns1:Body></ns1:Envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV=" Id="994"><Evaluate Verbose="on" PolicyId="K37M6K86-M788-1NPP-15PK-9069K "><ContextDataElement Value="3A2EDA43D25C3D43620D2F FF0" Enum="2551"/></Evaluate></NXPES></SOAP-ENV:Body></SOAP- ENV:Envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV=" Id="994" Status="success"><EvaluateResponse><DoAction ActionName="Permit" ActionTTL="-1" Enum="2610"/></EvaluateResponse></NXPES></SOAP-ENV:Body></SOAP-ENV:Envelope> 25

20 AG/ESP Logs - Catalina /var/opt/novell/nam/logs/idp(nesp)/tomcat/catalina.out esp logs for communication with proxy and IDP esp inherits IDP logging settings ('Application, Liberty, Web Service Provider/Consumer) Used to troubleshoot import, authentication and policy issues Can search for JSESSIONID, Policy ID or threadid (Processor string) Display IDP/ESP statistics Performance issues running out of threads (maxthreads, Xmx, LDAPLoadThreshold, Attribute queries) 26

21 Troubleshooting Slow Performance HTTPWatch output with timestamps LAN traces with private keys Check whether ESP is slow (from server-status or enable access log in server.xml) Check whether backend is slow Enable extended logs and check response time logged X-mag header FP4 gives the response time for each request. 27

22 Troubleshooting Services Issues Look at server-status output (w3m > w3m-mag-status.out) Accelerator: nam32vm-pxy-srvc Accelerator Type: Https Listener HostName: nam32app-vm.lab.novell.com Listen: :443 AltHost: ncsles11ws.lab.novell.com CookieDomain:.lab.novell.com Reverse Proxy bal_nam32vm-pxy-srvc SSes Timeout Method ZNPCQ byrequests Sch Host Stat Route Redir F Set Acc Wr Rd http Ok a1b14cc k 31k Reverse Proxy bal_owa-pbmh SSes Timeout Method ZNPCQ byrequests Sch Host Stat Route Redir F Set Acc Wr Rd https Ok 0b45d31b k 9.6k Look at 'netstat -patune grep -i listen' output Check all listeners active for IP address and Port combinations 28

23 Troubleshooting SSL Issues Listener not getting created on 443 Certificates not present check if required certs are present in /etc/opt/novell/apache2/conf/certs/ search for certificates in /var/log/novell-ag-logs/novell-apache2/error_log to see the reason for failure (LogLevel debug!) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Generating temporary RSA private keys (512/1024 bits) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Generating temporary DH parameters (512/1024 bits) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Initializing (virtual) servers for SSL Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC4366) Jcc timing issue with certificate command and reinitialize try restarting proxy and jcc with /etc/init.d/novell-apache2/novell-jcc restart 29

24 Troubleshooting SOAP Channel Issues Used to view all data on backchannel TCPDUMP to trace the loopback interface Soap Requests to :9009 tcpdump -t -p -s 0 -i lo 'tcp port 9009' -w soapch.cap to log all soap policy requests between proxy and ESP Unsolicited Response from ESP ESP signals end of authentication login/logouts through these requests tcpdump -t -p -s 0 -i lo 'tcp port 8181' -w unsolicited.cap 'DumpSoapMessages on' Advanced Option can view Identity Injection, formfill and Authorization policy interactions in logs 30

25 Troubleshooting SOAP Channel 31

26 Troubleshooting SOAP Channel Issues CATALINA.OUT output Enable IDP logging for Liberty component as well as content filters for Liberty tail -f /var/log/novell-ag-logs/maglogs/catalina.out <SOAP-ENV:Envelope xmlns:soap-env=" <SOAP-ENV:Body> <CookieBrokerRequest verb="add" correlationid="560"> <AddEntry key=" c8946e8826de f94bf03f6dbb058" owner="true" seconds="571"> <SessionData UserName="cn=ncashell,o=novell" UserRole="" AuthStatus="1" HardTimeout="16038" SPSessionID="80B294FDCEE4ED69792F78C712DC374F"SoftTimeout="10422" CreationTime="1333" ContractNameKey=" "/> </AddEntry> </CookieBrokerRequest> </SOAP-ENV:Body> </SOAP-ENV:Envelope> AJP connector configuration Increase number of threads 32

27 Troubleshooting Connection Issues Verify listener is active netstat -patune grep -i listen verify all listeners are there for the configured TCP ports Check error_log log file for 'Sockets' component Nov 21 15:35:28 mag32app-vm httpd[20855]: [info] proxy: HTTP: fam 2 socket created to connect to Nov 21 15:35:28 mag32app-vm httpd[20855]: [info] proxy: HTTP: connection complete to :80 ( ) Check LAN trace for communication issues (retransmits and delays) Check /var/log/messages for NIC specific errors Check ip statistics using netstat output for icmp errors and tcp/ip stats 33

28 Troubleshooting Rewriter Issues HTTPWatch output with timestamps or Fiddler (free) View source going direct and through AG Compare the differences (search scheme, internal info, ctype) Advanced Options available NAGDisableExternalRewrite X-mag header FP4 gives the rewriter profile executed May need additional entries in rewriter policy 34

29 Troubleshooting Authorization Issues Confirm X-MAG headers and Via eventid Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: matched PR:rewriter-pr Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL eval sending Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Sending value 23B586CA4B1E942464CA9169CF4A88F2 for enum LibertyID Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL policy id:65kmm806-84k3-m4lm-nm71-l036l9o07415 Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: subreq nam32app-vm.lab.novell.com:/nesp/app/soap Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Received ACL Eval Permit Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL eval success Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Allow Nov 13 18:11:36 mag32app-vm httpd[23538]: [warn] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: status:200 GET < ca61522f67560acf117d2ca59cdc4> X-Mag: <45B6586EB94FC2A7;ca59cdc4;6;usrLkup->0;usrBase->0;LocUsr;rewriter-pr;Contract-valid- >0;aclEvalTout->0;EvalACL->14;Allow->14;aud->14;FPE->14;> [ :4429-> :80]service:nam32vm-pxy-srvc (10:0) ESP evaluation <amlogentry> T18:11:36Z INFO NIDS Application: AM# : AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: PolicyID#65KMM806-84K3-M4LM-NM71-L036L9O07415: NXPESID#6: AGAuthorization Policy Trace: ~~RL~1~~~~Rule Count: 2~~Success(0) ~~RU~RuleID_ ~Authz-SecNamPwdNotManagers-pol~DNF~~1:1~~Success(0) ~~CS~1~~ANDs~~2~~True(69) ~~PA~1~~Permit Access~~~~Success(0) ~~PC~1~~Document=(ou=xpemlPEP,ou=mastercdn,ou=ContentPublisherContainer,ou=Partition,ou=PartitionsContainer,ou=VCDN_Root,ou=accessManagerCont ainer,o=novell:romacontentcollectionxmldoc),policy=(authz-secnampwdnotmanagerspol),rule=(1::ruleid_ ),action=(permit::1)~~~~success(0) </amlogentry> 35

30 Troubleshooting Identity Injection Confirm X-MAG headers and Via eventid [Wed Apr 11 18:03: ] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D E18794C231F29E: AMEVENTID#7: status:200 GET < c859be a2eca59cdc4> X-Mag: 45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;ConfigII->120;configACL->186;NoPol;ConfigFF->251;formfill-pr;Contract-valid->251;usrPr- >252;Allow->252;aud->252;nam32vm-pxy-srvc;EvalII->296;CHd;AH;QS;FP2->297;WS=a1b14cc2;default;FP4->305;C005; Confirm HTTP headers sent (error_log or httpheaders) Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: GET /formfill/phpinfo.php?x-roles=authenticated HTTP/1.1 Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Host: ncsles10.lab.novell.com Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/ : Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-client-IP: Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Auth-Cont: name/password/uri Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-mail: [email protected] Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Authorization: Basic bmnhc2hlbgw6bm92zwxs Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-For: Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Host: ncsles10.lab.novell.com Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Server: nam32app-vm.lab.novell.com Confirm ESP evaluation <amlogentry> T17:03:39Z DEBUG NIDS Application: AM# : AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D E18794C231F29E: PolicyID#N885856P-48PP-9NN7-1K15-N8O6NOP040L2: NXPESID#7: AGIdentityInjection Policy Trace: ~~RL~1~~~~Rule Count: 4~~Success(67) ~~RU~RuleID_ ~Identity-Inj-All-Pol~DNF~~0:1~~Success(67) ~~PA~ActionID_ ~~Inject Auth Header~uid~uid(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A ~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDA PCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D:~Ok:ttl -1~Success(0) ~~PA~ActionID_ ~~Inject AuthHeader~password~pwd(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A ~2Fcp~3ASecrets~ 2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPredentials~2 2~5D~2Fcp~3AEntry~5Bcp~3AName ~3D~22UserPassword~22~5D:~Ok~Success(0) 36

31 Troubleshooting Formfill Issues STRACE output to look at the form details Could get LAN traces with private keys Confirm policy evaluated in the catalina log file Search for 'AGFormFill Policy Trace' or 'NXPESID#EventIDNumber' Enable 'NAGGlobalOptions DebugFormFill=on' Advanced Option X-mag header FP4 gives the response time for each request [Wed Apr 11 17:43: ] AM# AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D E18794C231F29E: AMEVENTID#7: status:200 GET < c859be a2eca59cdc4> X- Mag:45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;NoPol;ConfigII->116;NoPol;configACL->174;ConfigFF- >236;EvalFF->247;formfill-bootcamp-pr;Contract-valid->247;mastercdnFormfill-Pol-bootcamp3310;FF4GUD- >267;FillSilent;Match FormName;Match;username;Miss;title;Match;password;Miss;ldap;FF4End->267;FP4->267; Enable NAGGlobalOptions InPlaceSilent 37

32 Troubleshooting Case Study: Single Sign-On to Back-End App Fails with Formfill

33 Policy Case Study Background Customer enabled a Formfill policy to apply to a single protected resource (/Citrix/XenApp/auth/login.aspx) with login page populating: username and password into the form using LDAP credentials Formfill policy javascript, auto-submit and mask data options enabled enabled After user logs in and accesses Access Gateway protected resource, the user could not SSO to the back-end Web App authentication failed, and an internal error messages was returned from the back-end application The Web site is experiencing technical difficulties. We apologize for any inconvenience. The internal error may only be temporary. Try reconnecting and, if the problem persists, contact your system administrator 39

34 Policy Case Study Troubleshooting Get policy and where policy applied (get screenshot) of protected resources and export of policy) 40

35 Policy Case Study Troubleshooting Enable logs for policies and Proxy and gather all the key logs (from cheatsheet) and LAN trace after duplicating issue Enable logs for policies and Proxy Must understand where in the policy flow the request is failing (Web server, Proxy server, esp, IDP, user store)? 41

36 Policy Case Study Proxy Log Analysis Check browser HTTP trace cookies for form and X-MAG header to verify FF triggered 45B6586EB94FC2A7;ca59cdc4;461;usrLkup->0;usrBase->0;LocUsr;getPRBefFind->0;PRAfterFind->1;ConfigFF- >228;EvalFF->240;Citrix-loginpage-ff-pr;Contract-valid->241;nam32vm-pxy-srvc;HdrRRwNo;FF1End->241;FP2- >241;WS=a1b14cc2;default;$custom_citrix-ff-rewriter;setupFF-interested;mastercdncitrix-ff-pol3310;FF4GUD- >638;FillSilent;Match FormName;Miss;SESSION_TOKEN;Miss;LoginType;Match;user;Match;password;Match;tree;FP4- >644; 42

37 Policy Case Study Proxy Log Analysis Check browser HTTP trace to see if credentials POSTed they are but error back 43

38 Policy Case Study Proxy Log Analysis Check what AG POSTs to Web server need tcpdump output 44

39 Policy Case Study Solution Confirmed that FF policy triggered from X-MAG header Confirmed credentials POSTed to AG from browser and to Web server Verified that Web server failed to validate credentials Disabled mask data option worked Content-length sent by AG to Web Server not adjusted correctly Bug on NAM side 45

40 Additional Reading Best Practices Guide data/bookinfo.html Avoiding memory and performance issues with Java Troubleshooting /43 errors and errors-access-manager Troubleshooting SAML Performance and Sizing Guidelines White Paper ormance_sizing/performance_sizing.pdf 46

41 Don t miss the Identity-Powered Experience in IT Central. Thank you NetIQ Corporation. All rights reserved.

42

43 This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time. Copyright ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.