TUT5860 Troubleshooting and Optimizing NetIQ Access Manager #BrainShare #NetIQ5860
Agenda General Networking troubleshooting tools Access Manager troubleshooting tools Access Manager protected resource flow Access Manager log settings and log files Case study Additional reading 2
Networking Tools Ethtool (-S, -K TSO) netstat -patune connection and stat info tcpdump/wireshark/tshark (SSL private key) netcat general ip/icmp/tcp/udp stats under /proc/net/snmp ipsysctl TCP settings Network layout (firewall blocking data, SSL terminators; Load Balancers redirecting ports; masquerading) 3
Generic NetIQ Access Manager Troubleshooting Tools (cont.) Certificates and keystores openssl s_client -connect idpcluster.lab.novell.com:8443 CONNECTED(00000003) depth=1 /OU=Organizational CA/O=linuxlab5_tree verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/cn=idpcluster.lab.novell.com i:/ou=organizational CA/O=linuxlab5_tree 1 s:/ou=organizational CA/O=linuxlab5_tree i:/ou=organizational CA/O=linuxlab5_tree keytool -list -keystore /var/opt/novell/novlwww/devman.keystore -v Your keystore contains 1 entry Alias name: tomcatcreation date: 13-Dec-2013 Entry type: keyentrycertificate chain length: 2 Certificate[1]:Owner: O=novell, OU=accessManager, CN=linuxlab5 Issuer: O=linuxlab5_tree, OU=Organizational CA :Certificate[2]: Owner: O=linuxlab5_tree, OU=Organizational CA Issuer: O=linuxlab5_tree, OU=Organizational CA : 4
Generic NetIQ Access Manager Troubleshooting Tools (cont.) HTTP Request generator Curl (http://curl.haxx.se/docs/manpage.html) HTTPRequester FF plugin D71B8B5632BC0372969BD0D 1FAAB4AD8 6
Generic NetIQ Access Manager Troubleshooting Tools (cont.) IDP config 'Logging' TAB configuration 7
Generic NetIQ Access Manager Troubleshooting Tools (cont.) AC general logs from 'Auditing' TAB 8
Generic NetIQ Access Manager Troubleshooting Tools (cont.) Performance analysis tools on dependencies http://www.novell.com/communities/node/7063/elapsed-time-416 (LDAP performance on edirectory) HTTP common or extended logs (Web server performance) X-MAG FP4 header timestamp (DebugHeaders on) 9
Generic NetIQ Access Manager Troubleshooting Tools (cont.) Statistic logging (Auditing Device Health Device Statistics) 11
Access Gateway Overview Identity Server Identity Store 3 2 4 1 5 1. User Accesses protected resource 2. User is redirected to Identity Server and is presented with an http login form requesting their username and password 3. The Identity Server verifies the username and password against the Identity Store 4. Once the user's identity is validated, the Access Gateway retrieves the user's common name and password 5. The Access Gateway injects the username and password into the authentication header and allows access to the encrypted Web content Access Gateway Apache or IIS web server configured to accept header-based authentication 13
Access Manager Advanced Overview Existing Session with Web Single-Sign-On and Access Gateway Cluster Assume User already had active session on AG2. Identity Server 5, 57 6 Identity Store 1. User Accesses protected resource on AG1 and the browser presents AG1 a valid Access Gateway session cookie created earlier by AG2 for this user session. 2. AG1 doesn't have a session for the user so it asks the other AGs in the Group to see if they have a session for the user 3. AG2 responds claiming ownership for the user session AG2 4. AG1 asks AG2 for the policy and user data required for the user to access the protected resource 2, 2, 3, 2, 3, 24, 348 5. AG2 requests policy and user data from the Identity Server (if it isn't cached) 6. The Identity Server gets the user data from the Identity Store (if it isn't cached) Web Browser 1 AG1 9 Access Gateway Group (Load Balanced by L4 Switch) Web Servers 7. Identity Server responds to AG2 with the policy and user data 8. AG2 responds to AG1 with the policy and user data Assume authentication headers used for SSO to origin web servers 9. AG1 processes the policy and user data and allows access to the protected resource 14
AG Architecture 17
AG Tools Advanced Logging Options Advanced options Adds custom NAM level logging to error_log for each request /var/log/novell-apache2/error_log Can directly modify httpd.conf with these lines at the end and restart /etc/opt/novell/apache2/conf/ directory 18
AG Tools : X-MAG Headers Short descriptions about the processing path. 20
AG Tools - Server Status Http://localhost:8181/server-status (use w3m) Gives web based real time statistics (load, CPU, connections, balancer) from httpd & AG module State (waiting, writing, keepalive, closing, etc) and number of free (idle) network slots that can serve Server generation Uptime Traffic data 21
AG Troubleshooting Logs /var/log/novell-apache2/rcnovell-apache2.out Apache startup messages (N/A for Windows) /var/opt/novell/amlogging/logs/ags_error.log NAM specific Apache startup messages and configuration updates /var/log/novell-apache2/access_log or extended_log Uses CommonLog or ExtendedLog module from apache 22
AG Logs - error_log /var/log/novell-apache2/error_log Httpd logs for GET/Response traffic from browsers here. Most of the logs will be here. General apache errors so can Google for generic issues Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#: AMEVENTID#994: Requ: GET https://nam32app-vm.lab.novell.com/rewriter/phpinfo.php service:nam32vm-pxy-srvc (147.2.47.176:2551->147.2.34.116:443) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994: validatecookie:local user. Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600100 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994: Restricted URL Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994: matched PR:rewriter-pr Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994: Contract-valid contract(secure/name/password/uri - >secure/name/password/uri) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994: balancer cookie is ZNPCQ003-31353600=a1b14cc2; Path=/; Domain=.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] proxy: HTTP: fam 2 socket created to connect to 147.2.16.154 Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] proxy: HTTP: connection complete to 147.2.16.154:80 (147.2.16.154) Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: connected from 147.2.34.116:46079 to 147.2.16.154:80 Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: sending request to webserver Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: received response from server Nov 14 19:35:19 mag32app-vm httpd[31648]: [info] AMEVENTID#994: received status 200 from server Nov 14 19:35:19 mag32app-vm httpd[31648]: [warn] AM#304600001 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#3A2EDA43D25C3D43620D2F8838202FF0: AMEVENTID#994:status:200 GET https://nam32appvm.lab.novell.com/rewriter/phpinfo.php <0100140093022fb0aeb56e9b4636e881ca59cdc4> X-Mag: <45B6586EB94FC2A7;ca59cdc4;994;usrLkup- >0;usrBase->0;LocUsr;rewriter-pr;Contract-valid->0;aclEvalTout->0;EvalACL->11;Allow->11;aud->11;nam32vm-pxy-srvc;default;SH;FP2- >11;WS=a1b14cc2;default;$custom_urs-ff-rewriter;FP4->17;C005;> [147.2.47.176:2551->147.2.34.116:443]service:nam32vm-pxy-srvc (185:3) - 23
AG Logs - httpheaders /var/log/novell-apache2/httpheaders HTTP headers output from browser <-> Proxy and Proxy <-> Web server Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from client 147.2.47.176 for id 995:185: Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Host: nam32app-vm.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Connection: Keep-Alive Nov 14 19:35:19 mag32app-vm httpd[31648]: Received from client for ID:995:185: Cookie: IPCZQX03bafce9af=0100140093022fb0aeb56e9b4636e881ca59cdc4; ZNPCQ003-31353600=a1b14cc2 Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers to webserver 147.2.16.154 for id 995:185: Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: GET /neil/phpinfo.png HTTP/1.1 Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: Host: ncsles11ws.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: If-None-Match: "a18068-12e-38c234eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: If-Modified-Since: Wed, 22 Aug 2001 19:23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: Via: 1.1 nam32app-vm.lab.novell.com (Access Gateway-ag- 45B6586EB94FC2A7-995) Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-For: 147.2.47.176 Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-Host: ncsles11ws.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: Sending to webserver for ID:995:185: X-Forwarded-Server: nam32app-vm.lab.novell.com Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Server: Apache/2.2.3 (Linux/SUSE) Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Last-Modified: Wed, 22 Aug 2001 19:23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:ETag: "1aed0-12e-4eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Accept-Ranges: bytes Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Content-Length: 302 Nov 14 19:35:19 mag32app-vm httpd[31648]: received from webserver for ID:995:185:Content-Type: image/x-icon Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Last-Modified: Wed, 22 Aug 2001 19:23:26 GMT Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: ETag: "1aed0-12e-4eec8780" Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Accept-Ranges: bytes Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Content-Length: 302 Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Content-Type: image/x-icon Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: X-Mag: 45B6586EB94FC2A7;ca59cdc4;995;usrLkup- >0;usrBase->0;LocUsr;_public_;publicURL->0;nam32vm-pxy-srvc;default;SH;FP2->35;WS=a1b14cc2;FP4->37; Nov 14 19:35:19 mag32app-vm httpd[31648]: Headers from proxy to client for ID:995: Via: 1.1 nam32app-vm.lab.novell.com (Access Gateway-ag- 45B6586EB94FC2A7-995) 24
AG Logs - soapmessages /var/log/novell-apache2/soapmessages Logs AG <-> ESP SOAP traffic (authentication and policy evaluation) Nov 14 19:35:11 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><CookieBrokerRequest verb="add" correlationid="990"><addentry seconds="130" owner="true" key="0100140093022fb0aeb56e9b4636e881ca59cdc4"><sessiondata UserRole="k9" UserName="public" pid="0" SPSessionID=""><contracts/></SessionData></AddEntry></CookieBrokerRequest></SOAP-ENV:Body></SOAP- ENV:Envelope> Nov 14 19:35:11 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><CookieBrokerResponse correlationid="990"><addentryresponse key="0100140093022fb0aeb56e9b4636e881ca59cdc4" status="ok"/></cookiebrokerresponse></soap-env:body></soap-env:envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <ns1:envelope><ns1:body><nidpsetsession softexpire="194" refreshcache="false" pid="kpo-ps)znt.qz6kpzeocrgiv]" id="3a2eda43d25c3d43620d2f8838202ff0" hardexpire="299" XLibid="0100140093022fb0aeb56e9b4636e881ca59cdc4"><store type="ldap"><dn>cn=ncashell,o=novell</dn></store><authentications><contracts><contractset="true">secure/name/password/uri</ contract></contracts></authentications><roles><role>secnamepwdauthrole</role><role>authenticated</role></roles></nidpset Session></ns1:Body></ns1:Envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><NXPES Id="994"><Evaluate Verbose="on" PolicyId="K37M6K86-M788-1NPP-15PK-9069K8746517"><ContextDataElement Value="3A2EDA43D25C3D43620D2F8838202FF0" Enum="2551"/></Evaluate></NXPES></SOAP-ENV:Body></SOAP- ENV:Envelope> Nov 14 19:35:19 mag32app-vm httpd[31648]: <SOAP-ENV:Envelope xmlns:soap- ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><NXPES Id="994" Status="success"><EvaluateResponse><DoAction ActionName="Permit" ActionTTL="-1" Enum="2610"/></EvaluateResponse></NXPES></SOAP-ENV:Body></SOAP-ENV:Envelope> 25
AG/ESP Logs - Catalina /var/opt/novell/nam/logs/idp(nesp)/tomcat/catalina.out esp logs for communication with proxy and IDP esp inherits IDP logging settings ('Application, Liberty, Web Service Provider/Consumer) Used to troubleshoot import, authentication and policy issues Can search for JSESSIONID, Policy ID or threadid (Processor string) Display IDP/ESP statistics Performance issues running out of threads (maxthreads, Xmx, LDAPLoadThreshold, Attribute queries) http://www.novell.com/communities/node/9321/how-configure-access-gatewayembedded-service-provider-reduce-access-gateway-load-and-impr 26
Troubleshooting Slow Performance HTTPWatch output with timestamps LAN traces with private keys Check whether ESP is slow (from server-status or enable access log in server.xml) Check whether backend is slow Enable extended logs and check response time logged X-mag header FP4 gives the response time for each request. 27
Troubleshooting Services Issues Look at server-status output (w3m http://127.0.0.1:8181/server-status > w3m-mag-status.out) Accelerator: nam32vm-pxy-srvc Accelerator Type: Https Listener HostName: nam32app-vm.lab.novell.com Listen: 147.2.34.116:443 AltHost: ncsles11ws.lab.novell.com CookieDomain:.lab.novell.com Reverse Proxy bal_nam32vm-pxy-srvc SSes Timeout Method ZNPCQ003-31353600 0 byrequests Sch Host Stat Route Redir F Set Acc Wr Rd http 147.2.16.154 Ok a1b14cc2 1 0 31 13k 31k Reverse Proxy bal_owa-pbmh SSes Timeout Method ZNPCQ003-32343200 0 byrequests Sch Host Stat Route Redir F Set Acc Wr Rd https 151.155.134.178 Ok 0b45d31b 1 0 32 14k 9.6k Look at 'netstat -patune grep -i listen' output Check all listeners active for IP address and Port combinations 28
Troubleshooting SSL Issues Listener not getting created on 443 Certificates not present check if required certs are present in /etc/opt/novell/apache2/conf/certs/ search for certificates in /var/log/novell-ag-logs/novell-apache2/error_log to see the reason for failure (LogLevel debug!) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Loading certificate & private key of SSL-aware server Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Generating temporary RSA private keys (512/1024 bits) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Generating temporary DH parameters (512/1024 bits) Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Init: Initializing (virtual) servers for SSL Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [info] Configuring server for SSL protocol Nov 19 11:45:19 mag32app-vm httpd[11469]: [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC4366) Jcc timing issue with certificate command and reinitialize try restarting proxy and jcc with /etc/init.d/novell-apache2/novell-jcc restart 29
Troubleshooting SOAP Channel Issues Used to view all data on backchannel TCPDUMP to trace the loopback interface Soap Requests to 127.0.0.1:9009 tcpdump -t -p -s 0 -i lo 'tcp port 9009' -w soapch.cap to log all soap policy requests between proxy and ESP Unsolicited Response from ESP ESP signals end of authentication login/logouts through these requests tcpdump -t -p -s 0 -i lo 'tcp port 8181' -w unsolicited.cap 'DumpSoapMessages on' Advanced Option can view Identity Injection, formfill and Authorization policy interactions in logs 30
Troubleshooting SOAP Channel 31
Troubleshooting SOAP Channel Issues CATALINA.OUT output Enable IDP logging for Liberty component as well as content filters for Liberty tail -f /var/log/novell-ag-logs/maglogs/catalina.out <SOAP-ENV:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Body> <CookieBrokerRequest verb="add" correlationid="560"> <AddEntry key="00000000930224c8946e8826de2839030f94bf03f6dbb058" owner="true" seconds="571"> <SessionData UserName="cn=ncashell,o=novell" UserRole="" AuthStatus="1" HardTimeout="16038" SPSessionID="80B294FDCEE4ED69792F78C712DC374F"SoftTimeout="10422" CreationTime="1333" ContractNameKey="1487985654"/> </AddEntry> </CookieBrokerRequest> </SOAP-ENV:Body> </SOAP-ENV:Envelope> AJP connector configuration Increase number of threads 32
Troubleshooting Connection Issues Verify listener is active netstat -patune grep -i listen verify all listeners are there for the configured TCP ports Check error_log log file for 'Sockets' component Nov 21 15:35:28 mag32app-vm httpd[20855]: [info] proxy: HTTP: fam 2 socket created to connect to 147.2.16.133 Nov 21 15:35:28 mag32app-vm httpd[20855]: [info] proxy: HTTP: connection complete to 147.2.16.133:80 (147.2.16.133) Check LAN trace for communication issues (retransmits and delays) Check /var/log/messages for NIC specific errors Check ip statistics using netstat output for icmp errors and tcp/ip stats 33
Troubleshooting Rewriter Issues HTTPWatch output with timestamps or Fiddler (free) View source going direct and through AG Compare the differences (search scheme, internal info, ctype) Advanced Options available NAGDisableExternalRewrite X-mag header FP4 gives the rewriter profile executed May need additional entries in rewriter policy 34
Troubleshooting Authorization Issues Confirm X-MAG headers and Via eventid Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504600000 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: matched PR:rewriter-pr Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504600005 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL eval sending Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504601203 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Sending value 23B586CA4B1E942464CA9169CF4A88F2 for enum LibertyID Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504601203 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL policy id:65kmm806-84k3-m4lm-nm71-l036l9o07415 Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504600404 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: subreq nam32app-vm.lab.novell.com:/nesp/app/soap Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504601301 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Received ACL Eval Permit Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504600005 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: ACL eval success Nov 13 18:11:36 mag32app-vm httpd[23538]: [info] AM#504600005 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: Allow Nov 13 18:11:36 mag32app-vm httpd[23538]: [warn] AM#304600001 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: AMEVENTID#6: status:200 GET http://nam32app-vm.lab.novell.com/rewriter/phpinfo.php <01000600952ca61522f67560acf117d2ca59cdc4> X-Mag: <45B6586EB94FC2A7;ca59cdc4;6;usrLkup->0;usrBase->0;LocUsr;rewriter-pr;Contract-valid- >0;aclEvalTout->0;EvalACL->14;Allow->14;aud->14;FPE->14;> [149.44.166.21:4429->147.2.34.116:80]service:nam32vm-pxy-srvc (10:0) ESP evaluation <amlogentry> 2012-11-13T18:11:36Z INFO NIDS Application: AM#501102050: AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#23B586CA4B1E942464CA9169CF4A88F2: PolicyID#65KMM806-84K3-M4LM-NM71-L036L9O07415: NXPESID#6: AGAuthorization Policy Trace: ~~RL~1~~~~Rule Count: 2~~Success(0) ~~RU~RuleID_1333023317359~Authz-SecNamPwdNotManagers-pol~DNF~~1:1~~Success(0) ~~CS~1~~ANDs~~2~~True(69) ~~CO~1~AuthenticationContract(6620):no-param:secure/name/password/uri~com.novell.nxpe.condition.NxpeOperator@stringequals~SelectedAuthenticationContract(6621):hidden-param:hidden-value:~~~True(69) ~~CO~2~CurrentRoles(6660):no-param:SecNamePwdAuthRole,authenticated~com.novell.nxpe.condition.NxpeOperator@stringequals~SelectedRole(6661):hidden-param:hidden-value:~~NOT~True(69) ~~PA~1~~Permit Access~~~~Success(0) ~~PC~1~~Document=(ou=xpemlPEP,ou=mastercdn,ou=ContentPublisherContainer,ou=Partition,ou=PartitionsContainer,ou=VCDN_Root,ou=accessManagerCont ainer,o=novell:romacontentcollectionxmldoc),policy=(authz-secnampwdnotmanagerspol),rule=(1::ruleid_1333023317359),action=(permit::1)~~~~success(0) </amlogentry> 35
Troubleshooting Identity Injection Confirm X-MAG headers and Via eventid [Wed Apr 11 18:03:39 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32app-vm.lab.novell.com/formfill/phpinfo.php <01001100952c859be154910448048a2eca59cdc4> X-Mag: 45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;ConfigII->120;configACL->186;NoPol;ConfigFF->251;formfill-pr;Contract-valid->251;usrPr- >252;Allow->252;aud->252;nam32vm-pxy-srvc;EvalII->296;CHd;AH;QS;FP2->297;WS=a1b14cc2;default;FP4->305;C005; Confirm HTTP headers sent (error_log or httpheaders) Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: GET /formfill/phpinfo.php?x-roles=authenticated HTTP/1.1 Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Host: ncsles10.lab.novell.com Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 : Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-client-IP: 149.44.133.155 Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Auth-Cont: name/password/uri Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-mail: ncashell@ag4cdemo.info Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Authorization: Basic bmnhc2hlbgw6bm92zwxs Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-For: 149.44.133.155 Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Host: ncsles10.lab.novell.com Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Server: nam32app-vm.lab.novell.com Confirm ESP evaluation <amlogentry> 2012-04-11T17:03:39Z DEBUG NIDS Application: AM#501103050: AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: PolicyID#N885856P-48PP-9NN7-1K15-N8O6NOP040L2: NXPESID#7: AGIdentityInjection Policy Trace: ~~RL~1~~~~Rule Count: 4~~Success(67) ~~RU~RuleID_1333023361576~Identity-Inj-All-Pol~DNF~~0:1~~Success(67) ~~PA~ActionID_1333023364340~~Inject Auth Header~uid~uid(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDA PCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D:~Ok:ttl -1~Success(0) ~~PA~ActionID_1333023364340~~Inject AuthHeader~password~pwd(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~ 2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPredentials~2 2~5D~2Fcp~3AEntry~5Bcp~3AName ~3D~22UserPassword~22~5D:~Ok~Success(0) 36
Troubleshooting Formfill Issues STRACE output to look at the form details Could get LAN traces with private keys Confirm policy evaluated in the catalina log file Search for 'AGFormFill Policy Trace' or 'NXPESID#EventIDNumber' Enable 'NAGGlobalOptions DebugFormFill=on' Advanced Option X-mag header FP4 gives the response time for each request [Wed Apr 11 17:43:08 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32appvm.lab.novell.com/formfill/bootcamp.htm <01001100952c859be154910448048a2eca59cdc4> X- Mag:45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;NoPol;ConfigII->116;NoPol;configACL->174;ConfigFF- >236;EvalFF->247;formfill-bootcamp-pr;Contract-valid->247;mastercdnFormfill-Pol-bootcamp3310;FF4GUD- >267;FillSilent;Match FormName;Match;username;Miss;title;Match;password;Miss;ldap;FF4End->267;FP4->267; Enable NAGGlobalOptions InPlaceSilent 37
Troubleshooting Case Study: Single Sign-On to Back-End App Fails with Formfill
Policy Case Study Background Customer enabled a Formfill policy to apply to a single protected resource (/Citrix/XenApp/auth/login.aspx) with login page populating: username and password into the form using LDAP credentials Formfill policy javascript, auto-submit and mask data options enabled enabled After user logs in and accesses Access Gateway protected resource, the user could not SSO to the back-end Web App authentication failed, and an internal error messages was returned from the back-end application The Web site is experiencing technical difficulties. We apologize for any inconvenience. The internal error may only be temporary. Try reconnecting and, if the problem persists, contact your system administrator 39
Policy Case Study Troubleshooting Get policy and where policy applied (get screenshot) of protected resources and export of policy) 40
Policy Case Study Troubleshooting Enable logs for policies and Proxy and gather all the key logs (from cheatsheet) and LAN trace after duplicating issue https://www.novell.com/support/kb/doc.php?id=7015707 Enable logs for policies and Proxy Must understand where in the policy flow the request is failing (Web server, Proxy server, esp, IDP, user store)? 41
Policy Case Study Proxy Log Analysis Check browser HTTP trace cookies for form and X-MAG header to verify FF triggered 45B6586EB94FC2A7;ca59cdc4;461;usrLkup->0;usrBase->0;LocUsr;getPRBefFind->0;PRAfterFind->1;ConfigFF- >228;EvalFF->240;Citrix-loginpage-ff-pr;Contract-valid->241;nam32vm-pxy-srvc;HdrRRwNo;FF1End->241;FP2- >241;WS=a1b14cc2;default;$custom_citrix-ff-rewriter;setupFF-interested;mastercdncitrix-ff-pol3310;FF4GUD- >638;FillSilent;Match FormName;Miss;SESSION_TOKEN;Miss;LoginType;Match;user;Match;password;Match;tree;FP4- >644; 42
Policy Case Study Proxy Log Analysis Check browser HTTP trace to see if credentials POSTed they are but error back 43
Policy Case Study Proxy Log Analysis Check what AG POSTs to Web server need tcpdump output 44
Policy Case Study Solution Confirmed that FF policy triggered from X-MAG header Confirmed credentials POSTed to AG from browser and to Web server Verified that Web server failed to validate credentials Disabled mask data option worked Content-length sent by AG to Web Server not adjusted correctly Bug on NAM side 45
Additional Reading Best Practices Guide https://www.netiq.com/documentation/netiqaccessmanager4/bestpractices/ data/bookinfo.html Avoiding memory and performance issues with Java http://www.novell.com/communities/node/9321/how-configure-accessgateway-embedded-service-provider-reduce-access-gateway-load-and-impr Troubleshooting 100101044/43 errors http://www.intl.novell.com/communities/node/2297/troubleshooting- 100101043-and-100101044-errors-access-manager Troubleshooting SAML http://www.intl.novell.com/communities/node/2303/configuring-andtroubleshooting-saml-11-novell-access-manager Performance and Sizing Guidelines White Paper https://www.netiq.com/documentation/novellaccessmanager32/resources/perf ormance_sizing/performance_sizing.pdf 46
Don t miss the Identity-Powered Experience in IT Central. Thank you. 54 2014 NetIQ Corporation. All rights reserved.
This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time. Copyright ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.