OIOSAML 2.0 Toolkits Test results May 2009

OIOSAML 2.0 Toolkits Test results May 2009 5. September 2008 - Søren Peter Nielsen: - Lifted and modified from http://docs.google.com/a/nemsso.info/doc?docid=dfxj3xww_7d9xdf7gz&hl=en by Joakim Recht 12. May 2009 Søren Peter Nielsen Added updated information regarding support of ispassive flag This document describes how different SAML 2.0 reference implementations/toolkits have been tested according to the Danish egov OIOSAML 2.0 profile. The tested reference implementations/toolkits are: OIOSAML.JAVA, version 2965 OIOSAML.NET version 1.0 SimpleSamlPHP version svn-585 Test cases The toolkits have been tested against the following Service Provider test cases listed in the table below. The test results are shown in the following table Test Case ID IT-LOGON-1 IT-SSO-1 IT-SPSES-1 IT-SLO-1 IT-SLO-2 IT-LOA-1 IT-TIM-2 IT-CERT-1 IT-CDC-1 IT-ATTQ-1 Description The user accesses a protected web page at the service provider without the prior session and with empty common domain cookie. The re-direct must go to the default IdP where the user performs log-in, the user is sent back and have access to the desired page at the service provider. The user accesses a protected page at the service provider and already have a session with an IdP. Which IdP is resolved through the common domain cookie, and SSO is performed via the IdP, after which the user can access the page from the service provider without authenticating again. The user accesses a protected page at the service provider and already have a session at the service provider. The user must be able to access the page without being sent to the IdP. The user selects single logout from the current service provider, and is logged out of all the sessions in the federation. This tests that the service provider can initiate the single logout. The user selects single logout from another service provider and must be logged out of the session with the current service provider. This tests that the service provider can be included in the single logout (without being the initiating party). The user accesses a protected resource at the service provider with a level of authentication that is too low for the required resource. Access must be denied. Variations: with and without a current session. User accesses a protected resource at the service provider after his session has timed out. On the other hand, his IdP session is still active. The user must no be prompted to authenticate. Validation of signature or certificate on the assertion fails. Variations: a) invalid signature b) revoked certificate c) expired certificate d) non-trusted certificate e) CA does not respond Test that the common domain cookie actually is used by the service provider. Testing attribute query against the IdP. The table is extracted and translated from the document Integrationstest ved føderationstilslutning - Version 0.62 - Udkast

Test results Overall, the test results are summarized in the table below. Further notes about setup and the individual toolkit testing follows later on. OIOSAML.OIO OIOSAML.NET IT-LOGON-1a ok ok ok IT-LOGON-1b ok ok ok IT-SSO-1a ok ok ok IT-SSO-1b ok ok ok IT-SSO-2 ok ok ok IT-SPSES-1 ok ok ok IT-SLO-1 ok ok ok IT-SLO-2 ok ok ok SimpleSamlPHP SP IT-LOA-1 ok Not passed Not passed IT-TIM-2 ok ok ok IT-CERT-1a ok ok ok IT-CERT-1b ok Not passed Not passed IT-CERT-1c ok Not passed Not passed IT-CERT-1d n/a n/a n/a IT-CERT-1e Not tested Not tested n/a IT-CDC-1 ok Supported not tested Not passed IT-ATTQ-1 ok ok Not passed Notes to the Not passed test cases IT-LOA-1: In both cases the Level of Assurance is passed correctly to the Service Provider, so it is simply a question about missing implementation in the reference implementation being tested IT-CERT-1 b & c: Currently testing whether certificate used with the assertion is revoked or expired has to be added by the implementer. We welcome this functionality as give-back. IT-CDC-1: OIOSAML.NET support common domain cookie, but it has not been tested formally. SimpleSamlPHP does currently not support common domain cookie IT-ATTQ-1: SimpleSamlPHP does currently not support attributequery

Test Identity Provider: PingFederate 4.4 The IdP used for testing is based on the PingFederate 4.4. Generally, the default settings have been used. All service providers have been configured with 5 minutes assertion time window All SSO and SLO profiles enabled Standard identifier mapping using the idpoces2 adapter SAML_SUBJECT is mapped to subject, dk:gov:saml:attribute:assurancelevel is mapped to "3". No other attributes are included Signature policy: Require signed requests, always sign the SAML assertion Encryption policy: Encrypt entire assertion, nothing else Backchannel: Require signatures, no http basic auth, don't include key info In the test, the IdP is located at https://saml.idp.trifork.com:9031/idp. OIOSAML.JAVA - Testing notes Test run using Apache Tomcat 6.0.14 and Java 5 under Ubuntu Linux 8.04 Tomcat unpacked Copied lib/endorsed to the unpacked Tomcat dir Environment variable set: JAVA_OPTS= -Doiosaml.home=/tmp/oiosaml Configured OIOSAML.java using the autoconfigure mode Each test starts with a new browser (or clearing all sessions). IT-LOGON-1 Configure Ping to use OCES login Enter <base>/sp/priv1.jsp Check that the returned assertion contains valid attributes Repeat for b) for both variations IT-SSO-1 at IdP Enter <base>/sp/priv1.jsp Check that an assertion is returned at once, and that it contains valid attributes for both variations

IT-SSO-2 Enter the IdPSample application Initiate SSO for the service provider Check that SSO is performed IT-SPSES-1 IT-SLO-1 IT-SLO-2 IT-LOA-1 Enter <base>/sp/priv1.jsp Reload <base>/sp/priv1.jsp and check that no redirects are performed Enter <base>/sp/priv1.jsp and authenticate Access <base>/saml/logout Final page should be <base> Access <base>/sp/priv1.jsp and check that authentication is performed Enter <base>/sp/priv1.jsp and authenticate Access other SP and make sure SSO is performed Access logout from other SP Access <base>/sp/priv1.jsp and check that authentication is performed Edit oiosaml-sp.properties and set oiosaml-sp.assurancelevel=4 Reload server Access <base>/sp/priv1.jsp and authenticate Check that an error is displayed IT-TIM-2

IT-CERT-1 Result: IT-CDC-1 IT-ATTQ-1 Access <base>/sp/priv1.jsp and authenticate Delete local cookie Access <base>/sp/priv1.jsp and check that SSO is performed Use the revoked certificate from https://www.certifikat.dk/developer/eksempler.html for the IdP Export new metadata from Ping and save it in.oiosaml/metadata/idp, overwriting the existing file Configure OIOSAML.java to use http://test.crl.oces.certifikat.dk/oces.crl as crl Access <base>/sp/priv1.jsp and check that an error ocurs after receiving an assertion Invalid signature: ok Revoked certificate: ok Expired certificate: ok Certificate not trusted: Not tested, trust is not checked or used CA not responding: Not tested Configure an additional IdP and add metadata to.oiosaml/metadata/idp Configure the Discovery Service by deploying the war file and setting oiosamlsp.discovery in oiosaml-sp.properties to point to the service Write the CD cookie, for example by accessing https://samlidp.trifork.com:9031/idp/writecdc.ping?targetresource=someurl Restart the server and access <base>/sp/priv1.jsp Check that a redirect is performed to the discovery service Check that the redirect back contains the correct _saml_idp value Configure Ping to accept AttributeQuery. This test has been performed by accessing the cn attribute Go to <base>/sp/query.jsp Enter valid nameid and attribute name Check that valid response is returned

IT-FORCE-1 Set oiosaml-sp.authn.force=.* in oiosaml-sp.properties Sign on at the IdP Access <base>/sp/priv1.jsp Check that authentication is forced IT-ISPASSIVE-1 Set oiosaml-sp.passive=true and oiosaml-sp.passive.user=anonymous in properties Access <base>/sp/priv1.jsp Check that no authentication is performed Result: Fails with Ping 4.4, IsPassive is ignored. Request sent contains ispassive= true UPDATE: Later versions of PingFederate support IsPassive, and works correctly with OIOSAML.JAVA 4340. Notes Attribute profiles have not been configured according to OIOSAML Tests have been run on Ping 4.4 OIOSAML.NET Service Provider - Testing notes Test run on Windows 2003 server SP2, IIS 6.0,.NET 3.5 Toolkit installed using default location by following the installation guide Virtualhost configured in IIS pointing to c:\program files\dk.nita.saml20\bin\websitedemo Edited Web.config to set base url and Entity id Each test starts with a new browser (or clearing all sessions). IT-LOGON-1 Configure Ping to use OCES login Enter <base>/mypage.aspx Check that the returned assertion contains valid attributes Repeat for b) for both variations. Fails if assertion contains attributes not defined in OIOSAML/are not named as uris.

IT-SSO-1 at IdP Enter <base>/mypage.aspx Check that an assertion is returned at once, and that it contains valid attributes for both variations. Same output as for IT-LOGON-1. IT-SPSES-1 IT-SLO-1 IT-SLO-2 IT-LOA-1 Enter <base>/mypage.aspx Reload <base>/mypage.aspx and check that no redirects are performed Enter <base>/mypage.aspx and authenticate Access <base>/logout.ashx Final page should be <base> Access <base>/mypage.aspx and check that authentication is performed Enter <base>/mypage.aspx and authenticate Access other SP and make sure SSO is performed Access logout from other SP Access <base>/mypage.aspx and check that authentication is performed Included application does not support AssuranceLevel directly. Assurance level checking must be coded manually by retrieving the assurancelevel attribute. IT-TIM-2 Access <base>/mypage.aspx and authenticate Delete local cookie Access <base>/mypage.aspx and check that SSO is performed

IT-CERT-1 Result: IT-CDC-1 Use the revoked certificate from https://www.certifikat.dk/developer/eksempler.html for the IdP Export new metadata from Ping and save it c:\saml20\metadata, overwriting the existing file Access <base>/mypage.aspx and check that an error ocurs after receiving an assertion Invalid signature: ok - see general notes for test description Revoked certificate: fail no place to configure CRL, and embedded crl info is not used Expired certificate: fail expired certificates are not checked Certificate not trusted: Not tested, trust is not checked or used CA not responding: Not tested The SAML Discovery profile is supported, but it is not possible to configure redirects to the common domain. IT-ATTQ-1 Configure Ping to accept AttributeQuery. This test has been performed by accessing the cn attribute. Furthermore, MyPage.aspx has been extended to do an AttributeQuery inline Go to <base>/mypage.aspx Check that valid response is returned Error handling not very good exception a non-success status from the IdP returns an XML Attributes are added to session object, not returned from PerformQuery call. Only attributes for the current user can be retrieved. IT-FORCE-1

Support for ForceAuthn is not exposed in the demo app. The API seems to support it, but there is no way of accessing it. IT-ISPASSIVE-1 Support for IsPassive is not exposed in the demo app. The underlying API seems to support IsPassive, but there is no way of accessing it. UPDATE: OIOSAML.NET 1.4_plus supports the ability to set IsPassive dynamically and have been verified to work correctly with a PingFederate IdP. Notes Exported metadata from IdP must be edited to contain only one SingleSignonService Tests have been run on Ping 4.4 Only OIOSAML compliant attributes are acceptet. SSO fails otherwise. No logging all requests and responses are taken from the IdP Metadata seems to be cached this is not documented, and it is hard to find out what metadata is actually used. This has been communicated to Safewhere, and a new version should fix the undefined behavior. Documentation is ok, but certificate installation is not completely clear. Section 5.1.4 (Running/Sample/ACL) does not really make sense. Except for problems with certificates, installation is easy when following the documentation guide The demo app is not very polished - no styling or html layout SimpleSamlPHP Service Provider - Testing notes Installed apache2 and php5 under Ubuntu Server 8.04 (older versions run a version of php incompatible with oces). Http://simplesaml.trifork.com/sp points to the installation. svn co http://simplesamlphp.googlecode.com/svn/trunk sp Copied config-templates/config.php to config/config.php Set baseurlpath to 'sp/' Set logging.level to LOG_DEBUG Set logging.handler to file Set default-saml29-idp to NULL Copied metadata-templates/saml20-idp-remote.php and saml20-sp-hosted.php to metadata/ Removed DYNAMIC from saml20-sp-hosted

Inserted the following instead: 'simplesaml.trifork.com' => array( 'host' => 'simplesaml.trifork.com', 'request.signing' => true, 'privatekey' => 'simplesaml.trifork.com.pem', 'certificate' => 'simplesaml.trifork.com.crt' ) Generated new key for SP (in certs/): openssl genrsa -out simplesaml.trifork.com.key 1024 openssl rsa -in simplesaml.trifork.com.key -out simplesaml.trifork.com.pem openssl req -new -key simplesaml.trifork.com.key -out simplesaml.trifork.com.csr openssl x509 -req -days 600 -in simplesaml.trifork.com.csr -signkey simplesaml.trifork.com.key -out simplesaml.trifork.com.crt Exported IdP certificate and saved it in certs/idp1.test.oio.dk.crt Added IdP metadata to metadata/saml20-idp-remote.php. Go to Simplesaml installation page and select 'XML to simplesamlphp metadata converter'. Paste IdP metadata xml and add the result to the php metadata file. Added these properties to the idp declaration: o 'request.signing' => true o 'certificate' => 'idp1.test.oio.dk.crt', o 'assertion.encryption' => true] Get SP metadata from simplephp installation page and import it into Ping. Also import the simplesaml.trifork.com.crt file as the signing certificate for the SP. Each test starts with a new browser (or clearing all sessions). To login, go to <base> and click SAML 2.0 SP example, and select idp1.test.oio.dk as IdP. IT-LOGON-1 Start SSO Check that the returned assertion contains valid attributes Repeat for b). Attributes not tested. IT-SSO-1 at IdP Start SSO in Simplesaml Check that an assertion is returned at once, and that it contains valid attributes. Attributes not tested IT-SPSES-1 Enter <base>

IT-SLO-1 IT-SLO-2 IT-LOA-1 Reload <base> and check that no redirects are performed Start SSO Logout by clicking the Logout link Final page should be <base> Access <base>/sp/priv1.jsp and check that authentication is performed Enter <base> and authenticate Access other SP and make sure SSO is performed Access logout from other SP Access <base> and check that authentication is performed SimpleSAMLphp does not support AssuranceLevel. IT-TIM-2 IT-CERT-1 Result: Access <base> and authenticate Delete local cookie Access <base> and check that SSO is performed Use the revoked certificate from https://www.certifikat.dk/developer/eksempler.html for the IdP Export new metadata from Ping, convert it in SimpleSAML and save it in metadata/saml20-idp-remote.php Access <base> and check that an error ocurs after receiving an assertion

Invalid signature: ok - see general notes for test description Revoked certificate: fail no place to configure CRL, and embedded crl info is not used Expired certificate: fail expired certificates are not checked Certificate not trusted: Not tested, trust is not checked or used CA not responding: Not tested IT-CDC-1 SimpleSAMLphp does not support the SAML discovery profile. IT-ATTQ-1 SimpleSAMLphp does not support Attribute Query. IT-FORCE-1 Set 'ForceAuthn' => true in metadata/saml20-sp-hosted.php Sign on at the IdP Access <base> Check that authentication is forced IT-ISPASSIVE-1 Set 'IsPassive' => true in metadata/saml20-sp-hosted.php Access <base> Check that no authentication is performed Result: Fails with Ping 4.4, IsPassive is ignored. Request sent contains ispassive= true UPDATE: Later versions of PingFederate support IsPassive, General notes Responses are not logged to the logfile NPE when receiving an error response to AuthnRequest Attributes are not mapped or checked in the test Signature validation was checked by using the Tamper Data extension for Firefox, capturing the SAMLResponse value, running it through the script below, and

inserting the modified value in the SAMLResponse. The script changes "joetest" to "recht", and assumes that the nameid is joetest. During the signature validation test, assertion encryption must be disabled. #!/usr/bin/ruby require 'cgi' require 'base64' q = CGI::unescape(STDIN.readlines.to_s) q = Base64.decode64(q) q = q.gsub(/joetest/, 'recht') print "SAMLResponse=" + CGI::escape(Base64.encode64(q)) /end of document