DirectTrust Interoperability Benchmarking April 9, 2013 Luis C. Maas III, MD, PhD CTO, EMR Direct Co-Chair, DirectTrust SATC Workgroup
Purpose of Testing: Strengthen DirectTrust Network New HISP to HISP connections generally result in minor interop issues Strong signaling from customer community about expectations for DirectTrust Network It should just work Customers cannot tolerate unpredictable failures 18 difference reference models (now more!) no two deployments of the RI are the same pairwise testing across this variety of systems reveals issues the TTT cannot not all HISPs perform MU2 certification Strong community of collaborators exists within DirectTrust history of connect-a-thon participation, good communication DirectTrust Network removes uncertainty in exchange through security policies, a common Certificate Profile, preliminary inspection by anchor bundle committee, removing incompatible certificates Interop testing can be performed on a continuous basis, with very little time commitment Demonstrate current level of success, take inventory of shortcomings Result: introduce essential points of contact at different HISPs Develop tools for improved onboarding of new HISPs moving forward 2
Purpose of Testing: Strengthen DirectTrust Network We re on each other s team --Lorde 3
Network Interoperability Over Time August, 2013 February 11 and 28, 2014 April 7, 2014 8 HISPs 14 HISPs 25 HISPs 4
5
Most Recent Interoperability Notes 14. SES reports considerable lag receiving cert over LDAP from Medicity 15. MHIN reports the following with respect to ICA: "This certificate cannot be verified up to a trusted certification authority." Upon reviewing the anchor certificate, an error was noted in the certificate: "The CA Root certificate is not trusted because it is not in the Trusted Root Certification Authorities store." 16. MHIN reports the following error message when attempting to send to Orion: "Please contact your system administrator; the following email(s) are not set up properly and can't be sent to securely: interop.testing@direct.webmailtest2.orionhealthmail.info" 17. ICA CRL is out of date 18. San Diego Health Connect's (Resolved), HealtheConnections RHIO of CNY systems return "unknown address" for both Covisint test addresses 19. San Diego Health Connect's system returns Message from system: Certificate CN=direct.webmailtest2.orionhealthmail.info,O=Orion Health\, Inc.,L=Santa Monica,ST=California,C=US is not trusted. when attempting to send to Orion 20. Sender reports having requested but not received a Dispatched MDN from recipient 21. CareAccord reports "Cannot send email from CareAccord, The public key certificate needs to be installed in to Medicity DNS or LDAP" Medicity is investigating issue with their LDAP server taking up to 15 seconds to respond. This is causing some HISPs to time out. CareAccord is one of them. 22. Updox MDN subject field consists of "Processed: [***** SPAM 5.4 *****]" 23. MDN not received; recipient (SES) does not support sending of Dispatched MDN at this time 24. server message size limit=2097152, resent without payload 25. Medicity could not find HIXNY certificate may not support LDAP 26. Covisint investigating not being able to send to Medicity 27. Failed to find CA cert 28. Orion notes the following: The trust anchor CN=Orion Health Direct Secure Messaging Public HISP CA is signed by the trust anchor CN=Orion Health Direct Secure Messaging. Therefore, the trust anchor CN=Orion Health Direct Secure Messaging Public HISP CA should be trusted. 6
What s Behind the Numbers Great interop among most HISPs Reporting at 70% as of 4/7/14 22 HISPs testing out of 25 total 7
Key Findings: Common Problem Areas MDN-related: Format of MDNs (extra spaces or line breaks, missing fields, etc.) Variable support for Dispatched MDNs Null Sender for MDNs Delay in transmission of Processed MDNs Variations in MDN requests Service-related: Unable to connect to SMTP server Slow LDAP server response time for certificates 8
Key Findings: Common Problem Areas Certificate-related: Variance from DT certificate profile DNS hosting of certificates S/MIME-related: lack of encryption/decryption support for multiple recipients CA-related: Stale CRLs Certificates chain to incorrect anchor 9
Key Findings: Areas for Improvement Tighten up MDN tolerances in general Add Dispatched MDN support Adhere to DT certificate profiles Develop a more mail server troubleshooting mindset When issues are found, make modifications to fix or accommodate 10
Action Items (1 of 2) Education Troubleshooting collaboration Documentation of interop challenges A new facet to our community of best practices High degree of willingness to participate Technical Resources Sometimes code fixes take a few days; some have taken a couple of months Some HISPs have limited resources even for the testing; still, progress continues pretty much across the board 11
Action Items (2 of 2) Adapting requirements to ensure better interoperability Accreditation is challenging today; even so, members would like to see interop requirements added Potential new accreditation criteria or guidance New test cases: handling MDNs with a few different common variations Minimum functionality requirements (e.g. ability to build AIA chains) System-wide adoption of Dispatched MDN capabilities More comprehensive certificate profile review as part of accreditation Update required evidence to reflect advances in MU2 testing tools. Interop testing requirements prior to accredited bundle participation 12
How to Perform Interop Testing Worksheet Color coded results If red/yellow, figure out how to get to green Read the Notes on the results matrix reports from other HISPs about the counter party HISP reports from other HISPs about you Other HISPs with a similar red/yellow pattern Check your mail server and STA logs If the message or MDN came in, where did it go? Leverage test environment to diagnose issues and test fixes Check the troubleshooting guide 13
Troubleshooting Guide: Relevant Standards Successful Direct messaging depends on understanding & correctly implementing numerous inter-related standards Direct Project Applicability Statement Implementation Guides for Certificate Discovery, Delivery Notification, Trust Bundle Distribution, and Edge Protocols DirectTrust Certificate Policy and HISP Policy RFC 2315, 2585, 2782, 2821, 3798, 4398, 4511, 4516, 5280, 5322, 5652, 5751, 5752, and more 14
Troubleshooting Guide: Sending & Receiving Errors Confirm your Direct address s certificate chains up to the right trust anchor Confirm your certs follow the DirectTrust Certificate Profile just like they did when you were approved by the trust bundle committee and update any old non-compliant certs. Confirm you are searching for certificates both in DNS and LDAP, and are reliably hosting in one or the other May need Java RI patch for AIA support if not at least agent 2.0.2 (RI 3.0.1) Make sure you are issuing CRLs on schedule and not allowing them to become stale without updating Monitor uptime of all components of your system (SMTP, DNS, LDAP, STA) 15
Troubleshooting Guide: MDN Errors Confirm Dispatched MDNs supported by counter party Confirm MDN wasn t received and overlooked: check mail server Common issues: extra spaces, line breaks, > s, unexpected sender display name, NULL sender, MDN not encrypted/not a Direct message, lack of message ID. Check IG/RFCs to make sure you are requesting properly Ask recipient to confirm MDN was sent. Good protocol stewardship involves letting the counter party know if they are not in compliance with spec(s) OR adjusting your system to accept broader interpretations of specs, as appropriate. Help strengthen the network by sharing your findings, either way. Any notes submitted are added to the interop matrix and/or used to classify duplicate issues. Takeaway: generate MDN requests & responses as closely to specs as possible; accept/parse as broadly as possible, EXCEPT community convention is to override NULL sender specification. NULL MDN sender breaks TTT; some senders (as MDN recipients) can t process 16
General Observations: the Testing Process Interop Enthusiasm Some HISPs seem less responsive, but all generally distill interop results into greener and greener results over time Some HISPs send more messages than might be required. It s safe to assume they re troubleshooting. Interop testing etiquette: Send a message or two each month OR as part of regression testing OR if you are trying to diagnose a problem Explain in the subject or message body the intent of your test, whether a response is requested, and whether you would like to know that a certain attachment type(s) were readable Do not reply to all via Direct message or regular email Arrange in advance a time to do performance testing with a counter party, in a mutually agreeable ecosystem such as both parties test environments 17
General Observations: Community Feedback Out-of-Network Service Parallel HISP or CA services offered by DTAAP Accredited or Candidate members outside of the DirectTrust network Customer confusion: most people think that if their HISP is a DirectTrust member or their certificate is from a DirectTrust CA that they should be able to interoperate with other HISPs in the DT network. Subsequent inability to interoperate is incorrectly attributed to failure of the DirectTrust network, diminishing the DirectTrust brand. 18
Next Steps What to test in next round of interop testing? Revoked certificates; lapsed CRLs Dispatched MDNs: DirectTrust will likely require support as of a due date TBD Should only be requested when required (per CLIA/lab reporting, for example) Handling multiple recipients any variations in deployments leading to decryption issues? CCDA and other payloads, possibly EHR to EHR & other than for payload confirmation tests, eliminate human reply component moving forward DNS, LDAP, SMTP related testing 19
The End Questions? 20