Interoperability Test Tools State of the Art and Future. Marco Eichelberg OFFIS Institute for Information Technology

Transcription

1 Interoperability Test Tools State of the Art and Future Marco Eichelberg OFFIS Institute for Information Technology

2 2- Agenda Introduction Survey of test tools available today ( Today ) Tool landscape Observations Needs for future tools ( Tomorrow ) Vision Gaps Conclusions

3 3- Introduction: Interoperability Health IT systems need to exchange and process patientrelated information within a department or institution beyond departmental, institutional or even national borders Interoperability: the ability to exchange and understand communicated information between systems. a prerequisite for a working ehealth infrastructure only established to a limited degree today Reasons for interoperability problems: lack of standards, ambiguous standards, incorrect interpretation of standards. This can be prevented to some degree by adequate interoperability testing of systems before deployment.

4 4- Introduction: Test Tools Test tools: an important factor in conformance and interoperability testing. Think of an IHE Connectathon without MESA & Gazelle Many tools available, but no adequate overview Are there test tools for all relevant protocols and content formats? Are the tools sufficiently stable and well-maintained? Can the tools be integrated into a testing infrastructure? What s missing today? HITCH developed a survey of test tools to identify needs for future tools that allow for better conformance & interoperability testing

5 5- Scope of the Test Tool Survey Look at the big picture beyond tool functionality: Vitality: Maintenance, release frequency, user community Maturity: Quis custodiet ipsos custodes? Consider integration into test management infrastructure Open source tools (license is important!) Open source test scripts with (partly) closed interpreters (TTCN3) Closed source tools with open API Exhaustive survey turned out to be infeasible: Health IT standards like DICOM, HL7 quite complex Health IT also uses generic IT standards like SOAP, TLS, ebxml Therefore, survey focuses on most important tools (not perfect, but better than what we had before )

6 6- Survey Criteria General Tool name, description, author, version, license, programming language Websites Homepage, documentation, source code repository, bug tracker Similar projects Tool category Validator: checks the behavior of the tested communication partner Sniffer: records, decodes and analyzes network traffic Simulator: connection partner for testing peer s communication behavior Library: implementation of standard that can be used to build test tools Test Management Tool: manages test scenarios for peer-to-peer tests Out of scope: tools for performance tests, load tests, security tests.

7 7- Tools Evaluated (1/3) Name of Tool Type of Tool Application Domain AHML Message Testing Validator HL7 CDA Validation NIST Validators IT chiba Library IT dcm4chee Library DICOM DCMTK Library DICOM DICOM Router Tool DICOM dicom3tools Library/Tools DICOM dicom4j Library DICOM DVTk Validator / Simulator DICOM ejbca Library Other Wireshark Validator Other Gazelle Tool Other GDCM Library DICOM GDCM Conformance Tests Testdata DICOM gnutls Validator Other HAPI Library HL7 HL7 Message Maker Tool (incl. Validator) HL7 HL7 Insight Tool HL7 HL7 Inspector Tool HL7 HL7 Validation Nist Validator HL7

8 8- Tools Evaluated (2/3) Name of Tool Type of Tool Application Domain Iheos Library IT MESA Validator / Simulator / HL7/DICOM/IT Reference Implementation Microsoft Connected Health Library/Tools IT Platform Solution Accelerator MirthConnect Tool (Message Broker)HL7/DICOM/IT MWB HL7 HL7 Nule.org: HCM (HL7 Comm Tool HL7 Multi) Nule.org: HL7 Anon Tool HL7 Nule.org: HL7 Browser Tool HL7 Nule.org: HL7 Diff Tool HL7 Nule.org: HL7 Comm Tool HL7 Nule.org ::HL7 Snip Tool HL7 OHT :: mdht Library + Validator HL7 OHT :: Open Health Library + Validator HL7 Workbench Open ehealth Foundation Implementations HL7/IT

9 9- Tools Evaluated (3/3) Name of Tool Type of Tool Application Domain Openca Library Other OpenHealthTools Implementations HL7/IT pixelmed Library/Tools DICOM SOAPUI Simulator / Validation Other TestLink Tool Other The Eurorec Use Tools Tool Other TTCN3 Tool HL7 Web Services Validation Tool Validator Other for WSDL and SOAP

10 10- Results Database at

11 11- Results Database: Example

12 12- Survey Criteria: Radar Plot Categories The soft factors besides tool functionality were assessed in 7 categories Development status ( maturity ): in development, beta, in production Usage: no / few / many users or project use the tool Reactivity: how fast do the developers react e.g. to bug reports Support: means of user support (forum, mailing list, hotline ) Community: size of community maintaining the tool Quality Management: use of QM tools by the developers Openness: open source status, license Rating: 1=weak, 2=intermediate, 3=strong Subjective rating by expert consensus

13 13- Radar Plots (Examples)

14 14- Test Tool Survey: Landscape Privacy Security Health Information Data Structure and Terminologies Patient and Providers Identification Health Info Exchange Infrastructure Test Management Tools Health Service Management (e.g. WS/XDS, SMTP) Logical Transport (e.g. TCP/IP) Physical Transport (e.g. Ethernet, Bluetooth) Interoperability Test Tools (simulators, validators, etc.) Test processes, Languages, Methodologies Green=mature, orange=intermediate, yellow=immature

15 15- Test Tool Domain Map: HL7

16 16- Test Tool Survey: Observations 1 The more mature a standard or and specification is, the higher is the quality of test tools. As the maturity of a tool increases, so does QM (vice versa?) Some test tools are turn-key solutions, other are toolboxes Generic toolboxes may require significant configuration / scripting Turn-key solutions may not be possible for rather generic standards Tool support for the following domains is weak so far: Test management tools (IHE Gazelle, EuroRec UseTools) Tools for processes and methodologies Tools for health information exchange infrastructure TTCN3 a future basis for ehealth test management? ETSI proof-of-concept during the 2010 Connectathon.

17 17- Test Tool Survey: Observations 2 When standards are only available for a fee, implementers tend to use public final draft bad for interoperability. Free availability of standards is preferable (e.g. DICOM, IHE) Interfaces are not an innovative aspect where vendors need to differentiate their products from each other. Common implementations (toolkits) make life easier for everyone Common test tools make life easier for everyone Complex & evolving standards require complex & evolving test tools. Testing tools require an ecosystem of developers, users and funding for their maintenance. This is a challenging task!

18 18- Strategic Vision What is needed to solve the interoperability challenge in the ehealth sector? International agreement on common standards (or at least cross-references/mappings between standards) Close cooperation between political mandates, SDOs, testing/auditing organizations and implementers. Standard development always accompanied by test tool development. Regular testing events like IHE Connectathon. Quality labeling/certification programs that include system testing. Alignment of functional auditing and technical interoperability testing, based on realistic & commonly agreed use cases. A testing tools ecosystem

19 19- Technical Vision: Tool Development Test Plan Definition We need test management tools that support The creation and organization of test criteria (functional statements and interoperability criteria). Links between criteria and the underlying standards The composition of test plans from those criteria. Test Design We need test design tools that combine functional requirements and interoperability criteria into a test case that can be assigned to a number of test partners and test tools (simulators, validators). The tools should be aware of the concepts to be tested (model aware) in order to guide the test designer

20 20- Technical Vision: Tool Development Test Tool Development and Selection Develop those test tools still missing today (gap analysis) Improve existing tools where needed Test Case and Test Tool Validation Make sure test software works correctly and tests comprehensively, document in validation report Maintain bug tracking tools for bugs in test cases and test tools Test Session Execution Provide training material for auditors and testers Integrate test tools with test management tool

21 21- Technical Vision: Tool Development Test Session Reporting Evaluation of the system under test: features tested, failed tests, list of non-conformities, 3rd party systems tested with Evaluation of the test session: number of tests performed / verified / failed / not verified Evaluation of the monitor work: was staffing sufficient? Evaluation of the testing quality: Satisfaction questionnaire for participants and monitors Report to the QA management in order to identify weaknesses and improvements for the next cycle.

22 22- Conclusion Driver Projects are important for implementing the vision of an interoperable ehealth ecosystem epsos defines typical ehealth use cases and provides technical specifications for tasks like eprescription, edispensation etc. SEHGovIA proposes solutions for policy, legal & regulatory and governance issues in cross-border ehealth communication Few successful international initiatives in ehealth testing exist, namely IHE and EuroRec. Approaches and concepts of these organisations need to be aligned. Test tools are only one building block for interoperability, but arguably among the most important. Deserve more attention than they had in the past.