Present an overview of the Meaningful Use Certification process

Transcription

1 EHR Conformance Testing for Enhanced Syndromic Surveillance System Interoperability 2013 International Society for Disease Surveillance Conference Translating Research and Surveillance into Action, New Orleans. Robert Snelick, National Institute of Standards and Technology (NIST) Charlie Ishikawa, International Society for Disease Surveillance (ISDS) Rebecca Zwickl, International Society for Disease Surveillance (ISDS) Sheryl Taylor, Booz Allen Hamilton (BAH) December 11 13, 2013

2 Objectives Present an overview of the Meaningful Use Certification process Focus on the NIST/ISDS/CDC Collaboration to build the certification test procedure, test cases, and test tool Identify key aspects of the Syndromic Surveillance ONC criterion and test procedure Describe the Test Tool Describe context-free and context-based testing approaches Present the test scenarios and test cases Illustrate the use of the certification test tool Report Syndromic Surveillance certification testing results Provide Resources 2

3 Outcomes Improved Syndromic Surveillance Standard Feedback provided from extracting and encoding requirements Test Scenarios and Test Cases Real world scenarios and data that demonstrate the use of the standard Examples messages that are conformance to the standard Validation Suite Syntactic and Semantic Testing Is being used Nationally by the ATLs to test EHR technologies for meaningful use certification EHR vendors while developing their technologies Locally by healthcare providers to validate electronic messages before sending their Syndromic Surveillance system data to Public Health Authorities Is helping drive the industry toward the level of interoperability needed to promote efficient reporting and utilization of syndrome-based public health surveillance information 3

4 Overview: Meaningful Use Certification Process ONC Criterion ISDS/CDC Feedback Addenda & Clarification Documents Feedback Syndromic Surveillance Implementation Guide Use Cases Conformance Requirements NIST Test Procedure Test Cases NIST Test Tool Test Cases ATLs Test Script Test Procedure NIST Test Tool ATLs Certification Testing ACBs Certification ONC Post CEHRT on CHPL NIST Test Stories Test Data ISDS SMEs Local Site Testing ISDS International Society for Disease Surveillance NIST National Institute of Standards and Technology ACB Authorized Certification Body SME Subject Matter Expert CHPL Certified Health IT Product List CDC Centers for Disease Control and Prevention ATL Accredited Testing Laboratory ONC Office of the National Coordinator for Health Information Technology CEHRT Certified Electronic Health Record Technology 4

5 Overview: Meaningful Use Certification Process ONC Criterion NIST/ISDS/CDC Collaboration ISDS/CDC Test Stories Addenda & Clarification Documents Feedback Syndromic Surveillance Implementation Guide Use Cases Conformance Requirements NIST Test Procedure Test Cases NIST Test Tool Test Cases ATLs Test Script Test Procedure NIST Test Tool ATLs Certification Testing ACBs Certification ONC Post CEHRT on CHPL NIST Test Stories ISDS SMEs Test Data Local Site Testing What we are covering today ISDS International Society for Disease Surveillance NIST National Institute of Standards and Technology ACB Authorized Certification Body SME Subject Matter Expert CHPL Certified Health IT Product List CDC Centers for Disease Control and Prevention ATL Accredited Testing Laboratory ONC Office of the National Coordinator for Health Information Technology CEHRT Certified Electronic Health Record Technology 5

6 Syndromic Surveillance Certification Criteria and Testing For certification, message creation function must conform to standards Capability Testing (not site testing) 2014 Certification: (NIST Test Tool Target) Inpatient EHR Technology = Hospital (e.g., ED) HL format PHIN Guide for Syndromic Surveillance Release 1.1 PHIN Addendum to Release 1.1 Conformance Clarification PHIN Testing Clarification Document Ambulatory EHR Technology = Outpatient EP s, Urgent Care) (e.g., HL format (Mandatory, but no Test Tool) PHIN documents (Optional, use Test Tool as above) 2011 Certification: HL or format Too broad; no conformance test tool) Test Procedure Scenario-based Message focused (A01, A03, A04, A08) Inpatient procedure Three different scenarios, three test cases each Ambulatory procedure Urgent care scenario only, three test cases (optional) 6

7 Why is Testing Important? Technology standards frequently implemented with inconsistencies due to Guidance ambiguities Misinterpretations Feedback Loop Provides confidence that implementation is correct Mandating a standard does not ensure Use of standard as intended Conformant Systems Interoperability of EHR Syndromic Surveillance data among systems Conformance testing to the standard early in technology development life cycle can Increase probability of conformance to standards Facilitate interoperability Increase product reliability Reduce overall costs 7

8 TEST TOOL 8

9 Messaging Validation Suite Overview Purpose: This tool validates Syndromic Surveillance messages created by Electronic Health Records (EHR) technology. Key Capabilities Context-free Testing Context-based Testing Profile Viewer Vocabulary Browser Documentation (No Test Cases - Test any Syndromic Surveillance message created by EHR) Context is unknown to validation tool Provides a simple and convenient method for testing message structure and most vocabulary (Test Cases - Test Syndromic Surveillance message associated with a specific test scenario) Context is known to validation tool All conformance requirements of the SS Implementation Guide can be assessed Used for certifying 2014 Edition Meaningful Use EHR technology Provides a browsable version of the conformance profile which encapsulates the requirements. Can be used to assist in the interpretation of errors. Provides a browsable view of the vocabulary requirements. Can be used to assist in the interpretation of value set errors. Provides access to documents which will assist in using the tool (including test procedure, test cases, profile descriptions, vocabulary descriptions and validation tool download). 9

10 10

11 CONTEXT-FREE VALIDATION 11

12 Context-free Testing No Test Cases provided Context (Test Scenario, etc.) is unknown to validation tool May be used to test any syndromic surveillance message created by an EHR Provides a simple and convenient method for testing message structure and most vocabulary Typically is not used for certifying EHR technologies for the ONC certification criteria, but may be used for certification testing in specific instances (the Tester must perform visual inspection to validate content of message) 12

13 Syndromic Message Testing - Context-free Validation 1. Click on the Context-free Validation tab. 2. Select profile to use to validate the message. 5. Syndromic Test Tool produces Validation Report. 4. Validate the message. 3. Browse for message to be validated and import it. 6. Tester analyzes Validation Report. 13

14 Import test message 1 2 Open Syndromic Surveillance Validation tool using link: Click on Context-free Validation tab. Context-free Validation page This page validates any Syndromic message. It is disassociated from a test script, test case, or specific content. Testing will include the technical requirements and contentspecific requirements specified in the selected profile Select conformance profile to use to validate the message. Click on Browse Message button to load message into Message Content window. Message Uploader Dialog window displays. Click on Select Message button. Open test message file to upload it. 14

15 Validate test message and review message validation errors 1 Uploaded test message displays. 2 If message fails validation, errors will display in Message Validation Result section of page. 3 Click on location link to highlight the data element causing the error within the Message Tree and Content. Note: Location link may not be available if the message element location does not map to a message element in the message tree. 15

16 Validate test message and review message validation errors (cont d) 4 Total number of errors displays. 5 Description explains why error occurred. Format or table of data element may be provided. 6 Line number and column of errors are provided. 7 Location link displays the location of the data element. 8 Click on page numbers to view additional pages of errors. 16

17 Profile Viewer - look up data element information 1 Click on Profile Viewer tab. 2 Click on a tab to select the conformance profile. 3 Data elements may be filtered by Usage. Select R, RE, C (Only) to view only required, required but may be empty and conditional elements. Select R, RE, C, O, X (All) to view all data elements, including optional and not supported elements. 4 Click on a tab to filter data elements by segment. The Full tab displays data elements of all segments. 5 Locate data element using element name from the location link. 6 Usage will indicate whether data element is required. The value R indicates it is required. 7 Minimum and maximum valid lengths of data element are listed. 8 Make a note of the table ID. Profile Viewer page This page allows tester to view data element information including usage, cardinality, data type, length, table, condition predicate and conformance statements. 17

18 Vocabulary - look up tables and valid data element values (cont d) 9 Click on Vocabulary tab. 10 Locate and select table ID. 12 Use the search box to search for value code, value set, table name, table ID or description. 11 Valid values are listed. Vocabulary page This page provides the ability to browse the vocabulary requirements. Search capabilities are provided and include searching on value, table name, table ID and description. 13 Click on Context-free Validation tab to return to message. 18

19 Documentation Documentation page The Documentation page provides the ability to download all documents and files for testing, including Data Spreadsheet, Test Cases, Profile and Vocabulary Description files, and validation tool application file (.war file). 19

20 CONTEXT-BASED VALIDATION 20

21 Context-based Testing (Sending Application) Test Case Test Story Implementation Guide Requirements Sets the clinical backdrop Test Tool Tester Test Data EHR Enters Test Data HL7 V2 Message Validation Validation Report Typical data available in the real world Content Requirements Tester 21

22 Context-based Testing Test Cases provided Context (specific Test Scenario, etc.) is known to validation tool Messages created by the EHR technology are populated using the test data provided in the Test Cases associated with specific Test Scenarios Provides a method for testing / assessing a syndromic surveillance message for all conformance requirements of the Syndromic Surveillance Implementation Guide Is used for certifying EHR technologies for the ONC certification criteria Significantly expands the scope of testing Usage: Required, but may be empty (RE), Conditionals (C) Cardinality: Ranges Length: Ranges Vocabulary Helps Interpretation and Use of the Standard 22

23 Test Story 23

24 Test Data 24

25 Message Content 25

26 Syndromic Test Scenario Matrix UC Visit Pt Death Pt Admit ADT A04 Registration X X X ADT A03 Discharge X X X ADT A08 Update X X ADT A01 Admit Facility ID is from the treating facility X X X Report Date/Time is the date/time that information was sent from the treating facility; will vary among messages for a visit Visit Date/Time, Unique Visit ID, and Unique Patient ID are consistent among messages for a single visit X X X X X X X Patient Age is sent in years X X X Free-text Chief Complaint can be sent X X If patient presents unconscious, patient demographics (e.g., Race, Ethnicity, Zip Code, Age, and Patient's Chief Complaint) are null or a default value A coded Reason for Visit can be sent When multiple Diagnoses are made, the primary is sent first X X X Diagnosis ECODEs can be sent X X Discharge Disposition of death can be sent X X X 26

27 Scenario #1 Urgent care Test Scenario #1 Routine Urgent Care A. Infant with influenza-like-illness B. Adult with a urinary tract infection C. Adult with broken arm Test Scenario #1 Messages ü ADT A04 and A03 ü Free-text chief complaint 27

28 Scenario #2 Patient dies Test Scenario #2 Patient Death A. Skill fracture w/ intracranial injury from bike accident B. Acute respiratory failure due to house fire C. Subarachnoid hemorrhage due to fall Test Scenario #2 Messages ü ADT A04, A08, and A03 ü No demographic data or chief complaint in A04 ü Diagnoses include ECODE ü Patient death sent in final message 28

29 Scenario #3 - Admission Test Scenario #3 Patient Admission A. CO-poisoning during severe whether power outage B. Critical dehydration due to cryptosporidium outbreak C. Pneumonia during flu season Test Scenario #3 Messages ü ADT A04, A08, A03, and A01 ü Free-text chief complaint ü Hospital admission 29

30 Syndromic Certification Testing - Context-based Validation 30

31 1 Select Test Case Open Syndromic Surveillance Validation tool using link: 2 Click on Context-based Validation tab. 5 Test Case may be downloaded as a PDF file. 3 Click on arrows to expand the first Test Scenario. Expand the Test Cases and select a Test Case. 4 Test Case displays. 31

32 Select Test Step 1 and review Test Story 3 Step 1 Test Story may be downloaded as a PDF file. 4 Step 1 Test Package may be downloaded as a PDF file. 1 Select Step 1 of Test Case. 2 Test Story displays. 32

33 Review Test Data Specification and Message Content 1 Click on Test Data Specification tab. 4 Test Data Specification may be downloaded as a PDF file. 2 Full tab is selected by default. To view a specific section, click on a tab. 3 Relevant real-world clinical data is displayed. Test Data Specification page This page specifies the data that are entered (automatically/manually) into the EHR and included in the message that is created and submitted from the EHR to the Syndromic validation tool. The tester shall identify an existing patient record in the EHR or shall create a patient record in the EHR using the data in the Data Sheet associated with the Test Case. 33

34 Review Test Data Specification and Message Content (cont d) 5 Click on Message Content tab. 6 7 Location specifies the location of a data element within the message. Name of data element is provided. 10 Message Content may be downloaded as a PDF file. 8 Exact value of the data element for the selected Test Step is provided. 9 Categorization indicates if the data is fixed or can be changed. Message Content page Create test message 1 Using the EHR technology, create the Syndromic Surveillance test message with the test data provided for the selected Test Step (step 3). This page provides details of the message segment, as well as an example of a valid instance of each data element. 34

35 Load Test Step and import test message 1 Click on Select button to load the Test Step. 2 Validation page displays. 3 Click on Browse Message button. 4 5 Message Uploader Dialog window displays. Click on Select Message button. 6 Open test message file created in step 4 to upload it. 35

36 Validate test message and review validation errors 1 Uploaded test message displays. 2 If message fails validation, errors will display in Message Validation Result section of page. 3 Click on location link to highlight the data element causing the error within the Message Tree and Content. Note: Location link may not be available if the message element location does not map to a message element in the message tree. 36

37 Validate test message and review validation errors (cont d) 4 Total number of errors displays. 5 Description explains why error occurred. Format or table of data element may be provided. 6 Line number and column of errors are provided. 7 Location link displays the location of the data element. 8 Click on page numbers to view additional pages of errors. 37

38 Generate Message Validation Report 1 Click on Report Details icon to generate a report. 2 Message Validation Report may be downloaded as a PDF, XML, Word doc, HTML file. 3 Message Validation Report may be printed. 38

39 RESULTS 39

40 Results As of November vendors/organizations certified EHR technologies for Syndromic Surveillance messaging ATLs tested these EHRs using NIST Syndromic Surveillance Test Tool Certified EHR Technologies are listed on the ONC Certified Health IT Product List (CHPL) at: Feedback about Test Tool indicates EHR technology vendors and other groups Have been motivated to think through the steps for capturing/reporting Syndromic data in more detail than before Are challenged by the new Syndromic Surveillance data requirements and standards Appreciate the subject matter expertise that generated the tools Plan to use these tools locally to help real-world implementations 40

41 Vendors/Organizations with Certified EHR Technologies As of November 2013 Agastha, Inc. Allscripts AmkaiSolutions LLC Plexus Information Systems, Inc. athenahealth, Inc General Electric Healthcare IT Dynamic Health IT, Inc. Corepoint Health CPSI (Computer Programs and Systems), Inc. Henry Schein Practice Solutions, Inc. DigiDMS, Inc. eclinicalworks LLC EMD Wizard, Inc. Epic Systems Corporation T-System Technologies, Ltd. gmed, Inc. Glenwood Systems LLC Greenway Medical Technologies, Inc. Healthcare Management Systems, Inc. Cerner Corporation InterSystems Corporation CareEvolution, Inc. McKesson ipatientcare, Inc. Medflow, Inc. SuccessEHS, Inc., a Division of Vitera Healthcare Solutions LSS Data Systems Advanced Data Systems Corporation Medical Information Technology, Inc. (MEDITECH) HealthFusion Henry Schein Medical Systems, Inc. Insight Software, LLC NextGen Healthcare Northwestern University Bogardus Medical Systems, Inc. NTT DATA, Inc. Aprima Medical Software, Inc. Iatric Systems, Inc Pulse Systems Orion Health SRSsoft Ulrich Medical Concepts, Inc. The Shams Group Truven Health Analytics VisionWeb Computer Sciences Corporation (CSC) Vitera Healthcare Solutions, LLC Wellsoft Corporation From ONC CHPL at: 41

42 Outcomes Improved Syndromic Surveillance Standard Feedback provided from extracting and encoding requirements Test Scenarios and Test Cases Real world scenarios and data that demonstrate the use of the standard Examples messages that are conformance to the standard Validation Suite Syntactic and Semantic Testing Is being used Nationally by the ATLs to test EHR technologies for meaningful use certification EHR vendors while developing their technologies Locally by healthcare providers to validate electronic messages before sending their Syndromic Surveillance system data to Public Health Authorities Is helping drive the industry toward the level of interoperability needed to promote efficient reporting and utilization of syndrome-based public health surveillance information 42

43 Resources for Syndromic Surveillance Test Tool Test Procedure f3transmissiontopubhealthsyndsurv_2014_tp_approved_v1.3.pdf Test Tool Google Group Implementation Guide PHIN_MSG_Guide_for_SS_ED_and_UC_Data_v1_1.pdf ONC CHPL 43

44 44