Portavita Benchmark: A Dataset Generator for Healthcare

Transcription

1 Portavita Benchmark: A Dataset Generator for Healthcare Editors Albana Gaba, Yeb Havinga, Tom van der Weide License Creative Commons, Attribution-ShareAlike Date Februari 3, 2015 Contributors Albana Gaba, Yeb Havinga, Tom van der Weide, Jasper Visser, Evert Jan Hoijtink, Heimen Brons, Jan Willem Kijne, Pieter Spoelstra (Portavita) - Willem Dijksta, Fabian Walraven (MGRID) The research leading to these results has received funding from the European Union s Seventh Framework Programme (FP7/ ) under grant agreement nr

2 Contents 1. Summary 5 2. Introduction 5 3. Background on healthcare information model standards HL7 Version Clinical Document Architecture CDA Fast Healthcare Interoperable Resource FHIR Building data models Overview of Portavita s data representation Organizations Roles Treatments Examinations Modeling Clinical Data Modeling organizations Modeling Patients Modeling examinations frequency Modeling Examinations The problem of missing values Limitations Dataset generator Synthetic data generation process Validation Single variable Multi-variable Performance evaluation Portavita Benchmark v Portavita Benchmark v1 Performance evaluation Portavita Benchmark v Portavita Benchmark v2 Performance evaluation Conclusions on performance Benchmark queries Queries overview Conclusions 27 A. Clinical Document Architecture 29 2

3 B. List of Observations per Examination 32 Bibliography 43 List of Figures 3.1. Simplified representation of the RIM classes UML model of the FHIR Observation resource Data generation process Organization hierarchy Treatment representation An examination representation Steps for generating a synthetic dataset with only one organization Discrete real data Discrete synthetic data Continuous real data Continuous synthetic data Discrete real data Discrete synthetic data Continuous real data Continuous synthetic data Imputed missing real data Imputed missing synth data Co-missing real data Co-missing synthetic data The components involved in the batch processing architecture of Portavita Benchmark v Performance of Portavita Benchmark v1 grouped by component. Time required to create 1GB of synthetic data for various dataset sizes Portavita Benchmark v2. Micro-batch architecture Performance of Portavita Benchmark v2. Time required to generate 1GB of data Comparative performance evaluation. Amount of documents loaded per second for v1 and v

4 List of Tables 4.1. Data used to model a patient in relation to a treatment Data used to model the distribution of examinations per patient for each year An example of lab examinations. Each row represents a lab examination instance

5 1. Summary In this document we introduce Portavita Benchmark[6], a data generator for benchmarking on healthcare data. The generator is based on statistical models of anonymized clinical data from Portavita s care management system. It generates organizations, practitioners and nearly 50 different kinds of examinations that consist of 940 different kinds of observations. The data generated is fully compliant with the HL7 healthcare interoperability standards. Portavita Benchmark includes both clinical document generation and transformation to relational database persistence, and generates up to 1TB/hour of clinical documents. Benchmark queries are included for database performance measurements. 2. Introduction To develop effective techniques for processing and storing large medical data, it is often required to evaluate and compare the performance of these systems. In this document we introduce Portavita Benchmark, the first dataset generator specific for healthcare data. It uses real clinical data for building models, which are then used to create arbitrarily large synthetic datasets. A notable aspect that distinguishes medical data from other data is the use of domainspecific conventions, typically put forward by standards organizations. Standards are instrumental as health data is exchanged between various healthcare organizations, or even between various departments of the same organization. They provide, among others, well-defined information models and specific names for each medical concept. Portavita Benchmark is composed of two parts: data modeling based on existing clinical data and data generation. In the first part we see how data is structured in the Portavita care management system, compliant to Health Level 7 (HL7) standards. In the second part, synthetic health records are generated in two exchangeable HL7 formats, namely CDA and FHIR. This process is followed by the transformation and storage of these documents into a PostgreSQL DBMS. Finally, we provide a brief overview of the queries that come with the Portavita Benchmark for benchmarking purposes. 5

6 3. Background on healthcare information model standards Standards play a fundamental role in facilitating the interoperability between various healthcare organizations spread across different countries. They determine common ways to model data and to express domain-specific concepts. In this chapter we introduce three important Health Level 7 information models which are widely adopted by the healthcare community. Portavita Benchmark heavily relies on them, as we will see in the next chapters HL7 Version 3 HL7 Version 3 provides a object-oriented development methodology based on a reference information model (RIM). The RIM is an essential part of the HL7 Version 3 as it provides a universal information model for healthcare interoperability, covering the entire healthcare domain [2]. In the RIM there are six high-level concepts to describe all clinical data: entities, roles, acts, participations, role links, and act relationships, as shown in Figure 3.1. There are several specializations of the main classes. We describe briefly the most important classes that we use in our data model. Entity classcode name Role Participation classcode typecode classcode code effectivetime moodcode 1 0..n effectivetime 1 0..n 0..n 1 effectivetime 1 0..n confidentiality confidentiality Code Code statuscode negationind Act Act- Relationship Person Organization CareProvision Observation Substance- Administration Employee Patient VIPCode Assigned- Entity Figure 3.1.: Simplified representation of the RIM classes. Entities can be organizations or persons. A role can be played by an entity and scoped in another entity. For example, patient is a role that is played by a person (an entity) and is scoped by an organization (also an entity). Acts describe events. For example, 6

7 observations and examinations are acts. A role participates in acts. For example, a patient role can participate as a subject in an observation and a practitioner role can participate as a performer in an observation. Specializations, e.g., Observation, inherit properties from the class they specialize from, e.g., Act, while they have their specific attributes Clinical Document Architecture CDA Clinical Document Architecture, often referred to as CDA, like any clinical documentation, is used to describe care provided to a patient, to maintain a patient medical record and to exchange information between healthcare providers. The CDA is a XML-based markup standard, based on HL7 Version 3 RIM, and as such, it fully maps clinical data modeled with the RIM. A CDA document is comprised of two parts. The header contains contextual information, such as, the patient it applies to, the organization and the person who wrote it and the time when the document was written. The body contains humanreadable narrative text and optional structured clinical statements, including act, observation, substance administration, encounter, procedure, organizer and supply [3]. Appendix A illustrates an example of a CDA document Fast Healthcare Interoperable Resource FHIR FHIR has been introduced by HL7 as a next generation standards healthcare framework. It combines the best features of HL7 s Version 2, Version 3 and CDA product lines while applying a tight focus on easing the exchange of clinical documents and implementability. For example, FHIR provides highly modular components called Resources which can be easily assembled in a way that they can fully represent clinical data. Most resource elements and data type properties include mappings to the RIM. So, just like with the RIM components, FHIR provides resources for modeling RIM classes, like Observation, Patient, Organization and so on. Figure 3.2 shows the UML diagram of an Observation FHIR resource. Similarly to an observation RIM class, it has references to the patient, the performer, the organization, and among other fields, it includes the observation code and the corresponding value. FHIR resources are easily accessible in a wide variety of contexts, including mobile phone applications. In particular, FHIR provides additionally a RESTful API which defines a set of common interactions performed on a repository of typed resources (read, update, search, etc). These interactions follow the RESTful paradigm of managing state by Create/Read/Update/Delete actions on a set of identified resources [4]. 7

8 Figure 3.2.: UML model of the FHIR Observation resource. 8

9 4. Building data models An important aspect of Portavita Benchmark is that it generates data that resemble real clinical data. Figure 4.1 shows the processes involved by Portavita Benchmark to generate synthetic datasets based on real data. First, we retrieve the necessary data, according to the models we aim to create, and aggregate them into csv documents. Further, the aggregated data is used to create models by training Bayesian Networks. These models are finally used by the generator to create arbitrarily large datasets of CDAs and FHIR documents. We start by introducing the way data is structured originally in Portavita. This will help to understand later the structure of the synthetic datasets. Portavita Database Aggregate Create Bayesian Networks Dataset-generator CDA / FHIR Figure 4.1.: Data generation process Overview of Portavita s data representation Portavita provides a care management system for treating patients with chronic conditions. The treatments covered by Portavita are numerous, but for the purpose of Portavita Benchmark we consider only data from diabetes, COPD 1 and CVRM 2 treatments, which are the ones with the highest number of records. Portavita uses HL7 Version 3 RIM data models to represent clinical data. This section gives a short overview of the main data concepts, which are also represented in the data generated by Portavita Benchmark. 1 Chronic Obstructive Pulmonary Disease 2 CardioVascular Risk Management 9

10 Organizations The organizations of Portavita s customers are organized in hierarchies. For Portavita, a top-level organization is called a care group. A care group has a number of suborganizations, each of which can have sub-organizations on its own. As such, this structure can be represented as a hierarchy. An example hierarchy is given in the following figure. Care group GP Surgery GP Surgery Pharmacy Dietitian GP Surgery Figure 4.2.: Organization hierarchy Roles There is a large variety of roles played by users in Portavita s system. Here are some of the most relevant for Portavita Benchmark. A care group employee is a role played by users that work for the care group and act as a superusers. One of the main activities is monitoring organizations within the care group. A quality employee is a role that a user can have, which allows them to compare practitioners within an organization or set of organizations. A practitioner is a care provider that can be active in the treatment of patients. A patient is a person who receives treatment. A researcher is a person who is granted access to part of the data to perform research. Note that a person can play multiple roles inside the organization. For example, a practitioner could also be a patient within the same organization Treatments When a new patient is entered in the system, a treatment such as diabetes or CVRM, is assigned to them. Every treatment has at least two participations: a subject participation (i.e., patient), and a performer participation (i.e., principal practitioner). The principal practitioner has the final responsibility over the treatment. The diagram in Figure 4.3 illustrates the relationships between a treatment (a type of act), examinations (also acts) performed in the context of a treatment, and the patient 10

11 and principal practitioner participations in the treatment. It is important to note that typically each observation performed to a patient is linked to a care treatment the patient is part of. Patient Treatment Principal practitioner Examination Examination Examination Figure 4.3.: Treatment representation Examinations Examinations make up the largest part of the data in Portavita s database. They are typically performed in the context of a treatment. For example, in a diabetes treatment, a patient typically performs examinations on a regular basis, such as yearly checkups, foot examinations, eye checkups and so on. The general structure of an examination is shown in Figure 4.4. An examination has at least four participations: a subject participation (the patient), a performer participation (the practitioner who performs the examination), a data enterer participation (the person who enters the data into the system), and a legal authenticator participation (the practitioner who accords the data that was entered). Most often, though, a single practitioner participates in the role of performer, data enterer and legal authenticator. Performer Patient Examination Data enterer Organizer Legal authenticator Organizer Observation Organizer Observation Observation Observation Observation Figure 4.4.: An examination representation. Examinations consist of simple observations and organizers, which contain observations grouped together in meaningful sets. Typically, an examination has a small number of observations that are mandatory. However, larger examinations may 11

12 consist of up to a hundred of different observations. Typically, several observations can only be entered if certain other observations have been made. For example, only if a patient has a skin rash, the performer can enter details about the skin rash Modeling Clinical Data We create models of clinical data by training Bayesian Networks upon Portavita production data. Bayesian Networks capture dependencies between various variables and distribution of their values. For each concept to be modeled, we aggregate the necessary data by querying the production Portavita database. The queries are performed on anonymized patient data. We create models for the following concepts: organization, patient, examination, and treatment. We use the R package bnlearn for training Bayesian networks. To train Bayesian Networks it is important to differentiate between discrete and continuous variables. For example, a discrete variable is used to express whether a patient smokes, with possible values yes, no, or in the past. A continuous variable is, for instance, HbA1c which comprehends values in a certain range. Integer numbers (e.g., number of days) are considered as continuous values by bnlearn Modeling organizations To generate an organization with a certain patient population size and number of practitioners, we learn a mixture Gaussian from the sizes of the actual organizations from one caregroup. For this, we use the R package called mixtools Modeling Patients A patient is represented by his age, treatment he is part of and the duration of such treatment. Therefore, for each treatment we train Bayesian networks using the variables as shown in Table 4.1. Field Patient Age (days) Treatment Treatment duration (days) Type Continuous Discrete Continuous Table 4.1.: Data used to model a patient in relation to a treatment. 12

13 Modeling examinations frequency To capture the dependency between the age of the patient and the kind of examinations that are performed in one year, we train Bayesian networks using the patient s age, and for each type of examination, the number of examinations that were performed in that year. Table 4.2 shows the data used to model examinations. Field Type Patient Age (days) Continuous # Examinations A Continuous # Examinations B Continuous Table 4.2.: Data used to model the distribution of examinations per patient for each year Modeling Examinations There are about 50 examinations in Portavita s system for the treatments we have considered, such as yearly check up, foot checkup, lab examinations and so on. A list of examinations is provided in appendix B. We create a model for each of them. For every kind of examination that can be entered into the system, a query is performed that returns a table with a column for each possible observation value. In addition, for every examination instance, a single row is returned, which contains a cell for every observation that could be measured within that examination. As an example, let us look at the Lab examination. On a regular basis, the blood of the patient must be examined. When performing a lab examination, a blood sample is collected and sent to the lab. The lab examines the blood on a subset of around 30 measurable variables, such as HbA1c and Cholesterol. Not all variables are always measured, but only those that were requested. Furthermore, the data contain both continuous, discrete variables, or a missing value, as shown in Table 4.3. HbA1c HDL LDL Triglyceride.. Albumin Table 4.3.: An example of lab examinations. Each row represents a lab examination instance. The R package bnlearn cannot be used to learn hybrid models, i.e., models containing both discrete and continuous variables. We have tried other packages such as deal, but without success. The number of variables was too big (e.g. 80) resulting in the package being unable to allocate an array of the right size. Because the data 13

14 consist of discrete and continuous values, we came up with the following solution. Namely, for every examination we train three Bayesian networks: network for discrete values (missing values are represented explicitly) network for continuous values, and network for the missing values in continuous variables Note that missing values for discrete variables can be seen as simply another value, but that for continuous variables this is not the case. To generate examinations that have similar patterns of missing values, we therefore must train another network for the patterns in the missing data of continuous variables The problem of missing values Because the input data contain many missing values and many machine learning algorithms cannot deal with missing values, it is important to impute the data. To keep the performance within reasonable levels, we use two packages in R for imputation: mice and imputation. We start with the mice imputation method called norm predict. This method calculates regression weights from the observed data. However, the result of performing this method may still contain missing values. If this is the case, then we continue with the imputation package. If there are more than 1000 rows in the data, then we use the method gbmimput, which uses boosted regression trees for each column x to predict x using all other columns except x. GBM impute is only used when the dataset is large enough, otherwise it does not work well. If there are less than 1000 rows, then we use lmimput, which fills missing values in a column by running locally weighted least squares regression Limitations The models created to represent the original data determine the quality of the synthetic dataset. Therefore, it is worth recalling the limitations of the models built throughout this stage. Hybrid Bayesian Networks The networks we have trained, and hence modeled, are separate for continuous (or numeric) values, discrete and missing values. It implies that within an examination there may be discrete variables that are inconsistent with other continuous values. For example, discrete variable issmoker may be no, while the number of cigarettes smoked per day may be a non-zero integer. Time-based observations per patient Subsequent examinations related to a single patient are not co-related. This means that observations concerning a single patient over time are not consistent. Imputed missing values Since a number of observations were imputed, the accuracy of the models including such observations may be affected. As a consequence, the values generated may be less representative of the original data. 14

15 Natural numbers The models do not discern between natural and real numbers. As a result, all generated numeric data are real numbers. For instance, smoking daily units are natural numbers in the original data, but real numbers in the synthetic dataset. Another consequence is that negative numbers are created in the synthetic dataset, even when the data type in the original dataset has no negative values. 15

16 5. Dataset generator 5.1. Synthetic data generation process The generator uses the model of a healthcare organization, as described in Section 4.1.1, to assign the number of patients and practitioners to a synthetically generated organization. This way, the size of a generated dataset is determined by the number of organizations that are required to be generated (numorganizations). Figure 5.1 depicts the consecutive steps to construct synthetic healthcare information. After an organization is created, it is assigned a set of patients and practitioners. Further on, to each patient p is assigned a number of treatments. For each treatment of patient p, a number of examinations is generated distributed over the time of treatment duration. Finally, for each examination, a practitioner is randomly assigned from the set of practitioners of the organization and, based on the model of the examination, a number of observations types with their corresponding values. 1. Organization Practitioners Patients Treatment Examinations Observations Figure 5.1.: Steps for generating a synthetic dataset with only one organization. The patients and the organizations created by Portavita Benchmark are FHIR resources, while the rest, examinations and observations, are CDAs. Both CDA and FHIR are XML formats Validation In order to validate that the dataset generator produces meaningful healthcare information, we compare synthetic data with real healthcare data. 16

17 Figure 5.2.: Discrete real data Figure 5.3.: Discrete synthetic data Taking into account the way in which separate Bayesian Networks are used to model dependencies between variables, we can expect the following structure in the synthetic data: Correlation between continuous variables in the same examination instance. Refer to Section for a definition of an examination and Appendix B for lists of observations that occur within an examination. Correlation between discrete variables in the same examination instance. Percentage of missing values and correlation between occurrences of missing values. The synthetic dataset generator does not create the full statistical structure that is present in the real data. See section 4.3 for a discussion about limitations of the models. Two kinds of correlations that are not present are worth mentioning: Correlation between different instances of examinations. This means that a patient today can have a irreversible complication such as retinopathy, which is not present during the following examination. A consequence is that aggregation will cause loss of correlation between variables. For instance, there will be no correlation between yearly averages of systolic and diastolic bloodpressures. Correlation between discrete and continuous variables, such as the discrete value smoking y/n and the numeric variable amount of daily smoking units. Since the bayesian network toolkit we used does not support hybrid networks, separate networks for discrete and continuous variables are used to model dependencies, as described in Section Comparison of the real and synthetic dataset is done in four ways, as described in the next two sections. We compare the real data with synthetic data using a single variable for discrete and continuous attributes. We also compare the data sets for dependencies between two variables. We use Orange v2.7 [5] to analyze and visualize the data Single variable The histograms shown in figures 5.2 and 5.3 show the frequency of values for the discrete variable wellbeing of the real and synthetic dataset. Visual inspection reveals that the distribution of values is similar. 17

18 Figure 5.4.: Continuous real data Figure 5.5.: Continuous synthetic data Figure 5.6.: Discrete real data Figure 5.7.: Discrete synthetic data For continuous variables, figures 5.4 and 5.5 show box plots of the blood pressure on real and synthetic data. We can see that the statistical mean and standard deviation for the synthetic and real data is similar. The box plots also reveal a difference in the number of distinct values. As described in section 4.3, the generator makes no distinction between natural and real numbers and treats all numeric data as real numbers. As a consequence, almost every value in the synthetic dataset is unique, whereas observations with a natural number domain in the real dataset contain less distinct values Multi-variable The mosaic diagrams 5.6 and 5.7 give insight into co-occurrences of pairs of values for the discrete attributes exercise and wellbeing. The size of the area indicates the number of samples with the corresponding values in the the dataset. Both graphs show a similar structure for all combinations of values. For continuous variables, correlation between variables is shown using scatterplot. Figures 5.8 and 5.9 show the correlation between systolic and diastolic blood pressure. 18

19 Figure 5.8.: Continuous real data Figure 5.9.: Continuous synthetic data Figure 5.10.: Imputed missing real data Figure 5.11.: Imputed missing synth data Finally we consider percentage and co-occurence of missing values for continuous data. 1 To visualize missing values, we impute missing values for systolic and diastolic blood pressure with the value 300. Figures 5.10 and 5.11 compare frequencies of the missing values for the real and synthetic dataset for 330 samples. We can see that both datasets show a comparable amount of missing values for the systolic blood pressure. Besides amount of missing data, co-occurences of missing values in the real dataset should also be reflected in the synthetic data. Again we use data with missing values imputed to value 300. Figures 5.10 and 5.11 show systolic bloodpressure plotted against diastolic blood pressure. The presence of only a dot at point 300,300 in the graph, but no other dots on the x = 300 or y = 300 line, indicate that a missing value for systolic bloodpressure is always matched with a missing value for diastolic blood pressure, in both the real and synthetic dataset. 1 For discrete data, missing values are modeled with an additional nominal in the value domain, hence require no additional validation. 19

20 Figure 5.12.: Co-missing real data Figure 5.13.: Co-missing synthetic data 5.3. Performance evaluation To perform any operation on the dataset created by the generator, it is necessary to transform and store CDA/FHIR documents in a relational database. To this end, Portavita has delivered two versions of the database generator. This section describes architectural differences between the two versions, and provides results on tests that compare the speed of data generation Portavita Benchmark v1 As shown in Figure 5.14, Portavita Benchmark v1 consists of the following components: 1. Clinical Document Architecture and FHIR XML message generator (genxml) 2. Message converter from XML to SQL (xml2sql) 3. Loading SQL documents in a staging database (sql2db) 4. Update statistics used by the PostgreSQL planner to determine the most efficient way to execute a query (vacuumanalyze) 5. Transformation of staging data to dimensional warehouse format (transform2dimentional) 6. Loading the dimensional warehouse format to the final database. (copy2dwh) CDA/FHIR-generator (GENXML) XML2SQL SQL2DB Transform* Dimensional Data Warehouse Files on disk In DB *Transform to dimensional Data Warehouse Figure 5.14.: The components involved in the batch processing architecture of Portavita Benchmark v1. 20

21 The v1 database generator operates in batches; first all XML documents are created on the file system in step 1. Then the message converter reads the XML files and produces SQL scripts that can be run against a database, and so forth until the last step, that copies data from the staging to the final database. The batch-wise approach has the following drawbacks: Some of the steps are single threaded. The benchmark results show that the sequential steps dominate the generation time of large data sets. The larger batches are, the harder it gets to keep the state between various batches. For instance, using /tmp as storage for temporal steps will cause problems if /tmp is on the root filesystem with limited space Portavita Benchmark v1 Performance evaluation We measured the time it takes to create datasets of different sizes. These tests were performed on the AXLE Manchester server with PostgreSQL 9.5 development version from December 11th The specifications of this server are: 8 x 8 Intel(R) Xeon(R) CPU 2.20GHz 256GB RAM Each data point represents the average result of at least two executions with the same parameters. Figure 5.15.: Performance of Portavita Benchmark v1 grouped by component. Time required to create 1GB of synthetic data for various dataset sizes. 21

22 Figure 5.15 shows the time it takes to create 1 GB in the database for different scalings of , , , and XML documents respectively. The various database scalings are in conformity with the requests made by AXLE partners during internal mailing-list discussions. As Figure 5.15 shows, genxml and vacuumanalyze are the lowest resource-intensive components. In particular, the XML generator genxml requires ca. 3 seconds per GB, which translates to over 1TB/hour of XML data. The most CPU time is spent on tasks 2, 3 and 5. Portavita Benchmark v2 focuses on improving the performance of these tasks, as we will see in the next section Portavita Benchmark v2 The design of the Portavita Benchmark v2 was focused on improving the shortcomings of v1 as follows: 1. Redesign architecture from a batch-oriented to a near-real-time streaming architecture using micro-batches. The purpose-built multidimensional star schema model from v1 was removed; in v2 the HL7v3 RIM model is used directly as the source atomic data of the data warehouse. This resulted in the elimination of the sequential step 5 transform2dimensional. 2. Steps 2 and 3 were already parallelized on a single node using GNU parallel. We analyzed the performance of each component and mitigated performance bottlenecks. In addition, we added a scale-out option for steps 2 and 3, to also go beyond single node performance. CDA/FHIR-generator (GENXML) XML2SQL XML2SQL XML2SQL SQL2DB SQL2DB SQL2DB Data Lake Figure 5.16.: Portavita Benchmark v2. Micro-batch architecture. Together these two design decisions should lead to faster creation of synthetic databases. Nonetheless, scale-out of the system also introduces a new component, the RabbitMQ message broker, and with the broker new configuration and flow control options, that require configuration and monitoring to reach maximum throughput Portavita Benchmark v2 Performance evaluation In Portavita Benchmark v2 we use a different way to configure the amount of resulting data generated. Unlike v1, where we specify the number of documents we wish to generate, in v2 we set the number of organizations, which ultimately 22

23 determines the number of generated documents, hence the database size. Figure 5.17 shows the performance of the Portavita Benchmark v2 in terms of time it takes to generate 1 GB of data for databases of various sizes. The size of the databases is shown by both, the number of organizations and the number of documents generated. The graph shows that the generation rate is about 11 hours/tb of data. 45 Total number of documents 1,248,261 2,470,915 4,991, Time(seconds) / 1GB Number of organizations Figure 5.17.: Performance of Portavita Benchmark v2. Time required to generate 1GB of data. Since Portavita Benchmark v2 is a streaming architecture based on micro-batches, it is not possible to measure the processing time of each step, like was done for v1. Moreover, as the database format was changed in v2, we can only compare the database generation rate based on the number of documents loaded into the final database per second. Figure 5.18 shows the performance of both v1 and v2 on the same single-node server. The data generation rate of v2 is more than twice higher than that of v v1 v2 250 Documents/second ,000,000 4,000,000 6,000,000 8,000,000 10,000,000 Total number of documents Figure 5.18.: Comparative performance evaluation. Amount of documents loaded per second for v1 and v2. 23

24 Conclusions on performance The core of the generator of the AXLE synthetic dataset is the XML generator. It is this generator that is most useful as benchmarking tool for software related to the exchange of healthcare data, since it emits HL7v3 Clinical Document Architecture (CDA) XML and FHIR messages. The XML generator speed is over 1TB/hour. With additional transformation to database format and additional processing, generation speed is 1TB/11 hours on a single-node server. 24

25 6. Benchmark queries We provide a number of queries for secondary use, i.e., for reporting and analytics purposes. The full document describing the queries has been delivered in June 2014, whereas the queries sources are included in the github repository of AXLE Healthcare Benchmark [1] Queries overview Cross-Organization comparative analysis This query compares all the organizations based on the percentage of the patients who have had one of the following examinations in the last year: Fundus checkup, Foot checkup, Intermediary checkup, Risk inventory, Diabetes medication, Dietary advice. Cross-practitioner comparative analysis This query is quite similar to the previous query in that compares on the same indicators, with the only difference that this query focuses on the performance of each practitioner within an organization. Deviation of organization performance This query reports on organization average values with respect to a number of important patient measurements, such as glucose level, blood pressure and so on. These measurements give a highlevel overview of how well the patients within an organization are doing. Extreme values This query shows all patients with observation values that have been classified as extreme. Such patients are shown with some additional data such as gender, age, most recent HbA1c, triglyceride and blood pressure. Relative extreme values The definition of an extreme value is often determined by national benchmarks. But there are local patient populations with local averages that deviate considerably from the national average. Considering that the organizations are geographically distributed across The Netherlands, this query reports on the patients that have extreme values compared to the average observation values within the organization where these patients are treated. Abnormal blood pressure or macroangiopathy This query retrieves all patients that currently either have an abnormal blood pressure (highly related to the age of the patient) or macroangiopathy. Typically, these patients have an increased risk of developing new complications and are monitored closely. Data Analysis Influence of medication on HbA1c HbA1c is a type of blood value that shows glucose levels over longer periods of time. This query shows 25

26 for all patients and for all of their medications the average HbA1c one year before the patient started the initial medication, the average HbA1c in between the initial medication and the new medication, and the average HbA1c one year after the new medication. Trend Analysis Trends in the process This query aims to gain insight into how often examinations are performed across various organizations and the trend. So, for every organization, type of examination, and time period, this query shows the average number of times that the examination was performed per patient in that period in that organization. Six periods of three months each are defined starting from the current date. Trend Analysis Trends in smoking This query shows the trends in smoking per organization. Namely, in the last 6 periods of half a year, this query reports on the number of active patients in the organization, how many of them smoked in that period, and how many ceased smoking in that period. 26

27 7. Conclusions In this document we presented Portavita Benchmark, a dataset generator specific for healthcare. The generated data are based on models built upon real health records, and comply with the exchangeable HL7 formats, namely CDA and FHIR (XMLbased). Portavita Benchmark borrows libraries from MGRID in order to efficiently transform and store the generated data in a PostgreSQL DBMS. The synthetic clinical data includes examinations and observations which are assigned to synthetic patients in the context of a diabetes, COPD or CVRM treatment. The validation of Portavita Benchmark showed that observation values, correlations among various observations, and occurence of missing values in the synthetic dataset resemble the original data. We showed that the process of XML data generation is relatively fast, namely 1TB/hour, compared to the subsequent transformation and storage processes, that is, 1TB in about 11 hours on a single-node server. 27

28 Appendix 28

29 Appendix A. Clinical Document Architecture The following xml snippet depicts an example of a CDA document. The first part of the CDA consists of contextual information, such as the code to identify the type of document (code), the time it was issued (effectivetime), the classification level of the document (confidentialitycode), the patient reference (recordtarget) and information about the person who authored, entered and/or authenticated the document. In this case all three roles are covered by the same person. The second part of the CDA starts with the component structuredbody and has information about the observations performed. In this case, the element organizer contains more contextual information about the observations, such as time and performer, and an additional nested organizer with display name Blood pressure. This organizer contains two elements Observation that have values for the systolic and diastolic measurements. <?xml version="1.0" encoding="utf-8"?> <ClinicalDocument xmlns="urn:hl7-org:v3" xmlns:xsi=" <typeid root=" " extension="pocd_hd000040"/> <id root=" " extension="1aa415ca-61ed d1523a7477d3"/> <code code=" " codesystem=" " codesystemname="loinc" displayname="summarization note"/> <title>phr Update</title> <effectivetime value=" "/> <confidentialitycode code="n" codesystem=" " codesystemname="confidentiality" displayname="normal"/> <recordtarget> <patientrole> <id nullflavor="unk"/> <patient> <id root=" " extension=" "/> </patient> <providerorganization> <id root=" " extension=" "/> </providerorganization> </patientrole> </recordtarget> <author> <time value=" "/> <assignedauthor> <id root=" " extension=" "/> <representedorganization> <id root=" " extension=" "/> </representedorganization> </assignedauthor> </author> <dataenterer> <time value=" "/> <assignedentity> <id root=" " extension=" "/> <representedorganization> <id root=" " extension=" "/> 29

30 </representedorganization> </assignedentity> </dataenterer> <legalauthenticator> <time value=" "/> <signaturecode code="s"/> <assignedentity> <id root=" " extension=" "/> <representedorganization> <id root=" " extension=" "/> </representedorganization> </assignedentity> </legalauthenticator> <documentationof> <serviceevent classcode="pcpr"> <id root=" " extension=" "/> <code code=" " codesystem=" " codesystemname="snomed-ct" displayname="diabetes treatment"/> </serviceevent> </documentationof> <component> <structuredbody> <component> <section> <entry> <organizer classcode="battery" moodcode="evn"> <id root=" " extension=" "/> <code code="portavita1234" codesystem=" " codesystemname="portavita" displayname="self-check CVRM"/> <statuscode code="completed"/> <effectivetime> <low value=" "/> </effectivetime> <performer typecode="prf"> <assignedentity> <id root=" " extension=" "/> <representedorganization> <id root=" " extension=" "/> </representedorganization> </assignedentity> </performer> <component> <organizer classcode="battery" moodcode="evn"> <id root=" " extension=" "/> <code code="portavita1235" codesystem=" " codesystemname="portavita" displayname="blood pressure Self measurement"/> <statuscode code="completed"/> <effectivetime> <low value=" "/> </effectivetime> <component> <observation classcode="obs" moodcode="evn"> <id root=" " extension=" "/> <code code="portavita1236" codesystem=" " codesystemname="portavita" displayname="systolic BP, self measurement"/> <statuscode code="completed"/> <effectivetime> <low value=" "/> </effectivetime> <value value=" " unit="mm Hg" xsi:type="pq"/> </observation> </component> <component> <observation classcode="obs" moodcode="evn"> <id root=" " extension=" "/> <code code="portavita1237" codesystem=" " codesystemname="portavita" displayname="diastolic BP, self measurement"/> <statuscode code="completed"/> <effectivetime> <low value=" "/> </effectivetime> <value value=" " unit="mm Hg" xsi:type="pq"/> </observation> 30

31 </component> </organizer> </component> </organizer> </entry> </section> </component> </structuredbody> </component> </ClinicalDocument> 31

32 Appendix B. List of Observations per Examination Below follows the list of examinations and their observations generated by Portavita Benchmark. The examinations are denoted with ===== and a.json suffix and have a corresponding human-readable display name. Examinations are composed by a number of observations which are listed after the examination name. The observations have a code which is preceded by the corresponding data type, which is either PQ (physical quantity continous) or CD (coded description discrete). Examination/Observation code displayname ===== json ===== Systolic flow/diastolic flow:velrto:pt:cerebral artery anterior^fetus:qn:us.doppler pq_ Intravascular diastolic:pres:pt:arterial system:qn: pq_ Intravascular systolic:pres:pt:arterial system:qn: ===== json ===== Spirometry cd_portavita1327 Interpretation pq_ Peak expiratory flow rate after bronchodilation pq_ Peak expiratory flow rate before bronchodilation pq_ FEV1 before bronchodilation pq_ FEV1 after bronchodilation pq_ Forced vital capacity (FVC) after bronchodilation pq_ Forced expiratory volume 1 (FEV1)/ forced vital capacity (FVC) ratio before bronchodilator pq_ Forced expiratory volume 1 (FEV1)/ forced vital capacity (FVC) ratio after bronchodilator pq_portavita534 FEV1 Pre (% of predicted) pq_portavita535 FEV1 Post (% of predicted) pq_portavita536 Reversibility FEV1 (%) pq_portavita537 FVC Pre (Absolute) pq_portavita538 FVC Pre (% of predicted) pq_portavita539 FVC Post (% of predicted) pq_portavita540 PEF Pre (% of predicted) pq_portavita541 PEF Post (% of predicted) pq_portavita542 TLC Pre (Absolute) pq_portavita543 TLC Pre (% of predicted) ===== json ===== Standard ECG cd_ ECG finding ===== json ===== Follow-up diabetic assessment cd_ Deficient knowledge of dietary regimen cd_ Nocturnal hypoglycaemia cd_ Hypoglycaemia cd_ Irregular heart beat cd_ General well-being finding cd_portavita1342 Antihypertensives cd_portavita1343 Diuretics cd_portavita1344 Beta-blockers cd_portavita1345 Calcium antagonists cd_portavita1346 Drugs affecting the renin-angiotensin system cd_portavita1347 Alpha-blockers cd_portavita1348 Other antihypertensives cd_portavita1349 Blood-thinning drugs cd_portavita1350 Platelet aggregation inhibitors cd_portavita1351 Anticoagulants cd_portavita1352 Other blood-thinning drugs cd_portavita1353 Lipid-lowering drugs cd_portavita1354 Statins cd_portavita1355 Other lipid-lowering drugs cd_portavita1428 Extra attention for Individual care plan cd_portavita24 Fasting hypos cd_portavita25 Hypos after breakfast cd_portavita26 Hypos before lunch cd_portavita27 Hypos after lunch cd_portavita28 Hypos before dinner cd_portavita29 Hypos after dinner cd_portavita30 Hypos before bedtime cd_portavita34 Therapy compliance cd_portavita38 Dietary advice problems cd_portavita39 Insufficient application of guidelines cd_portavita648 Diabetes medication pq_ Frequency of hypoglycaemia attacks 32

33 pq_ Body weight pq_ Heart rate pq_ Glucose level - finding pq_ Body height measure pq_ Body mass index pq_ Intravascular diastolic:pres:pt:arterial system:qn: pq_ Intravascular systolic:pres:pt:arterial system:qn: pq_portavita175 Fasting blood glucose pq_portavita176 Blood glucose after breakfast pq_portavita177 Blood glucose before lunch pq_portavita178 Blood glucose after lunch pq_portavita179 Blood glucose before dinner pq_portavita180 Blood glucose after dinner pq_portavita181 Blood glucose before bedtime pq_portavita182 Nighttime blood glucose ===== json ===== Fundoscopy - diabetic check cd_portavita220 Assessment of fundus image ===== json ===== Diabetic annual review cd_ Cardiac auscultation finding cd_ Deficient knowledge of dietary regimen cd_ Visual symptoms cd_ [D]Shortness of breath cd_ [D]Other specified symptoms cd_ Current drinker cd_ Myocardial infarction cd_ Time spent exercising cd_ Cerebrovascular accident cd_ Nocturnal hypoglycaemia cd_ Thirst symptom cd_ Transient cerebral ischaemia cd_ Polyuria cd_ Chest pain cd_ Hypoglycaemia cd_ Diastolic murmur cd_ Foot abnormality - diabetes-related cd_ Microalbuminuria cd_ Systolic murmur cd_ Pruritus cd_ Irregular heart beat cd_ General well-being finding cd_ Tobacco use and exposure - finding cd_ Angina pectoris cd_ Dyslipidaemia cd_ Hypertensive disorder cd_ Tortuous coronary artery cd_ Peripheral vascular disease cd_ Medication side effects present cd_ Infectious disease cd_ Homocystinaemia cd_ Aortic bruit cd_ Heart failure cd_portavita10 Carotid aorta right cd_portavita11 Renal aorta left cd_portavita12 Renal aorta right cd_portavita13 Femoral aorta left cd_portavita1342 Antihypertensives cd_portavita1343 Diuretics cd_portavita1344 Beta-blockers cd_portavita1345 Calcium antagonists cd_portavita1346 Drugs affecting the renin-angiotensin system cd_portavita1347 Alpha-blockers cd_portavita1348 Other antihypertensives cd_portavita1349 Blood-thinning drugs cd_portavita1350 Platelet aggregation inhibitors cd_portavita1351 Anticoagulants cd_portavita1352 Other blood-thinning drugs cd_portavita1353 Lipid-lowering drugs cd_portavita1354 Statins cd_portavita1355 Other lipid-lowering drugs cd_portavita14 Femoral aorta right cd_portavita1428 Extra attention for Individual care plan cd_portavita24 Fasting hypos cd_portavita25 Hypos after breakfast cd_portavita26 Hypos before lunch cd_portavita27 Hypos after lunch cd_portavita28 Hypos before dinner cd_portavita29 Hypos after dinner cd_portavita30 Hypos before bedtime cd_portavita308 Assessment of ophthalmic examination cd_portavita34 Therapy compliance cd_portavita38 Dietary advice problems cd_portavita39 Insufficient application of guidelines cd_portavita438 Motivation to stop smoking cd_portavita439 Stop smoking advice given cd_portavita44 Symptoms indicating hypoglycemia cd_portavita440 Follow-up appointment made cd_portavita48 Decrease of physical capacity cd_portavita5 Auscultation indicated cd_portavita50 Pain in calves when walking cd_portavita52 Pain or tingling in legs cd_portavita54 Sexual dysfunction disorders cd_portavita6 First sound cd_portavita61 Hypoglycemia recognition cd_portavita63 Patient uses caffeine cd_portavita64 Products with glycyrrhizic acid cd_portavita648 Diabetes medication cd_portavita68 Diabetes in first-degree or second-degree relatives 33