Guidance on Data Exchange 1

Transcription

1 GUIDANCE OF EFSA Guidance on Data Exchange 1 European Food Safety Authority 2, 3 European Food Safety Authority (EFSA), Parma, Italy This guidance, published on 11 November 2010, replaces the earlier version published on 5 November ABSTRACT Data collection is an important task of EFSA and a fundamental component of many of its risk assessment activities. With the assistance of the Technical Working Group on Data Collection, EFSA has developed two guidance documents to facilitate the exchange of data between Member States and EFSA. The Standard Sample Description for Food and Feed, published in January 2010, aimed at harmonising the data content of submissions. The current document is complementary to the first guideline in that it addresses the transmission mechanisms, file formats, security requirements and message exchange protocols (including validation messages) to be used. These two guidance documents are intended to provide the basis for a general model to harmonise the collection and transmission of a wide range of measurements in the area of food and feed safety assessment. KEY WORDS Standard, Format, Chemical, Occurrence, Transmission, XML, Data, Security, Protocol. 1 On request of EFSA, Question No EFSA-Q , issued on 28 October Correspondence: datex@efsa.europa.eu 3 Acknowledgement: EFSA wishes to thank the members of the Technical Working Group on Data Collection for the preparation of this EFSA scientific output: Jens Hinge Andersen, Eileen O'Dea, Luisa Oliviera, Jean Cédric Reninger, Renata Del Rosario, Stijn Saevels, Lars Wiehle, Josef Wolf and Roger Wood and EFSA s staff members Fabrizio Abbinante, Stefano Cappè, Marco Leoni, and Jane Richardson for the support provided to this EFSA scientific output. For citation purposes: European Food Safety Authority; Guidance on Data Exchange [50 pp.]. doi: /j.efsa Available online: European Food Safety Authority,

2 SUMMARY Data collection is an important task of EFSA and a fundamental component of many of its risk assessment activities. EFSA receives an increasing volume of data from different providers (e.g. Member States, the European Commission, industry etc.) in support of its scientific activities. The Technical Working Group on Data Collection (TWG-DC) was mandated by EFSA to develop a proposal to harmonise the collection of analytical measurement data for the presence of harmful or beneficial chemical substances in food and feed. The TWG-DC was requested to produce two guidance documents: A Guidance on Standard Sample Description for Food and Feed published on 25 January 2010 covering the harmonised description of analytical measurement data for food and feed samples including a list of standardised data elements (items describing characteristics of samples or analytical results such as country of origin, product, analytical method, limit of detection, result, etc.), controlled terminologies and validation rules to enhance data quality. This Guidance on Data Exchange prescribing procedures to efficiently transmit and exchange data between Member States and EFSA including specific file formats for data transmission (e.g. XML, Microsoft Excel etc.) and specific data transmission protocols to support electronic data exchange. The TWG-DC aimed to develop the Guidance document on Data Exchange to be sufficiently generalised to be applicable to a wide range of data collection processes undertaken by EFSA, even if these data collections require a different data structure from that described in the Standard Sample Description for Food and Feed covering chemical and pesticide occurrence data. The TWG-DC agreed that the establishment of these guidance documents is only the first step in supporting the harmonisation of data transmissions between Member States and EFSA. Another necessary element for a successful implementation of an efficient European data collection system is the construction of a maintenance and development program to ensure that the guidance documents continue to evolve in line with the changing needs of the data analysis and risk assessment stakeholders. Additionally the standardisation effort should be extended to areas not currently in the scope of the guidance on Standard Sample Description to simplify the reporting requirements from the Member States perspective. The group finally recognises that, at the present time, the ability of each Member State to transmit data to EFSA according to the guidance will vary. Therefore this guidance should also be taken into account by member states when planning future developments and evolution of local, regional and national systems with the objective of harmonising data transmissions. 2

3 TABLE OF CONTENTS European Food Safety Authority (EFSA), Parma, Italy... 1 This guidance, published on 11 November 2010, replaces the earlier version published on 5 November Abstract... 1 Summary... 2 Table of contents... 3 Background as provided by EFSA... 4 Terms of reference as provided by EFSA... 4 Consideration Introduction Transmission Methods File Formats Security Requirements Automated Messages Exchange Protocol Type of operations supported Protocol for message exchange FTP implementation Web services implementation XML messaging Data message MRN message Acknowledgment Message Error Codes and Messages General business rules Specific business rules Conclusions and Recommendations Conclusions Recommendations References Appendices Appendix 1 Description of the MRN and ACK as currently implemented in EFSA A1.1 MRN message A1.2 Acknowledgement message A1.3 Transaction Identifier Synchronisation Appendix 2 Description of the business rules applied to the elements of the standard sample description

4 BACKGROUND AS PROVIDED BY EFSA Guidance on Data Exchange Data collection is an important task of EFSA and a fundamental component of risk assessment (Articles 22 and 23 of Regulation EC No 178/ ). EFSA receives from different providers an increasing volume of data in support of its scientific activities. Data may be submitted to EFSA by a variety of providers: national competent authorities, local authorities, laboratories, universities and others. The increasing volume of data transmitted to EFSA involves a number of challenges: Efficient use of human resources in data collection processes, both on the side of the data senders and on the EFSA side where data have to be collected, collated and analysed. Quality of transmitted data, which may differ in origin, language, description and codification. A standard data quality facilitates the task of collating and preparing data for analysis. Capacity of managing high volumes of data; many data collections contain a huge quantity of analytical results, such as data from control and monitoring programs and targeted surveys. The quantity of these data makes their manual processing unfeasible and introduces the need for automated methods of formatting, transmitting, processing and analysing the data. Capacity to analyse the data and to produce valuable reports for the different stakeholders; the increased volume of data requires more advanced techniques for storing and analysing the data. Data should be, as far as possible, readily available for standardised analysis functions to harmonise processing among the different food data stakeholders. EFSA requests that the Data Collection and Exposure Unit, through a self-tasking mandate, harmonise the collection of analytical measurement data for the presence of harmful or beneficial chemical substances in food and feed. The Data Collection and Exposure Unit (DATEX) in coordination and cooperation with the Pesticide Risk Assessment Peer Review Unit (PRaPeR) will establish, for this purpose, a working group of technical experts (Technical Working Group on Data Collection). The working group should report to the Chemical Occurrence Expert Group and to the Networking Group on Pesticide Residues. These networking groups will review and approve all deliverables of the Working Group. TERMS OF REFERENCE AS PROVIDED BY EFSA The working group should focus on both chemical occurrence and pesticide residue data, due to the close similarities between these areas. The working group should build on, expand further and finalise the effort of harmonising the analytical measurement data descriptions already started by the EFSA internal working group on controlled terminology and define the data exchange model format to be used. The working group is requested to produce guidance documents on: 4 Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety. OJ L 31, , p

5 the harmonised description of data on analytical measurements in food and feed samples (Guidance on Standard Sample Description for Food and Feed); The Standard Sample Description contains a list of data elements that are standardised and can be conveniently used by both data senders and data recipients to fully describe samples and analytical parameters for evaluation purposes. This standard provides: name and structure of the defined data elements to be used; controlled terminologies, where needed, with exclusion of the food dictionary which will be addressed by a specific working group; validation rules to assess the validity of the information supplied, to ensure an adequate level of data quality in data export, transmission and storage. The proposal for standardised sample description is based on the initiative developed by the EFSA internal Working Group on Controlled Terminology. The group elaborated a draft standard for the description of chemical contaminants and pesticide residues data. This document was presented at the IT Expert Meeting on February 2009 and was also the basis for a pilot project on pesticide residues data transmission and for an EFSA grant on chemical contaminant data. the methods to efficiently transmit and exchange data between Member States and EFSA (Guidance on Data Exchange); The working group should propose a Guidance on Data Exchange to define methods for the transmission and the exchange of data which would be suitable for the data collected with the standard sample description for food and feed. The guidance should take into account the software tools already set up by the EFSA IT Unit and provide inputs for their enhancement. The guidance should promote the use of semi-automated or automated methods for the data transmission to lower the costs of manual human intervention. The guidance documents should be presented for approval to the EFSA Chemical Occurrence Expert Group and to the EFSA Networking Group on Pesticide Residues. 5

6 CONSIDERATION 1. Introduction The Guidance on Data Exchange takes the logical structure defined in the Guidance on Standard Sample Description for Food and Feed (hereinafter referred to as SSD) and specifies: the file formats for the transmission of data (e.g. Microsoft Excel, Comma Separated Values - CSV, Extensible Markup Language - XML 5, etc.); the protocol for exchanging data between the sender and the receiver; the internet protocol supported to transmit the data electronically; the file formats for the validation rules; additional technical information and specifications to make the electronic transmission possible (error codes, security requirements, etc.) Specifically, the Guidance on Data Exchange will describe the transmission of data between Member States and EFSA, but the methods described can be applied for transmissions between any data senders and any data receivers. The data transmission system needs to meet the following requirements: 1) Simplicity: the system for both the sender and the receiver should be simple to ensure easy implementation. Simplicity should prevail, where possible, over other requirements such as memory occupation, disk occupation, bandwidth occupation; 2) Ensure secure transmissions; 3) Require minimum maintenance; 4) Flexibility: the system needs to be sufficiently flexible to ensure it can keep pace with scientific developments in the field of food and feed risk assessment. Changes should be predominantly managed through routine amendments to the controlled terminology dictionaries in consultation with relevant stakeholders. For more significant changes to the SSD elements, consultation with stakeholders and impact analysis would need to be made prior to implementation; 5) Involve minimal human resources: as far a possible the data collection process should be fully automated and ensure the most efficient use of human resources; 6) Cost effective: the implementation and maintenance costs for the sender organisation should be kept to the absolute minimum. 5 W3C Consortium - Extensible Markup Language (XML) 1.0 (fifth edition) available on line at last accessed on 14/07/2010 6

7 2. Transmission Methods Guidance on Data Exchange The SSD defines the content for the transmission of data between data senders and data receivers. This can be a manual process requiring user interactions or an automated process using procedures developed within information systems. In summary: Manual transmission methods: o Manual posting of files on the EFSA Data Collection Framework (DCF) Web Application; Automatic transmission methods: o Automatic transmission via Secure File Transfer Protocol (FTPS 6 ); o Automatic transmission via SOAP 7 (originally Simple Object Access Protocol). This protocol relies on the extensible Markup Language (XML) as its message format and other Application Layer protocols (mainly Remote Procedure Call (RPC 8 ) and Hypertext Transfer Protocol (HTTPS 9,10 ) for message negotiation and transmission). Automation of transmissions is considered to be a practical way in which large numbers of files and volumes of data can effectively be transmitted without incurring significant human resource demands both for the data sender and data receiver. The Working Group recognises that full automation of the data transmission may not be feasible for all Member States in the short term; however, the publication of this guidance is intended to assist Member States in developing their systems to allow automation in the future. The Working Group acknowledges in addition that for annual monitoring programs, manual transmission is often sufficient. It is anticipated that, for large data collections, batching the data into multiple transmissions (e.g. maximum of 300MB each) may be a pragmatic way to minimise any difficulties in transmitting very large files. 3. File Formats The WG recognises that the transmission format for the Standard Sample Description is XML. The automatic transmission with Web Services and FTPS will be supported in XML only. Tools such as XSLT to format the XML files (data and error messages) into human readable format such as HTML will be provided by EFSA. The main benefits of the XML file structure are: 6 IETF RFC Securing FTP with TLS available on line at based on FTP (RFC 959) and TSL (RFC 2246) last accessed on 17/03/ World Wide Web Consortium - SOAP Version 1.2 Part 1: messaging Framework (Second edition) available on line at last accessed on 17/03/ IETF RFC 707 A High-Level Framework for Network-Based Resource Sharing available on line at last accessed on 17/03/ IETF RCF Hypertext Transfer Protocol -- HTTP/1.1 available on line last accessed on 17/03/ IETF RFC 2818 HTTP over TSL available on line last accessed on 14/07/

8 it is an open standard and based on text files, therefore files can be opened without the need of specific tools; it is easy for machine interaction: information is structured, therefore it is easy for a computer program to read; it is currently broadly supported: there are parsers and tools in several programming languages and technologies; it supports UNICODE encoding therefore is natively appropriate to support a multilanguage user community such as the EU; it is easy to implement compared to other transmission standards and is platform independent; each data element is assigned a specific data type through the XML schema, resulting in improved data quality; basic file validation, which can be performed automatically on the sender side using an XML schema (XSD), prior to transmission to the receiver. This can reduce the number of errors and re-transmissions. Since some senders may not be ready to implement XML transmissions, the WG recommends that EFSA supports manual transmission of other formats (e.g. CSV and Microsoft Excel) using the Data Collection Framework for a limited, defined period. However the WG recognises the following limitations of these transmission formats: CSV: the presence of special characters in the data values (e.g. double quote character ) can result in uploading failures. In addition, the inclusion of delimiter characters within the data values can result in import errors which can seriously compromise the quality of the submitted results; Excel file: the wrong data type assignment (e.g. number formatted as text) can result in data values being converted so missing values during the upload process. Frequently the import tool provides no warning message and the missing values are only detected later in the analysis process. In contrast with XML, for CSV and Excel formats, no automatic validation can be performed by the sender prior to transmission resulting in either an increase in the number of cycles of resubmission of data or potentially onerous manual validation by the sender or receiver or both. The Working Group recommends also batching large transmission into files of approximately 300Mbyte or less (uncompressed) for efficient transmission and data management. In addition, it is recommended that the files are transmitted compressed using zip compression (with programs compliant with MIME type: application/zip or application/x-zip-compressed) in order to reduce the transmission time. 4. Security Requirements The main security requirements connected to data transmission are identified below. It should be noted that whilst the first two requirements are critical, the third is currently seen by the working group as desirable. 8

9 Identification of the sender: the receiver can identify the sender performing the data transmissions. Confidentiality of the transmission: the information involved in the data transmission is accessible only to the sender and to the receiver, even though a public network might have been used to deliver the message. Non-repudiation: the receiver has proof of participation by the sender in the data transmission that cannot be denied. There are Internet standards such as the AS1 and AS2 originally defined for Electronic Data Interchange (EDI), that can equally be used to transmit XML files and that address the three requirements. However the possible use of these standards would require difficult in-house development or the purchase of gateway software. It would also involve the implementation and the maintenance of client and server certificates to be used to digitally sign and encrypt the data transmissions. In order to reduce the complexity of the implementation and the financial impact of specific software or services for the data senders, a simplification of the security requirements is suggested by the Working Group. For the automatic transmission methods, the identification of the sender organisation is performed assigning a user identification and password to be used by the software performing the operation. The confidentiality is implemented using a standard Internet application protocol (FTP, HTTP, etc.) through SSL or TSL. The usage of these protocols implies the installation and the maintenance of server certificates on data receiver side. For the manual transmission methods, the identification of the sender is performed with user identification (USERID) and password. Specific requirements for the length, format and duration of the password are defined. The confidentiality is implemented using the standard HTTPS internet protocol, which requires the installation and the maintenance of a server certificate on data receiver side (e.g. EFSA), but it does not require client certificates for the users. The implementation of the non-repudiation requirement for both automatic and manual transmission methods is not seen as critical at this point, since the user community consists of competent authorities of Member States and EFSA. The implementation of this requirement would imply the usage and maintenance of digital certificates and the implementation of digital signatures by data senders which is currently seen as a significant work burden at the current stage of development of the data exchange through the Standard Sample Description. The working group recommends that the security requirements are re-assessed periodically with the growth of the user community. In case the Member States and EFSA would envisage the need to fully support all the security requirements for data transmission, the technical working group recommends the usage of some gateway software and the implementation of existing Internet standards, which involve the usage of certificates for electronic signature and encryption of the electronic transmissions. 5. Automated Messages Exchange Protocol 5.1. Type of operations supported The Message Exchange Protocol defines a file transmission as the unit of data to exchange between the sender and receiver. The file transmission is identified by the transmissionid (datatrx:receivertrxcode). This identifier allows the file transmission to be retrieved in the DCF and 9

10 should be included in all messages. Since the unit of data exchange is the file transmission, when it is necessary to change a single sample or result, all the samples and results belonging to the same file transmission should be retransmitted. This process, which at first implies bigger file sizes, allows much easier database management processes both at data sender and receiver side. The protocol defines a message framework to perform the following operations: Insert: A new data transmission file is inserted in the data receiver s database. Delete: A data transmission file included in the receiver s database is nullified. The data transmission file should never be deleted by the receiver s database, only flagged as noncurrent. Archiving procedures of nullified files should be set up on receiver s side. Update ( Replace ): A new data transmission file, provided by the sender, replaces entirely a transmission file previously loaded in the receiver s database. The update operation is a more complex combination of the delete followed by the insert procedure 11. The update operation uses a mechanism for tracking the IDs of updated transmissions, so that a link is kept with the old transmission. In addition the update operation should be performed in one transaction, so that in case of system errors at the end of transaction either the new transmission is loaded and the old is deleted or the old is active and the new transmission returned an error. During an update operation there should never be a case where two or more linked transmissions are current or flagged as deleted. The message initiating the data transmission will trigger the different operations using the datatrx:optype attribute. The datatrx:receivertrxcode must also be present for delete and update procedures Protocol for message exchange The Message Exchange Protocol describes the exchange of messages between Member States and EFSA: the data message, the MRN message and the acknowledgment message. Since the term message can be used in different contexts in information systems, it is fundamental to recognise that the term message refers to the EFSA XML message layer (i.e. the data message, the MRN message and the acknowledgment message) which is shown in Figure 1. This layer interfaces with EFSA s and Member States applications for sending and receiving data. This layer is independent and transparent from the specific transport layer protocol defined for this protocol (e.g. SOAP and FTP), therefore the term message should not be confused with FTP messages or SOAP messages. Transport Layer: Transport protocol (SOAP, FTP) EFSA XML message layer: Data message, MRN message, ACK message Application layer: Chemical Occurrence Data/ Pesticide residues data Figure 1: Protocol layers. 11 Currently EFSA Data Collection Framework does not implement an update procedure, therefore the tracking requirement between the different transmissions is not supported. Updates should therefore be downgraded to a delete plus an insert procedure. 10

11 The main participants in the electronic transmission protocol are represented in Figure 2. The Sender represents the reporting organisation (e.g. a competent authority of a Member State). The Receiver represents the organisation receiving the data (e.g. EFSA). Protocol sendresumedata * «extends» «extends» «extends» * * * Send Data Message Send MRN send Acknowledgment Sender (Member State) * * * * Validate XML «extends» «extends» * «extends» Receiver (EFSA) Validate data message Validate MRN Validate ACK Business rules validation * Figure 2: Participants in the protocol The basic steps of the protocol are the following: 1. The Data Sender creates, validates against the agreed schema and transmits an XML file with data to the Data Receiver. Data will be wrapped in an XML message envelope designed to support the protocol which will be defined in the paragraph The Data Receiver will send back a Message Receipt Notification (MRN) to confirm the transmission. At this stage the Data Receiver verifies whether the XML message is a wellformed XML file and also whether it is valid against the provided standard XML schema. 3. The Data Receiver will validate the XML message against a standard set of validation rules (e.g. validation rules published in the guidance on Standard Sample Description p. 39). On the basis of the result of the validation, an acknowledgment message is prepared The Data Receiver will then transmit the acknowledgment message back to the Data Sender, reporting on the positive/negative result of the data transmission. 5. The Data Sender will confirm the receipt of the acknowledgment message through a message receipt notification MRN. The sender should validate the acknowledgment before transmitting back the MRN since a positive MRN confirms also to the receiver that the acknowledgment was well-formed and valid XML. No business rules are applied prior to the 12 Currently the DCF only validate the XML messages against the defined controlled terminology, all the other validations specified in the standard sample description are implemented manually by the data receiver after the receipt of the message. 11

12 acknowledgment message. The MRN is sent back asynchronously calling the Data Sender Web Service. The sender will have to check for a well-formed XML. In Figure 3 the term XML validation refers to both checking for a well-formed XML file and a valid XML file against the specified schema since the XML file cannot be valid without being well-formed first. Sender (Member State) Receiver (EFSA) Data message XML validation MRN Business rules validation Acknowledgment XML validation MRN Figure 3: Message exchange protocol. This definition of the message exchange protocol is only logical and independent from the specific application protocol used for the physical delivery of the message itself. These can be: the File Transfer protocol (FTP); Web Services (WS). There are cases where the MRN may be not generated by the system: Network issues: the data transmission could be lost and the error will be indicated with a proper exception (it could be an HTTP error code for WS or a proper FTP error). 12

13 Security issues: In case of authentication failure the error will be indicated by the transmission protocol with the proper exception (an HTTP error code for WS or a proper FTP error in case of FTP usage). XML validation: In case of successful transmission of an invalid XML document within the transmission file there is no guarantee about the content and delivery of the MRN. Therefore it is always recommended for the Data Sender to validate the XML files against the provided schemas prior to transmission to avoid this problem. In these cases the sender should contact the receiver to find out the result of the transmission. In transmissions performed using mechanisms which allow for some synchronous transmission (e.g. in case of Web Services), a synchronous transmission confirmation will be implemented in order to reduce the possibility of MRNs not being generated by the receiver system. This case will be further explained in the web service section FTP implementation The sender opens an FTP connection on the receiver s FTP server. The Sender authenticates itself using the credentials provided and sends the file. Each Sender organisation will have its own FTP folder on the EFSA system where their files will be saved. Accounts will be created and distributed to each Data Sender by the EFSA Data Manager. In order to signal the completion of FTP file sending, the sender shall put a file with the same name of the transmitted data file, plus the suffix.go. For example: data file = dataprovider1.xml trigger file = dataprovider1.xml.go The content of the trigger file is irrelevant therefore does not need to contain data. It is necessary to put a go file for each transmitted XML file. No further processing will start until the.go has been placed in the target directory of the FTP site. The DCF checks the input file to verify that it is valid according to the defined XML schema. The system will send back asynchronously an MRN via using the information defined for the logged-in organisation. The MRN contains detailed information about the uploaded data file and the result of the XML validation. MRN also contains the transaction identifier (contained in the receivertrxcode of the MRN and of the acknowledgment) that can be used on the DCF web interface to check the complete validation of the message. The Sender will be able to find the file under the Data transmissions table, accessible through the DCF web interface. The file will be removed from the FTP folder once uploaded by the DCF. 13

14 The DCF will execute further validations applying business rules on the original file asynchronously 13. At the end of the process an acknowledgment will be sent to the Sender containing the validation result via . This FTP implementation contains some differences in respect to the Web service implementation proposed in the paragraph below. These differences are due to the fact that the FTP protocol is totally asynchronous while the WS protocol allows synchronous communication Web services implementation Both the data senders and the data receivers will use WS technology to allow back and forward 14 interaction (for MRN and acknowledgment exchange) during each transmission. The interface (WSDL) of the WS installed on the data sender s side is defined by EFSA and this interface is the same for all organisations involved in the electronic transmission. For simplicity also the interface installed on EFSA s side follows the same WSDL. The WSDL are therefore all the same with the exclusion of the URL address of the web service located in the element soap:address attribute location. This element must be modified manually by the sender in the WSDL obtained by the EFSA system. A Web Service interface contract (WSDL) is available and can be downloaded from Each organisation has the responsibility to implement and maintain the WS of their organisation as reported in Figure 4. The WSDL file includes three methods: sendfilename: This method is for internal EFSA use. Senders are not requested to implement this method ping: This method is for checking if the web service is up and running without the need of performing an actual data transmission. The synchronous response of this method is a text string reporting TRXOK. sendresumedata: This method is the only method available for transmitting all the XML messages (data messages, MRN messages and acknowledgment messages). The synchronous response of this method is a text string reporting TRXOK if the receipt of the XML file was correct otherwise TRXKO if the transmission of the XML was not correct. Therefore the Web Service transmission protocol proposed in 4 should be interpreted as described below. 13 Currently the DCF only validate the XML messages against the defined controlled terminology, all the other validations specified in the standard sample description are implemented manually by the EFSA data managers after the receipt of the message. 14 The transmission of each data message, MRN and ACK is a actually a one-way communication although in the WSDL the methods have been defined as two-ways communications: (this to overcome some library limitations) 14

15 Sender (Member State) Receiver (EFSA) 1.sendResumeData(Data message) TRXOK Minimum info in the header XML validation 2.sendResumeData(MRN) TRXOK 3.sendResumeData(ACK) Business rules validation XML validation TRXOK 4.sendResumeData(MRN) TRXOK Figure 4: Web service implementation. 1. The Sender (e.g. Member State) sends the data message to the Receiver side (e.g. EFSA) using a WS installed on the Receiver side, calling the method sendresumedata. The Data Receiver WS is protected and requires authentication: each Sender will be provided with its own credentials to be used to access the WS. The authentication will be based on the generic HTTP-basic mechanism but this may change to strengthen security. The receiver will verify that the essential information is present in the data message and contains valid information: a. sendercode must exist and be registered in DCF and for the data collection described in the dccode element; b. A valid reference to an XML schema is reported in the document. c. receivercode must be EFSA ; d. type must be a supported document type (e.g. dcfmsg ); e. version contains a supported version of the protocol (e.g. 1.0 ); f. sendertrxcode contains an identifier 15

16 g. receivertrxcode contains a valid and existent receivertrxcode in case of optype of update or delete; h. dccode: contains a valid data collection code registered in the DCF system. Where these checks are positive then the Receiver will confirm the transmission with a synchronous response TRXOK and the protocol will proceed with the MRN generation; otherwise the Receiver will generate a TRXKO and the transmission will be terminated. 2. The Receiver checks asynchronously whether the input file is well-formed and valid according to the XML schema. The Receiver will send back a MRN calling the sendresumedata method WS exposed function on the Data sender side. The Sender WS shall be security protected and the sender shall provide EFSA with a user ID and password to access their WS. The Receiver expects a positive response to the MRN transmission through a TRXOK. In case a TRXKO is received, the protocol is stopped and the data message is considered not to have been received. If the MRN was positive, the receiver moves to point 3; otherwise the protocol stops and the data message is considered not to have been received. 3. The Receiver will execute further validations applying business rules on the original file asynchronously. At the end of this process the Receiver will send back an Acknowledgment to the Sender containing the validation results calling the sendresumedata method of the WS of the Data Sender. The Receiver expects a positive response to the MRN transmission through a TRXOK. When a TRXKO is received, the protocol is stopped and the data message is considered not to have been received. 4. The Sender will validate the acknowledgment message and sends back an MRN to the Receiver calling again the sendresumedata method to notify the receipt of the acknowledgment message. The Receiver will validate the MRN and generate a positive response to the MRN transmission through a TRXOK. In case of errors in the MRN a TRXKO is sent back by the receiver, the protocol is stopped and the data message is considered not to have been received. In addition, the data message is considered not to have been received if the MRN transmitted by the sender is negative. 16

17 MemberState EFSA WebService Provider EFSA_WSDL IDs WebService Client elect Backend WebService Provider Internet SOAP / HTTPS Backend MS_WSDL WebService Client elect Figure 5: Web services architecture diagram. The Article 36 beneficiaries in CFP/EFSA/DATEX/2009/01 and EFSA IT will investigate possible enhancements in the implementation of the SOAP protocol particularly the synchronous MRN and the option of including information from the data message in the SOAP message. 6. XML messaging EFSA has currently set up a system, supporting XML messages to pilot this type of data transmission during the article 36 projects CFP/EFSA/DATEX/2009/01 and Pesticide Residues Monitoring This system already includes a data message, an MRN message and an acknowledgment message. The Working group revised and modified the XML schemas for the data message, the MRN and the acknowledgment. Particularly the names of some data elements were changed. The main changes were performed to the acknowledgment message to support a better reporting of the detected errors and warnings to the sender. The acknowledgment now includes only an error summary, while the detailed error report has been moved to a separate file which can be downloaded by the sender after the acknowledgment has been received. While the EFSA system already supports the new version of the data message, the MRN and ACK are still aligned to their former versions, which, for completeness, are listed in Appendix 1. The new versions of the data message, MRN and acknowledgment, as agreed by the Working Group, are described in the paragraph below Data message The Data Message is comprised of three complex elements containing data fields: The message header: contains information to allow the communication protocol to deliver the MRN message to the sender. 17

18 The data transmission (datatrx): contains the information if the Sender is performing an insert (code 01), update (code 03) or a delete (code 02) operation. It defines also the data collection to which the data contained must be appended. Dataset: contains the analytical measurement data as defined in the Standard Sample Description (SSD). The Data message diagram reports the structure of the MRN message, which is then further detailed in Data message elements. The element datatrx related attributes is defined as an attribute in the data message and as an element in the MRN and ACK messages. This is only due to compatibility and historical reasons. 18

19 Figure 6: Data message diagram. 19

20 Table 1: Data message elements. Element Name Type Description Mandatory Message Element Message element Header Element Message header type xs:string (20) Values: dcfmsg version xs:decimal (5, 2) Values: 1.0 Type of transmission Message version supported in this transmission. code xs:string (100) Unique code for the file transmitted, the file name could be used if unique. receivercode* xs:string (100) Code representing the organisation receiving the data transmission sendercode* xs:string (100) Code representing the organisation sending the data. sentdate xs:datetime Date the message has been sent from the sender; in case this date cannot be produced this element should include the date the data message was created. datatrx Element The data transmission part of the message contains the data for a specific data collection. receivertrxcode xs:string (100) Identifier of the data message (identified by the element message ) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication. sendertrxcode xs:string (100) Identifier of the data message (identified by the element message ) inserted in the receiver database. No optype xs:string (2) Values: Operation to be performed on the dataset: Insert=01, Delete=02, Update=03 15 dccode* xs:string (100) Identifier for the data collection. dcname* xs:string (500) Name for the data collection. No trxcomm xs:string (100) Text comment for this transmission No * These codes are defined by EFSA and provided to the sender at the beginning of the reporting period. 15 The update procedure is not implemented in the current version of the Message Transmission Protocol 20

21 6.2. MRN message The message structure is made up of two elements containing data fields: The message header: contains information to allow the communication protocol to deliver the MRN message to the sender The message MRN: contains the information if the message has been accepted for processing by the receiver and whether the file is well formed. In addition the MRN body contains the transmissionid (receivertrxcode) which allows the sender to find the transmitted dataset in the receiver s database. Even if the message transmission receives a positive MRN, the actual acknowledgment message could be negative. The MRN contains only the information that the message is well-formed and valid according to the defined XML schema, and can be processed by the receiver. The MRN does not contain the actual result of the business validation of the content of the XML message. The actual result of the business rules validation process is delivered by the acknowledgment message. The MRN message diagram reports the structure of the MRN message, which is then further detailed in Message MRN elements, where all elements included in the MRN message are listed. Figure 7: MRN message diagram. 21

22 Table 2: Message MRN elements. Element Name Type Description Mandatory header Element Message Receipt Notification (MRN) message header. The message header has the same elements as in the XML message mrnbody Element Content of the MRN message sendertrxcode xs:string (100) Identifier of the data message (identified by the element message ) inserted in the receiver database. receivertrxcode xs:string (100) Identifier of the data message (identified by the element message ) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication. msgreceivedate xs:datetime This field specifies when the message has been received by the receiver's system. dccode xs:string (100) Identifier for the data collection. dcname xs:string (500) Name for the data collection. No mrndate xs:datetime This field specifies when the MRN message has been created by the receiver's system. mrncode xs:string (2) Values: This field contains the MRN code. "01" means transmission accepted. "02" means transmission not accepted because of parsing errors in the message, or for errors at security level. parsingerror xs:string (4000) This element will contain the parsing error generated checking if the transmission file is a well-formed XML file and if it is valid according to XML schema. No To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example can be found in the annexed zip file (XML.zip) with the name formatmrn.xslt Acknowledgment Message The message structure is embodying three elements containing data fields: The message header: Contains information to allow the communication protocol to deliver the acknowledgment message to the sender. The message acknowledgment (msgack): Contains information on the result of the entire message processing. If the transmission acknowledgment code (trxackcode) contains the value: 22

23 o o o 01 : the message was loaded by the receiver system without errors and warnings. No further action is required by the sender of the message. 02 : the message was not loaded by the receiver system because of errors in the message. The sender has to correct the message and resubmit the data to the receiver : the message was loaded by the receiver system with warning, but with no errors. The sender may decide to correct the fields generating warnings but the action is not required. In addition this entry contains the transmissionid which allows the sender to find the transmitted dataset in the receiver s database. It is paramount that the sender keeps track of the transmissionid held in the receivertrxcode. The sender will have to use the transmissionid for any further operation of replacement or nullification (deletion) of the dataset. The transmission acknowledgment (trxack): This entry contains the result of the validation of the data contained in the data message in the form of error codes and error descriptions. The data fields in this entry provide to the sender information with the necessary information required to prepare a new data submission with the identified errors corrected. If no errors or warning have been detected by the receiver s system, then the entity transmission acknowledgment (trxack) is not generated. The Acknowledgment diagram reports the structure of the acknowledgment message, which is then further detailed in Acknowledgment elements, where all elements included in the acknowledgement are listed. 16 The element ackcode specifies a value 03 for warning to keep compatibility between the trxackcode of the ACK message and the mrncode of the MRN code. 23

24 Figure 8: Acknowledgment diagram. 24

25 Table 3: Acknowledgment elements. Element Name Type Description Mandatory ack element Acknowledgment element header element Message Acknowledgement (ACK) header. The message header has the same elements as in the XML message ackbody element Message Acknowledgement Body. msgack element This entity contains information on the overall validation of the message. msgreceivedate xs:datetime This field specifies when the message has been received by the receiver's system. msgackdate xs:datetime This field specifies when the acknowledgment message has been created by the receiver's system. trxackcode xs:string Values: This field contains the acknowledgment code. "01" means transmission successful. 02 means transmission not successful because of errors in the message. "03" means transmission successful with warning. 03 sendertrxcode xs:string Identifier of the data message (identified by the element (100) message ) inserted in the receiver database. receivertrxcode xs:string Identifier of the data message (identified by the element (100) message ) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication. dccode xs:string This field contains the code of the data collection which is (100) involved in the data transmission. dcname xs:string This field contains the name of the data collection involved (500) in the data transmission. trxack Entity List of elements reporting the summary acknowledgment information. errref xs:string This attribute contains the place where the detailed error list (500) can be downloaded. ErrorInfo Entity This entity provides summarised information regarding warnings and errors. errortype xs:string (1) The type of information the acknowledgment is returning. "E" Error, "W" Warning, I information. rulecode Xs:string Code of the validation rule applied to the element. (10) errorcode xs:string This field contains the error code generated by the receiver (10) system. errordescription xs:string This field contains the text description of the error, warning (500) generated by the receiver's system. var xs:string The name/s of the elements/s tested in the validation rule (can (1000) contain $ separators). example xs:string An example of the combination of data message values that (4000) failed the validation rule (can contain $ separators). records xs:long Number of records (Results in case of Standard Sample Description message) which failed this validation. No No No No No No 25

26 To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example and an example can be found in the annexed zip file (XML.zip) with the name formatack.xslt. The acknowledgment message, as anticipated, contains only a summary of the errors detected in the sender s message. In case the sender wants to access the full list of errors, they can be downloaded following the URL link contained in the acknowledgment message, in the attribute errref. The XML schema for the detailed error file is displayed in Figure 9 Detailed error report, which is then further detailed in Acknowledgment elements, where all elements included in the error report are listed. Figure 9: Detailed error report. 26

27 Table 4: Error report elements. Element Name Type Description Mandatory errackdetails element List of detailed errors encountered during the business rules validation process. receivertrxcode xs:string (100) Identifier of the data message (identified by the element message ) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication. errackdetail element This entity contains the output of an individual validation using the business rules. errorseq xs:long Identification number of the error in the file. errortype xs:string (2) The type of information the acknowledgment is returning. "E" Error, "W" Warning, I information. rulecode Xs:string (10) This field contains the code of the rule generating the error. errorcode xs:string (10) This field contains the error code generated by the receiver system. errordescription xs:string (500) This field contains the text description of the error, warning generated by the receiver's system. Var xs:string (250) List of the variables involved in the validation rule. No varval xs:string (250) List of variable values that are involved in the validation rule. recordcode xs:string (20) The "resultcode" from the data message file, identifying the result which failed validation. No No To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example can be found in the annexed zip file (XML.zip) with the name formatdetailederror.xslt. 7. Error Codes and Messages In general three steps of validation can be distinguished during the processing of an XML file. The steps are strongly consecutive. The three steps must be executed in sequence and the following step cannot be performed if the previous step was not successfully completed. 1. Check whether the XML is "well-formed": The file is tested to verify that it satisfies a list of general XML syntax rules, provided in the XML W3C specification, to verify if a generic file is an XML file. This control is performed automatically by any XML parser. The data message is verified well-formed upon arrival at EFSA and the output of this parsing process is part of the MRN message. The parsing error messages, if generated, will be inserted in the element parsingerror. 27