Regulation F1: EDI communication with the DataHub in the electricity market Appendix report 1: Syntax and structure of EDIFACT and XML messages October 2011 Version 3.0 Effective from 1 October 2012 1.0 2.0 16-6-2010 24-6-2010 27-6-2010 DATE BOO MBN JHH NAME 25-2-2011 1-3-2011 22-3-2011 14-06-2012 DATE FSD CCO JHH JHH NAME REV. DESCRIPTION PREPARED CHECKED REVIEWED APPROVED 64820-10 Energinet.dk DOC. NO.
Table of contents 1. Objective... 3 2. References... 4 3. EDIFACT syntax and structure... 5 3.1 EDIFACT structure and terms... 5 3.2 EDIFACT syntax... 6 3.3 Undescribed segments and elements in the message implementation guides... 7 3.4 Interchange control reference, message ID and interchange version... 7 3.5 Fixed segments... 7 4. XML syntax and structure... 11 4.1 General syntax rules... 11 4.2 XML namespace and versioning for messages... 12 4.3 Validation against XML schema... 13 4.4 Description of special fields... 13 4.5 UN/CEFACT Core Components... 13 Doc. 64820/10, Case 10/3365 2/14
1. Objective This appendix report describes current syntax and structure rules for the formats EDIFACT and XML. Both formats are used for the exchange of messages in the electricity market. It is therefore a requirement that the rules stipulated in the appendix report are observed to ensure good and seamless communication. The two formats are considered in separate sections. Doc. 64820/10, Case 10/3365 3/14
2. References This section includes a table specifying the references of Appendix report 1. The documents and information sources referred to may have been moved or changed. Consequently, the organisation responsible and version number have been included to facilitate any alternative searches if the reference has become obsolete. Organisation Document/ comments Reference Version ENTSO-E Central knowledge portal for European TSOs. Describes the upstream electricity market. http://entsoe.eu/ ENTSO-E ENTSO-E Scheduling System ESS. http://entsoe.eu/index.php?id =73 Version 3 R1 Used for submitting notifications and schedules. ebix CuS & EMD projects http://www.ebix.org/content. UN/CEFACT W3C (World Wide Web Consortium) Joint Syntax Working Group UN/CEFACT naming and design rules for XML. Description of the XML standard Group under UN/CEFACT working with EDIFACT syntax versions aspx?contentid=990&selecte dmenu=3 http://www.unece.org/cefact/ xml/uncefact+xml+ndr+v 3p0.pdf http://www.w3.org/standards /xml/ http://www.gefeg.com/jswg/ 2010.B version 3.0 Doc. 64820/10, Case 10/3365 4/14
3. EDIFACT syntax and structure This section describes the technical syntax and structure of the EDIFACT standard. The contents are based on the Common rules and recommendations of ebix. 3.1 EDIFACT structure and terms 1 EDIFACT is a United Nations standard (UN/CEFACT) used for structuring and transporting data. This means that EDIFACT is subject to syntax rules, which apply at all times and must always be complied with. EDIFACT contains the following terms: An interchange consists of all information in a transmission. A physical interchange can comprise one or more messages. An interchange has one sender and one receiver. A message is for instance: o Metered data o Notifications and schedules In Denmark, the general rule is that only one message per interchange is allowed. A segment is a group of related data elements/fields. It could, for instance, be an address. In the segment the address is specified by indication of name, road, postal code, town, country, etc. A composite data element comprises two or more data elements. A data element/field is the individual information transmitted in a segment. This may be a time indication or number of kilowatt hours. Physical interchange UNA UNB Message Message Message UNZ UNH Segment Segment Segment Segment UNT Data element+composite data element+composite data element+composite data element+... Data element:data element:data element:data element: Data element: Figure 1: EDIFACT structure 1 The section is based on UN/CEFACT http://www.gefeg.com/jswg/ Doc. 64820/10, Case 10/3365 5/14
The figure above illustrates the interconnected use of the four terms. UNA, UNB and UNZ are segments used for marking the beginning and end of an interchange. UNH and UNT are segments used for marking the beginning and end of a message. The following characters are used for marking separation, end, etc. in connection with data elements and segments: + Marks separation between data elements and between segments. ' Marks that a segment is finished. : Marks separation between the individual data fields in composite data elements; this means more fields describing the same. 3.2 EDIFACT syntax Basic rules for messages used in the electricity market: 3.2.1 Character set and structure Send EDIFACT interchanges as one long string without changing lines. To ensure conformity between messages irrespective of format, the UTF-8 character set (also known as UNOW) must always be used. Completely fill in yymmdd and hhmm in the UNB. Hence, it is insufficient to send, for instance, 645 to indicate the hour. To be correct, it should be 0645. Always write message names and segment names in UPPERCASE when they are transmitted. EDIFACT version 3 is used in Denmark, with the exception of character sets where UNOW (UTF-8) is used from syntax version 4. Denmark thus deviates from the standard. 3.2.2 Use of reserved characters The UNA segment describes the separators (reserved characters) used in the transmitted EDIFACT. The standard is: UNA:+.? If one of the reserved characters (+, :, ') is used, it should be preceded by a question mark (?). Thus, for the addition of two numbers (eg 10 + 20), write 10?+20. If a question mark is needed, write two question marks (??). The first question mark restores the normal meaning of the next one. If a release character is used, it is not included in the determination of the field length. If a field allows 20 alphanumeric characters and two release characters are needed, the field may be up to 22 characters long. 3.2.3 Avoid superfluous signs Delete trailing plus signs (+) if no data is available. Delete trailing colons (:) in each composite data element (see Figure 1) if they are not data. Do not delete leading plus signs (+) or colons (:). For instance, you cannot send 24500 + 45 + + + + + +, you need to send 24500 + 45, ie shorten where possible. Similarly, you cannot send 24500:45::::::::, you need to send 24500:45. Doc. 64820/10, Case 10/3365 6/14
The example below shows a wrongly and a correctly formatted NAD segment (wrong characters indicated in BOLD): Not shortened NAD+DP++++Energinet.dk::+Tonne Kjærsvej 65::+ Fredericia::++7000++' Correctly shortened NAD+DP++++Energinet.dk+ Tonne Kjærsvej 65 + Fredericia ++7000' 3.3 Undescribed segments and elements in the message implementation guides EDIFACT messages must meet all rules set out in the regulation. If a player or the DataHub receives EDIFACT messages containing additional segments and elements not described in the relevant guidelines, these will be ignored. It must therefore not give rise to an error message report to the sender as long as the message complies with the basic structure of the EDIFACT message in question (eg UTILMD or UTILTS). 3.4 Interchange control reference, message ID and interchange version The interchange control reference is found in the S004.0020 element of the UNB segment and must be unique over time for the sender, as defined in S002. If this is not the case, the latest received message must be rejected automatically. Message ID is found in the C002.1204 element of the BGM segment and is used for the unique identification of a message. Message ID must be unique over time for each player. Interchange version is not used in EDIFACT. 3.5 Fixed segments Below follows a description of selected service segments (UN segments). UNA and UNB are used for the interchange header and can be regarded as the envelope of the interchange. UNB is used by the receiving EDI system for identification of sender and receiver. 3.5.1 UNA Syntax segment UNA Service String advice Function: For specification of notation in EDIFACT interchange Classification: Conditional, (1) Comments: The examples below are default. Used in Denmark. UNA UNA:+.? ' Ref. Name Cl. Form. Description COMPOSITE DATA M an1 : (colon) ELEMENT, SEPARATOR DATA ELEMENT, M an1 + (plus) SEPARATOR DECIMAL NOTATION M an1. (full stop) Doc. 64820/10, Case 10/3365 7/14
RELEASE CHARACTER M an1? (question mark) RESERVED FUTURE USE M an1 (blank) SEGMENT TERMINATOR M an1 ' (apostrophe) 3.5.2 UNB EDIFACT envelope for all documents except CONTRL UNB Function: Classification: Comments: Example: Interchange Header For identification and addressing of the interchange, specification of a request for acknowledgement as well as test indication. Mandatory (M1). None UNB+UNOW:3+5790000395620:14+5790000432752:14+07012 0:1606+31929++DK-TIS-MET++1+DK Ref. Name Cl. Form. Description S001 SYNTAX IDENTIFIER M 0001 Syntax identifier M a4 Code: UNOW (UTF-8) 0002 Syntax version number M n1 Code: 3 Version 3 of EDIFACT syntax. S002 INTERCHANGE SENDER M 0004 Sender identification M an..35 GLN or EIC number. 0007 Partner identification code qualifier 0008 Address for reverse routing S003 INTERCHANGE RECIPIENT M R an..4 Code: 14 GLN (Global Location Number) 305 EIC (European Identification Code) O an..14 Only if bilaterally agreed. 0010 Recipient identification M an..35 GLN or EIC number. 0007 Partner identification code R an..4 Code: qualifier 14 GLN (Global Location Number) 305 EIC (European Identification Code) 0014 Routing address O an..14 Only if bilaterally agreed. S004 DATE AND TIME OF PREPARATION M 0017 Date M n6 Date of interchange creation (YYMMDD). Doc. 64820/10, Case 10/3365 8/14
0019 Time M n4 Local time of day when interchange was created (YYMMDD). 0020 INTERCHANGE CONTROL REFERENCE M an..14 Unique reference assigned to the interchange. Must match data element 0020 in UNZ. Must be unique over time for sender, as defined in element S002.0004; otherwise the interchange must be rejected. S005 RECIPIENTS REFERENCE, PASSWORD X 0022 Recipient's reference/ password 0025 Recipient's reference/ password qualifier X X an..14 an..2 0026 APPLICATION REFERENCE O an..14 Not used. 0029 PROCESSING PRIORITY CODE 0031 ACKNOWLEDGEMENT REQUEST X a1 O n1 Regardless of contents, the DataHub consistently responds with an acknowledgement of receipt of syntax (positive or negative) in the current web services session. 0032 COMMUNICATION O an..35 Not used. AGREEMENT ID 0035 TEST INDICATOR O n1 Not used. Doc. 64820/10, Case 10/3365 9/14
3.5.3 UNZ UNZ Function: Classification: Comments: Example: Interchange Trailer For termination of interchange and specification of total number of messages in interchange Mandatory (M1). Data element 0020 must be identical to 0020 in UNB UNZ+1+358765298' Ref. Name Cl. Form. Description 0036 INTERCHANGE CONTROL COUNT 0020 INTERCHANGE CONTROL REFERENCE M n..6 Number of messages in interchange. M an..14 Must match data element 0020 in UNB segment. Doc. 64820/10, Case 10/3365 10/14
4. XML syntax and structure This section describes current XML syntax and structure provisions for the exchange of XML messages in the electricity market. The section focuses on selected syntax and structure rules particularly important to ensuring optimum data interchange with the DataHub. The section supplements the rules and provisions for the XML format specified by the organisation W3C 2. The XML format can be used in all standard messages in the electricity market. Messages for the retail market are based on the industry standard from ebix in addition to which, the UN/CEFACT methodology for naming and designing XML messages is used. 4.1 General syntax rules All messages in the electricity market are described by means of XML schemas (XSDs). An XML schema describes how XML messages are built. XML schemas and XML messages are subject to the overall XML standard specified by World Wide Web Consortium (W3C). The basic principle for naming and structuring XML messages is that they observe the naming and design rules described in the following documents: - 'UN/CEFACT XML Naming and Design Rules Technical Specification Ver. 3.0 3 ' (NDR) - The harmonised role model for the electricity market that has been developed by ebix and ENTSO-E 4 - ebix UML Model for the European Energy Market, ver. 2010B 5. The structure of the XML schemas includes a selection of the following XML elements: XML Declaration Schema Module Identification and Copyright Information Schema Start-Tag Includes Imports Element Declarations o Root Element Declarations o Global Element Declarations Type Definitions In practice, a message is composed of several XML schemas that together define the message structure. The individual schemas are considered as independent blocks that are combined into a complete message. 2 http://www.w3.org/standards/xml/ 3 http://www.unece.org/cefact/xml/uncefact+xml+ndr+v3p0.pdf 4 http://www.ebix.org/documents/ebix_efet_and_entso_e_harmonised_electricity_ model.pdf 5 http://www.ebix.org/documents/eem%202010.b.zip Doc. 64820/10, Case 10/3365 11/14
4.1.1 Use of encoding and characters All messages in the electricity market are formatted into the UTF-8 character set (backwards compatible with ASCII encoding). UTF-8 is a 2-bytes-percharacter encoding, which is a compressed version of Unicode. Encoding is specified at the beginning of the XML document as follows: <?xml version="1.0" encoding="utf-8"?> The XML syntax contains a number of reserved characters, which are not allowed to appear in the XML syntax (elements or attributes) or in the actual message contents. Reserved character Alternative notation < Can be written as < & Can be written as & 4.1.2 Case sensitive XML notation Note that the XML notation is case sensitive (does not apply to the actual contents of the element). The element <Identifikation></ Identifikation> is not the same as <identifikation></ identifikation> because of the difference between upper- and lowercase I/i. 4.1.3 Superfluous signs Leading or trailing 'space signs' (or other invisible 'white space' signs before the first or after the last visible character in a data set) are deleted before sending. 4.2 XML namespace and versioning for messages XML schemas apply a target namespace expressed as an URI 6 and defined by Energinet.dk. These could be: - http://www.energinet.dk/schemas/<subnamespace>/<document>/v<vers ion> - un:unece:260:data:eem-dk_acknowledgment XML schemas are versioned using the file name. The file name is created using the name of the XML schema root element combined with a version number. The combination of the two parts is separated by a - (hyphen). Examples: - MarketScheduleDocument-1.xsd - ebix_dk_requestchangeofsupplier_0p9p0.xsd Energinet.dk s XML schema files are available at https://edi.energinet.dk/schemas. Here examples of messages can also be found. 6 Uniform Resource Identifier Doc. 64820/10, Case 10/3365 12/14
For further information about namespace and versioning for web services see Appendix report 4 about web services. 4.3 Validation against XML schema XML schemas 7 define content, structure and types for XML messages. Using an XSD definition it is possible to: - describe the XML message content - validate the XML message - define data facets (restrictions on data contents) - define data patterns (data formats) The DataHub validates all XML messages against the related schema. Validation takes place in the same web service session, and the sender is immediately notified of the result. At all times, Energinet.dk determines the XML schema to apply for a given XML message. 4.4 Description of special fields A number of attributes in the form of code lists are used in the XML messages. To qualify data further, the organisation administering the current code list must be indicated. The need arises when, for instance, an area or a metering point is identified. To clearly identify one of the two mentioned values, two further attributes are attached to the value. This is done to qualify the responsible organisation charged with issuing unambiguous identification values. In XML, these attributes are termed schemeagencyidentifier or listagencyidentifier. The above attributes describing the organisations responsible are always attached to the attributes schemeidentifier and listidentifier containing the actual values. The attributes have the following structure: - schemeagencyidentifier and schemeidentifier - listagencyidentifier and listidentifier Their application is specified below. <EnergyBusinessProcess listidentifier="dk" listagencyidentifier="260">d02</energybusinessprocess> <Identification schemeagencyidentifier="9"> 5734567890123</Identification> 4.5 UN/CEFACT Core Components UN/CEFACT is an international e-business standard comprising some technical specifications, including: Modelling Methodology (UMM) Core Component Technical Specification (CCTS) 7 XML Schema Definitions (XSD) Doc. 64820/10, Case 10/3365 13/14
XML Naming and Design Rules (NDR) UML Profile for Core Components (UPCC) Core Component Technical Specification comprises Core Components (CC) and Business Information Entities (BIE). Core Components are used as a reference model for data modelling messages for interchange of data between two or more parties. The European organisation ebix s data model is an integral part of UN/CEFACT s data model. The data model for the Danish electricity market is an integral part of ebix s and UN/CEFACT s data models. However, the Danish model contains a number of extensions in relation to the ebix data model specific for the Danish electricity market. The Danish data model is documented through the use of class diagrams applying UN/CEFACT entities, including ABIE 8, BBIE 9 and ASBIE 10. The Danish data model is used to realise the DataHub s XML schemas. The relation between UN/CEFACT data model, ebix data model and the Danish data model is illustrated below. The relationship between the three data models is illustrated in Figure 2 below. UN/CEFACT data model ebix data model Data model for the Danish electricity market Danish extensions to the data model Figure 2: Relationship between data models 8 Aggregate Business Information Entities 9 Basic Business Information Entities 10 Association Business Information Entities Doc. 64820/10, Case 10/3365 14/14