Public Report Retrieval

Transcription

1 PUBLIC IESO_SPEC_0110 Public Report Retrieval Anonymous FTP Site Issue 2.0 This document outlines acceptable practices to be used when retrieving documents from the IESO aftp site. Public

2 The posting of documents on this Web site is done for the convenience of market participants and other interested visitors to the IESO Web site. Please be advised that, while the IESO attempts to have all posted documents conform to the original, changes can result from the original, including changes resulting from the programs used to format the documents for posting on the Web site as well as from the programs used by the viewer to download and read the documents. The IESO makes no representation or warranty, express or implied, that the documents on this Web site are exact reproductions of the original documents listed. In addition, the documents and information posted on this Web site are subject to change. The IESO may revise, withdraw or make final these materials at any time at its sole discretion without further notice. It is solely your responsibility to ensure that you are using up-to-date documents and information. This document may contain a summary of a particular market rule. Where provided, the summary has been used because of the length of the market rule itself. The reader should be aware, however, that where a market rule is applicable, the obligation that needs to be met is as stated in the Market Rules. To the extent of any discrepancy or inconsistency between the provisions of a particular market rule and the summary, the provision of the market rule shall govern. Disclaimer Document ID IESO_SPEC_0110 Document Name Public Report Retrieval Issue Issue 2.0 Reason for Issue Revised to reflect new URL Effective Date September 14, 2005

3 Public Report Retrieval Document Change History Document Change History Issue Reason for Issue Date 1.0 First Issue June 1, Revised to reflect new URL September 14, 2005 Related Documents Document ID Document Title Issue 2.0 September 14, 2005 Public

4 Document Control IESO_SPEC_0110 Public Issue 2.0 September 14, 2005

5 Public Report Retrieval Table of Contents Table of Contents Table of Contents...i List of Figures...ii List of Tables...iii Table of Changes...iv 1. Introduction Purpose Scope Who Should Use This Document Conventions How this Document is Organized IESO Anonymous FTP Site Overview Available Documents File Formats HTML CSV XML Retrieving IESO Documents Document Naming Conventions File Directory Structure Manual Access HTML Interface FTP Interface Programmatic Access Retrieving a Report for a Specific Reporting Period Retrieving the Latest Report Recommended Polling Frequency...8 References...1 Issue 2.0 September 14, 2005 Public i

6 List of Figures IESO_SPEC_0110 List of Figures None ii Public Issue 2.0 September 14, 2005

7 Public Report Retrieval List of Tables List of Tables Table 1: Document Naming Conventions...4 Table 2: Recommended Polling Frequency...8 Issue 2.0 September 14, 2005 Public iii

8 Table of Changes IESO_SPEC_0110 Table of Changes Reference (Section and Paragraph) Description of Change iv Public Issue 2.0 September 14, 2005

9 Public Report Retrieval 1. Introduction 1. Introduction 1.1 Purpose 1 This document outlines acceptable practices to be used when retrieving documents from the IESO aftp site. 1.2 Scope 2 This document provides guidelines on when and how to download published documents from the IESO aftp site. These guidelines take into consideration the need to minimize the load on IESO servers and to make reports available in a timely manner to market participants. 1.3 Who Should Use This Document 3 This document is intended for market participants who require access to information published in IESO reports. The document is to be used as a guideline in developing / refining dataretrieval mechanisms used to obtain information from the IESO. It is highly recommended that market participants use and adhere to the guidelines specified in this document, as misuse of IESO web sites may result in users being blocked from retrieving data from the web sites. 1.4 Conventions 4 The standard conventions followed for this document are as follows: The word 'shall' denotes a mandatory requirement; Terms and acronyms used in this document that are italicized have the meanings ascribed thereto in Chapter 11 of the "Market Rules"; Double quotation marks are used to indicate titles of legislation, publications, forms and other documents. 1.5 How this Document is Organized 5 This document is organized to first present an overview of the IESO aftp site, followed by mechanisms in which hosted documents may be retrieved in compliance with IESO policy. End of Section Issue 2.0 September 14, 2005 Public 1

10 2. IESO Anonymous FTP Site IESO_SPEC_ IESO Anonymous FTP Site 2.1 Overview 6 The IESO publishes information for retrieval by market participants in three locations; a. Public documents on the IESO anonymous FTP (aftp) web site b. Public and confidential documents on the IESO Report web site c. Confidential documents via the Market Participant Interface (MPI) These dedicated sites have been established to provide standard interfaces for the retrieval of required information. As such, data retrieval from these sites is highly recommended as an alternative to scraping information directly from the IESO public website. HTML documents on the IESO public website may be altered or their location changed without notice, potentially causing programs which rely on the stability of these HTML files to malfunction. This document describes reports available through the IESO aftp site and mechanisms for retrieving required information. Information on documents provided on the IESO Report Site can be obtained through Quick Take 15: Retrieving Reports via New IESO Web. Also please refer to the IESO Market Manual 6: Participant Technical Reference Manual, Section 2, for the required workstation and browser requirements and settings. 2.2 Available Documents 7 The IESO aftp web site offers documents in HTML, CSV, and/or XML file formats, depending on the report. Specific details on available documents, publishing frequencies, and samples of reports are available on the Technical Interfaces page of the IESO public website. Reports hosted on the IESO aftp server contain detailed price and demand information for the IESO-administered markets and fall under the following general categories: Security and Adequacy Assessment / System Status Reports Pre-dispatch & Dispatch Transmission Rights Auction Transmission Outages Hourly Ontario Energy Price (HOEP) Generator Disclosure Reports Zonal Demands 2 Public Issue 2.0 September 14, 2005

11 Public Report Retrieval 2. IESO Anonymous FTP Site 2.3 File Formats 8 The IESO employs the HTML, CSV, and XML file formats to distribute information to market participants and the general public. It is strongly advised that CSV or XML versions of files are used, where available, when designing programs to read information from IESO documents. These file formats contain information in fixed locations, facilitating programmatic access to data. HTML files are intended for visual reference only. Template specifications for many of the report file formats are available on the Technical Interfaces page of the IESO Public Website HTML 9 The HTML file format is designed such that the display of data is intrinsically tied to its location in the file. Changes to the look and feel of an HTML page will thus disrupt the position of information contained within the file. Since the IESO may alter the display of information provided in HTML files without notice, programmatic interfacing with these files is not recommended and can result in market participants' applications malfunctioning as files are changed. Instead, HTML files are provided by the IESO as a user-friendly display of information, viewable within a web browser CSV 10 The IESO has provided Microsoft Excel comma-separated value (CSV) versions of most HTML documents hosted on the aftp site. Each CSV file contains all data present in the corresponding HTML file and organizes information in a standard format suitable for reading using Microsoft Excel or for programmatic interfacing. The IESO recommends using these files to obtain required information, as the location of data will remain fixed within the file XML 11 Some files on the IESO aftp site have been provided in the XML file format. XML provides a clear delineation between the presentation of information and the information itself, making this file format a good candidate for programmatic interfacing. It is not, however, considered to be a human readable format, and information contained in these files is usually displayed with the help of applications like Microsoft Excel and Internet Explorer to format the data. Instructions for viewing and printing IESO reports formatted in XML are available in the documents Quick Take 12: Accessing IESO Reports Formatted in XML, and Viewing XML documents in Excel. For more information on the XML file format, please refer to the document XML A Primer. Note: Reports in XML format were provided on the aftp site as a transitional arrangement until the IESO Report site was available in June It is planned to remove the XML reports from this site in the near future and hence it is recommend that participants use the IESO Report Site at: to retrieve these reports. Issue 2.0 September 14, 2005 Public 3

12 3. Retrieving IESO Documents IESO_SPEC_ Retrieving IESO Documents 12 The IESO aftp site is accessible to both market participants and the general public for downloading reports issued by the IESO. 3.1 Document Naming Conventions 13 In order to retrieve documents from the IESO, naming conventions for IESO public reports must be understood. Each file name contains information about the document; security class, document id, date, time, version, and format, and is composed as follows: securityclass_documentid_datetime_version.format Examples: PUB_GenOutputCapability_ xml PUB_SSR_ _v3.0.htm PUB_DispConsRegional_ csv Table 1: Document Naming Conventions Identifier Description Example securityclass documentid datetime All files are marked with their security class; PUB for public reports, CNF for confidential reports, or HCNF for highly confidential reports. Unique IESO identifier for each report describing the document. (YYYY) year for annual reports (YYYYMM) year and month for monthly or quarterly reports; month is last month in quarter for quarterly reports (YYYYMMDD) year, month and day for daily or weekly reports; day is last day in week for weekly reports (YYYYMMDDHH) year, month, day and hour for hourly reports; hour is hour ending (YYYYMMDDHHNN) year, month, day, hour, and minute for 5-minute interval reports; minute is last minute in interval, i.e. intervals 0, 5, 10, Since all reports provided by the FTP service are public, all documents accessed from this site will be prefixed with PUB_. SSR1_ SSADaily1_ YYYY 2002 for year 2002 YYYYMM for November of 2002 YYYYMMDD for November 6, 2002 YYYYMMDDHH for hour 13 on November 6, 2002 YYYYMMDDHHNN for the fifth interval (minutes 20-25) in hour 4 Public Issue 2.0 September 14, 2005

13 Public Report Retrieval 3. Retrieving IESO Documents Identifier Description Example version format on November 6, 2002 Versions apply for reports reissued for one reporting period where there is a requirement to audit the differences between the initial issued report and subsequently issued reports. Versions are of form vn.d where N is a version number and D is a draft version number. When N is greater than zero, D will be zero. When N is zero, D will be greater than zero The format of the file for HTML, CSV, and XML files. v1.0 is the 1st version v0.3 is the 3rd draft version HTML CSV XML.csv.htm or.html For more details on IESO document naming conventions, please refer to the document External Document Naming Convention Information Technology Standard..xml 3.2 File Directory Structure 14 To locate reports hosted by the IESO FTP service, the user will need to be aware of the file directory structure of the aftp server. The root directory of the aftp server, PUB, is indicated in the URL ftp://aftp.ieso.ca/pub/reports/pub/. All public reports can be accessed in subdirectories under this root directory. Subdirectory names are based on the documentid of the reports contained within. Example: PUB_SSR_ _v3.0.htm, is located with other SSR reports in the directory SSR, under the root directory. 3.3 Manual Access 15 Reports issued by the IESO may be accessed manually either through a direct connection to the FTP service, via a web-browser or other FTP client software, or by using the Report Navigator User Interface provided by the IESO on the Market Data page of the IESO Public website HTML Interface 16 The Report Navigator User Interface is a user-friendly layer on top of the FTP service which enables users to view documents published by the IESO within their web browser. Accessed through links on the Market Data page of the IESO public website, this interface allows for Issue 2.0 September 14, 2005 Public 5

14 3. Retrieving IESO Documents IESO_SPEC_0110 more intuitive access to reports than offered by direct connection to the ftp service. Documents offered through this interface are restricted to those formatted in either HTML or XML which have been issued within the current month. To download reports, and to access historical reports and those offered in the CSV file format, a direct connection to the FTP service is required FTP Interface 17 While it is more difficult to navigate, a direct connection to the FTP service provides faster access to both current and historical public reports than is available through the HTML Interface. Access to the FTP service requires a connection between the service and an FTP client via the URL ftp://aftp.ieso.ca/pub/reports/pub/ Internet Explorer and Netscape Navigator are both FTP-enabled web browsers, making them ideal candidates for accessing the FTP service. Connecting to the FTP service using a web browser simply requires clicking on the URL above or typing it into the address field of your web browser. Alternatively, several commercial and freeware FTP clients are available on the web and can be used to connect to the IESO FTP server. Refer to the help manuals of these software packages for assistance in configuring downloaded FTP clients. 18 Once the desired file is located on the FTP server (see section 3.2), it may be opened and viewed by double clicking on the filename. Downloading the file is achieved by clicking with the right mouse button on the filename. This activates a menu, which allows the user to copy the file from the FTP server to a location on the user s local drive. 3.4 Programmatic Access 19 To access the IESO aftp site programmatically, the basic concepts of the File Transfer Protocol will need to be understood by the programmer as well as the file directory structure of the FTP server (see section 3.2) Retrieving a Report for a Specific Reporting Period 20 The recommended programmatic access to retrieve a report for a specific reporting period is as follows: 1. Build the URL using the FTP service URL, ftp://aftp.ieso.ca/pub/reports/pub/, along with the documentid to target the directory of the desired report. Use this URL to FTP into the IESO FTP service. (e.g. ftp://aftp.ieso.ca/pub/reports/pub/ssr for System Status Reports). 2. Obtain a directory list of the available report files. 3. Filter the list for report(s) with file names containing the desired datetime identifier. 6 Public Issue 2.0 September 14, 2005

15 Public Report Retrieval 3. Retrieving IESO Documents 4. If no file is found with the desired date and time, pause for the recommended polling frequency time (stated in Table 2 below) and repeat steps 2 and Download (FTP GET) the desired file(s). 6. Save the file(s). 7. Exit program. 21 Alternatively: 1. Build the URL using the FTP service URL, ftp://aftp.ieso.ca/pub/reports/pub/, along with the documentid to target the directory of the desired report. Use this URL to FTP into the IESO FTP service. (e.g. ftp://aftp.ieso.ca/pub/reports/pub/ssr for System Status Reports). 2. Attempt to download (FTP GET) the desired file without checking for its existence. (eg. ftp://aftp.ieso.ca/pub/reports/pub/ssr/pub_ssr_ _v3.0.htm for the third version of the System Status Report for February 12, 2005). 3. If HTTP error 404, Object not found, occurs then pause for recommended polling frequency time and repeat step Save the file. 5. Exit the program Retrieving the Latest Report 22 The recommended programmatic access to retrieve the latest report is as follows: 1. Build the URL using the FTP service URL, ftp://aftp.ieso.ca/pub/reports/pub/, along with the documentid to target the directory of the desired report. Use this URL to FTP into the IESO FTP service. (e.g. ftp://aftp.ieso.ca/pub/reports/pub/ssr for System Status Reports). 2. Obtain a directory list of the available report files. 3. Identify new report files, which have a file date and time stamp greater than the last date and time for which the report subdirectory was polled. 4. If no new files are found, pause for the recommended polling frequency time (stated in Table 2 below) and repeat steps 2 and Download (FTP GET) the desired file(s). 6. Save the file(s). 7. Stop polling for at least the recommended polling frequency for the desired report. 8. Exit program. 23 Alternatively: 1. Build the URL using the FTP service URL, ftp://aftp.ieso.ca/pub/reports/pub/, along with the documentid to target the directory of the desired report. Use this URL to FTP into Issue 2.0 September 14, 2005 Public 7

16 3. Retrieving IESO Documents IESO_SPEC_0110 the IESO FTP service. (e.g. ftp://aftp.ieso.ca/pub/reports/pub/ssr for System Status Reports). 2. Attempt to download (FTP GET) the desired file without first checking for its existence. (e.g. ftp://aftp.ieso.ca/pub/reports/pub/ssr/pub_ssr_ _v3.0.htm for the third version of the System Status Report for February 12, 2005). 3. If HTTP error 404, Object not found, occurs then pause for recommended polling frequency time and repeat step Save the file. 5. Stop polling for at least the recommended polling frequency for the desired report. 6. Exit the program. 3.5 Recommended Polling Frequency 24 The recommended polling frequency is based upon the reporting period for the specific document as stated below. Table 2: Recommended Polling Frequency Document Reporting Period Recommended Polling Frequency 5 minutes 15 to 30 seconds Hourly 4 to 10 minutes Daily 30 to 60 minutes Note: Programs which poll the IESO site with a higher frequency than the published acceptable levels, whether due to malfunction or bad design, will be blocked from using the IESO sites. End of Section 8 Public Issue 2.0 September 14, 2005

17 Public Report Retrieval References References Document Name Market Manual 6: Participant Technical Reference Manual Section 2.0: Participant Computer, Network & Security External Document Naming Convention; Information Technology Standard Document ID IMO_MAN_0024 IESO_ISTD_0001 End of Document Issue 2.0 September 14, 2005 Public References 1