Sydney Ph: Melbourne Ph: Adelaide Ph: LISAsoft. Bespoke Development

Transcription

1 Sydney Ph: Melbourne Ph: Adelaide Ph: LISAsoft Bespoke Development August 2008

2

3 MER Program Data Access Consultancy Monitoring, Evaluation and Reporting Reporting With the adoption of state-wide targets and standards for natural resource management in 2006, the Monitoring, Evaluation and Reporting 1 (MER) strategy was developed to refocus the resources of the New South Wales natural resource and environment agencies and provide a mechanism for the coordination of efforts between agencies, Catchment Management Authorities (CMA), local governments and other natural resource managers. This strategy is designed to make the best use of existing resource condition information in informing policy decisions, aid in the integration of disparate monitoring programs, measure progress against various state-wide and regional targets, and enable the production of state of the environment reports. LISAsoft was contracted to assist in the reporting stage of the MER strategy. The three major components we are involved in are the data aggregation, data publication and client configuration. Data Aggregation The monitoring data is collected by a number of agencies, often from outside sources such as local governments or commercial organisations. Before any reporting can be done, the data must be collected into a database, metadata must be produced and published, and the data analysed to determine what is relevant for publication. While much of this process was performed by the agency responsible for the collection of the data, LISAsoft supported agencies in the data preparation. Data Publication Once the data was available, LISAsoft was responsible to publishing it via a GeoServer WFS service. This involved the determination of appropriate xml standards for representing the data such that outside agencies can not only consume in, but perform meaningful analysis with it. Once these standards was identified and the mappings between the source data store and the resulting data format were produced, the WFS was configured to publish a variety of views of the data appropriate for such uses and map, tabular and graphical visualisation as well as bulk download. 1

4 Client Configuration The Department of Natural Resources publishes a large number of datasets via its Natural Resources Atlas (NRAtlas). The NRAtlas is a web based application that is used to visualise datasets and groups of related datasets within the pre-defined context of various base layers such as Catchment Management Authority boundaries, groundwater bores and river networks. Datasets can be visualised by location alone, or summarized using standard distribution indicators to show sites at risk. Each site can be viewed graphically, plotting measurements over dynamically or statically defined time ranges, and can be downloaded. Entire map views can be saved as static images, and datasets within the map extents downloaded in bulk. LISAsoft was responsible for the configuration of the NRAtlas to provide the appropriate visualisation for new datasets, as well as providing links to the dataset metadata and developing new conversion scripts for serving data through the application, such as the production of KML output. Bug fixes and feature enhancements were also performed on the NRAtlas as needed.

5 State of Catchment Reporting SoC Reports LISAsoft worked with the Department of Environment and Climate Change, NSW to build a pilot web application for State of Catchment (SoC) reporting. The application provides consistant presentations for multiple themes and presents data in mulitple ways, including map, graphical and textual summaries of various indicators of the health of a selected catchment, side-by-side comparisons of different catchment areas and changes in the state of health of catchments. In order to encourage the use of this system within the target departments, an Excel template was developed to allow data custodians to input data using familiar tools. These templates are then processed by automated scripts to extract and transform the data into a more usable form. The backend data connector is flexible enough to support a variety of different backend data stores. The current scripts process incoming data into GeoJSON stored in files on the server, but the anticipated data source for a production system is a complex features enable WFS service, such as GeoServer. Due to the automated nature of the data extraction process, data updates can be accomplished in a near real-time and can incorporate business constraints such as data reviews and integrity checks.

6

7 AWDIP Testing The Australian Water Data Infrastructure Project 1 (AWDIP) was established to facilitate Australia-wide assessments of water resources through ongoing development of a comprehensive and accessible water information framework. The AWDIP was established under the national component of the Natural Heritage Trust. The AWDIP is a framework for a network of distributed hydrological databases. The framework enables on-line access to State and Territory agencies hydrographic data sets representing Australia s first distributed database of national natural resource management data. The infrastructure is designed to be interoperable, that is with the capability of exchanging data via a common set of business procedures, using standard file formats and protocols. Implementation of the Australian Water Data Infrastructure does not require specific operating systems nor applications, instead relying on Open Standards developed by World Wide Web Consortium (W3C Extensible Markup Language), Open Geospatial Consortium (OGC Observations and Measurements, Geographic Markup Language and Web Feature Services) and International Organisation for Standardization (ISO Geographic Metadata (ISO19139)). The use of these Open Standards permits well understood and unambiguous communication between the components of the infrastructure. In this way the different parts of the system communicate in a pre-defined way, enabling on-the-fly querying of the databases, and delivery of responses that are consistent and readily interpreted. The use of the standards also allows the AWDIP to take advantage of developments in similar implementations in other disciplines. 1

8 LISAsoft was contracted to develop a test suite to test the performance, robustness and conformance of WFS implementations to determine their acceptability for use in the AWDIP. The findings were compiled and reported back to the AWDIP along with recommendations on improving the underlying software. Community Schemas GeoServer had been identified as the best candidate WFS for use in the AWDIP. With the recent addition of Community Schema support, added in release 2.6, GeoServer is able to fully represent the complex feature types required by the AWDIP. Community Schemas leverage the complex features profile of GML, allowing richly defined semantics than is available in GML simple features profile. This allows communities, such as the hydrology community, to define schemas appropriate for their data. These schemas can then be referenced to ensure a consistent structure and taxonomy between related datasets, improving the communities ability to share data. For example, a hydrology schema may define a class called Water Use with acceptable terms defined as agricultural, domestic, and industrial. If a dataset is published against this schema with a Water Use value of irrigation, the data will not validate and the user will know to correct the mistake. More information on Community Schemas is available on the CSIRO website 2. DuckHawk In order to produce the test suite required, an open source testing framework called DuckHawk 3 was developed. This suite supports the development of reliability, load, performance, stress, error handling and conformance tests. The framework is configured using JUnit to allow the implementation of continuous testing, but will be extended to work in any number of continuous integration and build systems. After the framework was put in place, the project specific tests were be developed as pluggable modules, validating the utility and flexibility of the DuckHawk framework. DuckHawk lives on as an open source project that allows system integrators and developers to include performance testing in their automated build systems

9 GeoNetwork User Interface Background GeoNetwork opensource is a catalogue application that enables the management of spatially referenced resources. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. GeoNetwork opensource has been developed to connect spatial information communities and their data using a modern architecture, which is at the same time powerful and low cost, based on the principles of free and open source software and international open standards for services and protocols. With this in mind ANZLIC as well as various government agencies within Australia and New Zealand have selected GeoNetwork as the basis for a common framework for the creation, maintenance and exchange of spatial metadata. Shortfall During analysis of GeoNetwork prior to adoption, a number of shortfalls in the project were identified and remedies sought. One of these shortfalls was the interface presented to users to create or edit metadata records. This interface presented users with an disorganised list of fields that were automatically generated by the application based its analysis of the metadata schema. While this solution proved highly flexible and extensible,, it did not provide a pleasant user experience. The ordering of the fields was nonsensical and the interface was not configurable. LISAsoft was contracted to produce an improved metadata entry interface to complement the existing interface. The new user interface provides the same highly flexible and extensible capabilities as the original, as well as providing a mechanism to allow administrators to group metadata elements into related subsets of the document, and order the elements within each subset. LISAsoft engaged the GeoNetwork community prior to and during the development of the new interface to ensure that our development work would fit into the current and anticipated design strategy of the project, and all changes were accepted back into the project.

10

11 SARIG Review PIRSA - SARIG The South Australia Resources Information Geoserver (SARIG) is a web-based application that enables users to discover, view and download spatial information relating to mineral and petroleum exploration and geoscience in South Australia. It provides mineral exploration companies with the ability to apply online for mineral exploration licences. It was developed by the Primary Industries and Resources of South Australia (PIRSA) to promote the exploration and exploitation of mineral resources is South Australia. Due to the rapid growth in mineral, petroleum and geothermal exploration in SA, the SARIG application has been growing in a reactive manner to accomodate the needs of a rapidly expanding market. In order to combat this haphazard growth and develop a strategy for the future of SARIG, PIRSA contracted LISAsoft to perform a review of the system. The Future of SARIG LISAsoft began our review of the SARIG spatial content delivery application by documenting the current system architecture, the processes in place to maintain the system and the system stakeholders. During interviews with the stakeholders, their vision for the project was determined and documented. This was done to establish a current snapshot of the system as well and to determine what future requirements are likely to be placed on the system. Once the system was well described, a benchmarking test plan was developed and executed to profile the regular and peak usage of the system and project the system usage over the next five or more years. With these metrics established, a number of alternative GIS stacks were designed to meet the needs of SARIG. These systems leveraged a variety of different components in order to provide a wide view of the available products. These alternative systems were then implemented and benchmarked using the previously developed test plan.

12 With the results for the current SARIG system as well as several hypothetical systems in hand, recommendations for the improvement of the system architecture were made to address the short-term bottlenecks in the system, as well as providing a smooth scaling patch for the anticipated growth over the next five years and beyond. Finally, with our recommendations made, a deployment/migration plan was developed to allow the smooth migration from the current system to the proposed system.

13 Canadian Geospatial Data Infrastructure Interoperability Pilot Background Traditional mechanisms for the dissemination of geospatial data involved the production and distribution of paper maps. With the digitisation of spatial data, this transfer evolved into the distribution of datasets on disk, to FTP as the Internet has grown. While these transfer mechanisms reduced the physical cost of transferring datasets, the human cost remained high due to the effort involved in monitoring datasets for new releases and identifying and integrating changes into in-house data stores. These systems required the transfer of complete datasets, resulting in versioning discrepancies between organisations and reducing the frequency of data releases to improve the manageability of the system. This approach to dataset release was very detrimental to time-sensitive data, such as development proposals and road networks, as the most current data was unavailable, hidden behind the smokescreen of scheduled releases. As Internet technologies and bandwidth have continued to improve, there has been an corresponding increase in demand for information generated by government activities. The use of this data in private enterprise and across agency boundaries has been shown to reduce the cost of redundant data collection, improve the efficiency and efficacy of decision making in agencies ranging from city planners, primary resource exploration and emergency services. The quality of any decision is only as good as the data on which it is based. The Canadian Approach The Canadian Geospatial Data Infrastructure (CGDI) encompasses all of the technology, standards, access to systems and protocols necessary to provide a harmonious webaccessible interface to all of Canada's geospatial databases. The technology component is focused on integration, ensuring that the disparate hardware and software in use across the country can work together seamlessly. Geospatial standards are based on published and mature open standards to provide the glue that will bind all components of the national spatial data infrastructure together. Access to data and services is fundamental to the infrastructure, allowing the discovery and utilisation of national datasets. The data dissemination protocols and policies have been developed to ensure a consistent legal and procedural framework is in place across Canada. Finally, the framework data, the fundamental datasets required as a baseline for the majority of common analytical tasks, have been established and agreements on the provision

14 Open of data are being implemented in all provinces and territories. The principals of the CGDI project, used to guide its development, evolution and application, are as follows: The CGDI will be based on national and international standards that are both open and shared, allowing anyone to use or modify them for any purpose. Transparent The CGDI will allow users to access data and services in a simple, unified manner that hides the complexities of jurisdictional or technical boundaries in the implementation or collection of the data. Cooperative Evolving The CGDI will facilitate cooperation and interoperability between all participating organisations. The CGDI will evolve to meet the participating organisations changing requirements and business applications for information services. Self-organising The CGDI will enable participating organisations to contribute geospatial information, metadata, services and applications without the need for central administration and without relinquishing control over the contribution. Sustainable Timely The CGDI will remain relevant to the needs of participating organisations in order to ensure its long-term sustainability. The CGDI will recommend and encourage technologies that allow the real-time access to distributed geospatial information and services. Interoperability Pilot Through the GeoBase project, GeoConnections Canada was able to address the suite of technology, standards, access to systems and protocols over six years. The next step was the integration of the products developed for GeoBase into the seamless infrastructure envisioned by the CGDI project. This Interoperability Pilot (CGDI-IP) brought together a group of public and private sector participants to develop the technology framework required to provide access to the distributed provincial and territorial datasets, developed for GeoBase, but hosted and maintained across the country by the responsible government agencies. This framework allowed live access to each remote dataset through a central WFS service capable of translating a single request into a separate request for each remote service and aggregate the results as they are returned to produce a single unified result set for the query. This permits the municipal, provincial or territorial government department responsible for the dataset to maintain control of the update and correction of the data, while still allowing live access to the most up-to-date version available in a centralised manner.

15 Work-flows were developed to allow users of the data to submit requests for corrections, additions or clarifications to the data, using tools provided through the central service. These requests were divided into feeds based on the area of authority for the dataset, and these feeds published. Data maintainers in each jurisdiction monitor the request streams and have tools available to allow them to integrate the request into the current dataset, add it outright or remove the request without modification of the dataset. These changes are then published in an update feed to allow users of the datasets to monitor the status of their requests. This mechanism complements the distributed nature of the framework and ensures that the data is maintained close to the source. Project achievements The Pilot culminated in a nationwide demonstration of standards-based products that make the real-time aggregation of disparate and distributed systems a reality. In a series of webcasts 1 broadcast for both Canadian and Australian audiences, the pilot framework was exercised to demonstrate three key use cases. Use Case: Aggregation of Services The first use case addressed the need to aggregate each of the provincial and territorial data servers on a per-request basis to allow access to the entire national dataset through a single service. This required the user to be able perform the following actions: discover a WMS or WFS service or layer using a web browser interface to the metadata catalogue select a feature dataset (available in GML) constrained by OGC compliant spatial filters, using either a browser or desktop client build and share maps between applications 1 ftp://ftp.ardec.com.au/cgdi-ip/cgdi%20interoperability%20pilot%20demo % wmv

16 Use Case: Feedback and Update Mechanisms This use case addressed the need of data users to provide feedback, in the form of change or update requests, to the data custodian regardless of the physical location of either participant. The following work-flow was used in this use case. 1. The users sketches a point, line or polygon feature over the map in a browser client. 2. The user enters attribute information about the feature. 3. The user submits the addition request. 4. The request is added to the request feed for the jurisdiction responsible for the region the feature was sketched into. 5. The data custodian reviews entries from the request feed, published as a GeoRSS feed, including the request submitted above. 6. The data custodian copies the update into the local data store, or rejects the update. 7. The data custodians acceptance/rejection details are added to the update feed. 8. The user reviews the update feed, published as a GeoRSS feed, including the results of the data custodians response to the request. 9. The user performs a side-by-side comparison of their request details, and he data custodians changes to the dataset. Use Case: Geographic Event Notification The final use case involved the visualisation of a time series of GML data. In this case, the story of a toxic gas plume was used to demonstrate the activities of emergency services to retrieve the data in a GeoRSS feed and visualise it through a web browser interface. Credits The CGDI-IP project was coordinated by GeoConnections, the custodians of the CGDI, in partnership with the Open Geospatial Consortium (OGC) under the OGC testbed process. The data used in the pilot was provided and hosted by the provincial and territorial governments of Canada with the help of Galdos, Sunertek and CubeWerx. Two different server configurations were implemented in the pilot in order to showcase interoperability. CubeWerx implemented WFS and WMS services used by the provincial and territorial data servers and developed an aggregating WFS service based on their CubeSERV products. LISAsoft, sponsored by the Australian Corporate Research Centre for Spatial Information (CRC-SI), implemented the aggregating WFS service by extending GeoServer, an open source geospatial data server. Client applications we developed to allow the search and discovery of the GeoBase data, as well as accept and submit change requests to the GeoBase system. The Carbon Project customised GAIA, their desktop GIS application, to interact with the request and update feeds as well as consuming the general GeoRSS feed required for the third use case. LISAsoft developed a web browser based interface to the GeoBase systems that allowed the creation and submission of change requests, as well as the viewing of the update feed.