Oklahoma s Open Source Spatial Data Clearinghouse: OKMaps Presented by: Mike Sharp State Geographic Information Coordinator Oklahoma Office of Geographic Information MAGIC 2014 Symposium April 28-May1, 2014 Kansas City, MO
Why OKMaps? Sub-section D-3 of OS 82:1501.205.3 : Establish a central statewide geographic information clearinghouse A definite need existed for a central, authoritative source of geospatial data but no funding was available Growing awareness at OK Office of Homeland Security that GIS can help fulfill missions In 2008 funding became available through OK OHS from US Department of Homeland Security to support geospatial data development and distribution
System Goals Low or no ongoing maintenance costs Not technology or software-specific No license fees Open Geospatial Consortium (OGC) Web Service Standards (WMS, WFS, WCS, KML,CSW, SQL) Core software code is extensible and open- Java,JavaScript, AJAX, ASP.NET, HTML, CSS Integrated Catalog, Metadata Manager and Search Engine Output for Google Earth visualization Point, Line, Polygon, and Raster (including floating point for DEM/LIDAR) support for storage, view, download, and web services Download by area (state, county, USGS quad, PLSS, visible extent) with many data format choices in any commonly recognized projection Support for frequently updated layers (vehicle tracking, weather radar) Security (Data and system functionality limited by Role/User) Scalability
High-Level Architecture Clients Google Earth Users Web Service Consumers Data Downloaders Data Catalog Data Layer Selection Metadata OGC Standard Web Services (KML, WMS, WFS, WCS) Data Storage
Architecture Detail Google Earth KML Download SHP,DWG MAP etc Web Service Client WMS WFS WCF Feature Input/ Update Data Catalog Data Layer Selection Metadata Inter-layer analysis (eg buffer etc.) Raster Tiling or Caching (optional) File Download Generator KML, WFS Feature Server WMS Server WCS Server Data Conversion Raster Storage Vector Storage Raw Data
Open-Source Architecture Web Server GeoServer GeoWebCache Catalog GeoNetworks Database PostgreSQL/PostGIS Data Translator GeoTools-GDAL/OGR GUI-Viewer OpenLayers
Clearinghouse Clients Google Earth KML Web Service Client WMS WFS WCS Download SHP, CSV KML etc Feature Input/ Update Google Earth KML User will select different layers (both raster and vector) from the catalog. These will then appear in GE. It is possible to have GE request updates every few seconds or minutes to display realtime data. GE handles all the network connections and layers once they are set up. Web Service Client A web service client, such as OpenLayers, ArcMap, AutoCAD Map, Global Mapper, QGIS, udig, and many others make standard WMS, WFS, WCS requests to the warehouse and gets back data. The user will first select layers from the catalog, and add these to their client. Then, the client will talk directly to the WMS, WFS and WCS server(s).
Clearinghouse Clients (Continued) Google Earth KML Web Service Client WMS WFS WCS Download SHP,CSV KML etc Feature Input/ Update Download SHP, CSV, KML, etc Users that need downloaded files in a variety of formats, projections, datums, etc, will first select the data layers, formats, options and AOI using the catalog. The request is passed to the file download generator which returns the file for download. Feature I/O Client Adds and updates features in the warehouse. This can be done using a feature client like OpenLayers, QGIS, TinyOWS, or another client that supports WFS-T, or input-output of KML. The user would select an updateable layer from the catalog, then the software would talk directly to the feature I/O server.
Clearinghouse Data Catalog Data Layer Selection Metadata Data Warehouse Catalog The catalog is the central location where users choose their data from the warehouse. Metadata about the geographic data within the warehouse is presented in an easily searchable, easy to read fashion. The data can then be previewed on a map within the catalog. Once the user is happy with their choice, they can choose the method of receiving the data, in Google Earth, files, or via web services. The catalog has a secure way of allowing a fine degree of access rights depending on userid/password or other access control method. Metadata is published to FGDC standards. GeoRSS subscription service is available to notify user when new data layers are added or existing ones updated. Open Source: these tasks. an open source Java server that does most of
Clearinghouse WMS Server WMS Server A WMS (Web Map Service) server creates images of the underlying vector or raster data according to a standard request. The images can then be displayed in most clients, including Google Earth, OpenLayers, ArcGIS, AutoCAD Map, udig, and QGIS. In addition, WMS images are used in the Catalog to show previews of the warehouse data. Open Source: an open source Java server provides this service. - a Java web application used to cache tiles from the WMS that is bundled with GeoServer.
WFS Server Clearinghouse WFS Server A WFS (Web Feature Service) server returns the actual source of the underlying vector data according to a standard request. This raw data can then be displayed and analyzed in most clients, including OpenLayers, ArcGIS, AutoCAD Map, udig and QGIS. Open Source: an open source Java server provides this service.
Clearinghouse WCS Server WCS Server The OGC Web Coverage Service (WCS) defines a standard interface and operations that enables interoperable access to geospatial grid coverage's". The term "grid coverage" typically refers to content such as satellite images, digital aerial photos, digital elevation data, and other phenomena represented by values (color, intensity, elevation) at each measurement point (pixel). Clients include OpenLayers and Global Mapper. Open Source: -an open source Java server provides this service.
Clearinghouse File Download Generator File Download Generator This component constructs downloadable vector and raster files in a wide variety of formats, projections, and options. Given the user's requirements, it creates the files, then makes them available for download. This component can read from both the underlying raster and vector storage, as well as from WFS, KML, WMS and WCS sources. Open Source: -an open source Java library coupled with GDAL/OGR libraries provides this functionality.
Vector (Feature) Storage Clearinghouse Vector Storage Vector features can be stored either in flat files (e.g. SHP files), or in a geodatabase. A geodatabase has a number of advantages over flat files, such as built-in analysis and query tools that make them much preferable in a data warehouse application. Additionally, a geodatabase is not fragmented into many different files. Interoperability: If the geodatabase supports standards like the OGC SQL standards, one geodatabase can be substituted for another if required. Open Source: the spatial-enabled database used worldwide in many applications.
Clearinghouse Raster Storage Raster Storage Large raster images can be stored as flat files in a variety of formats, such as GeoTiff, MrSID, JPEG2000, ERDAS, JPEG. There are also ways to store them in databases, or in proprietary servers. Besides the image data itself, each image file needs to have metadata information that describes the image, such as spatial location, projection, date, source. Aerial photo/drg/dem images in OKMaps are stored as jpeg compressed GeoTiff quarter-quad files bound together in mosaic coverage's generated by GeoServer. LiDAR LAS data is stored in a database utilizing PDAL (Point Data Abstraction Library) and the pgpointcloud extension for PostgreSQL developed by Paul Ramsey in early 2013. Data is stored in the database as patches where a patch = 1 row and a patch consists of 400 LiDAR points. Currently there are 171.9 Million rows in the database representing approximately 2.5 TB of data.
Clearinghouse Data Conversion Raw Data Data Conversion of Raw Data Data for the warehouse will need to be converted into common formats for storage. The same tools that are used for the file download generator can be used for this. Additionally, standardized metadata needs to be tracked, and added to the catalog for the data as it is input into the warehouse. Open Source: control software. - -GDAL/OGR with some custom for metadata creation/maintenance.
Clearinghouse Raster Tiling or Caching Raster Tiling/Caching This component provides faster access to raster data. Many client software, including Google Earth, and OpenLayers can consume standard sized image tiles much more quickly and seamlessly than arbitrarily sized images. A tiling/caching system can increase the perceived speed of both WMS servers, and raw imagery access. In particular, Google Earth provides a way to consume tiles that appears much more seamless than access to WMS servers. Open Source:
Clearinghouse Inter-layer analysis (eg buffer etc.) Inter Layer Analysis It may be useful to include a component that allows for inter-layer analysis. For example, you may have a road feature that comes from one WFS server, and you need to find all schools that are within 1km of this road which is on a KML server (buffer analysis). It is possible to provide an intermediate analysis component that has access to both servers and can perform the operation. Possible Software: Custom software using tools such as GDAL/OGR and a GeoDatabase like PostGIS or new OGC-WPS extension to GeoServer.
Clearinghouse http://ogi.state.ok.us/ogi/search.aspx