TOWARDS VIRTUAL WATERSHEDS: INTEGRATED DATA MINING, MANAGEMENT, MAPPING AND MODELING

Transcription

1 AWRA 2010 SPRING SPECIALTY CONFERENCE Orlando, FL March 29-31, 2010 Copyright 2010 AWRA TOWARDS VIRTUAL WATERSHEDS: INTEGRATED DATA MINING, MANAGEMENT, MAPPING AND MODELING Yang Cao, Daniel P. Ames, and Ping Yang ABSTRACT: As the amount of satellite telemetry, remote sensing and other automated data collection systems increases, a great quantity of geospatial data is being accumulated in online spatial databases and spatial information repositories. This data is accessed using standards-based web services such as those defined by the Open Geospatial Consortium (OGC). Additionally, a growing quantity of climate and hydrologic point observation data is being stored and distributed in standards-based web systems and services such as defined by the Consortium of Universities for the Advancement of Hydrologic Sciences (CUAHSI). These developments, together with the rapidly increasing availability of open source GIS and modeling software tools, represent the beginning of a truly open and accessible standards-based virtual watershed system which could be used for education, research, and management purposes. This paper explores the fundamental requirements of such a system including data mining through on-line analytical processing (OLAP), data management using well defined server side and client side data management tools and standards for both spatial and temporal data, visualization through both web-based and client side open source mapping systems, and modeling using OpenMI as a model interoperability standard. KEY TERMS: HydroDesktop; open source; data filter; data cube; OLAP. INTRODUCTION As the use of satellite telemetry systems, remote sensing systems and other computerized data collection tools increases, so too does the amount of spatial data stored in geographic information systems, spatial databases and other spatial information repositories. It has become necessary to develop efficient methods to analyze, understand, and effectively utilize such huge amounts of spatial data. As a crucial task in information technology, especially data mining technology, the OLAP (On-Line Analytical Processing) phenomenon is interesting from both an academic and an industrial point of view. The traditional analysis methods of water resources are limited, to a certain extent, because water resources data involves a wide range and is vast, distributed, and heterogeneous. However, the application of data mining technology can fully utilize water resources data, such as water resource management, evaluation, forecasting, dynamic simulation, etc. In the case of Hydrologic Information Systems (HIS), it is helpful to users if the systems can not only provide plentiful and useful information by querying and viewing the geographic map, but also that they can determine the underlying rules. As such, research focused on designing and implementing OLAP hydrologic data is needed to aid HIS users in the decision making process. Some scholars have already applied OLAP technology to Hydrologic data. OLAP technology has been applied to environmental data catalogs using SQL Server 2008 Analysis Services, and the query results can be visualized in both Excel and Virtual Earth (1). Also, the Map Maker application leverages OLAP technology to quickly perform a summary analysis and then generates a map to reflect the analysis (2). However, most studies concerning OLAP, rely on the Microsoft SQL Server, and the function of information visualization related to mapping needs to be improved. This paper discusses how to filter downloaded data, how to analyze this data to obtain useful information using OLAP techniques, and the different methods of query result visualization. Also, two plug-ins that have been developed for the free and open source desktop application, HydroDesktop, will be presented. Respectively, Ph.D. Student, Liaoning Technical University, Department of Surveying and Mapping, No. 47 Zhonghua Road, Fuxin, China ,Phone:(208) , caoyang@isu.edu; Associate Professor, Idaho State University, Department of Geosciences, Idaho Falls, Idaho 83401, amesdani@isu.edu; and Post-doc Researcher, Idaho State University, Department of Geosciences, Idaho Falls, Idaho, yangping@isu.edu. 1

2 METHODOLOGY The proposed method for obtaining data from the CUAHSI HIS WaterOneFlow web service can be broken down into two basic steps: 1) A Data Filter, which uses SQL language to help users select the data they downloaded according to their interests. 2) An Information Presenter, which implements OLAP technique to help users better understand the data they downloaded. Structured Query Language Once observational data have been retrieved and downloaded, the Data Filter Tool will provide users with functions to quickly access attribute tables and select the desired data and then visualize and store the data. This tool is useful for narrowing the list of data series for download. Three options to filter the data are needed, namely: a simple filter option, a complex filter option and an all filter option. Each option applies SQL language to manage the data. The workflow for the Data Filter Tool using SQL languages is shown in Figure 1. Figure 1. Workflow of Using SQL in Data Filter On-line Analytical Processing OLAP is an abbreviation of On-line Analytical Processing, which is not simply the name of a particular product, but the name of real technology (3). OLAP is a category of software technology that enables analysts, managers and executives to gain insight into data through fast, consistent, interactive access to a wide variety of possible views of information that has been transformed from raw data to reflect the real dimensionality of the enterprise as understood by the user. The main object in OLAP is the cube, which contains the current analytical data of interest to the end user. To support the questions which the user asks, the cubes organize the data in dimensions and measures in multidimensional structure. For example, What were our general hardware sales in the North - eastern region over the first quarter of this year? The data cube, which answers this question, includes three dimensions and one measure: 1) The Product dimension, which contains the category of the hardware. 2) The Geography dimension, which contains the North-Eastern region. 3) The Time dimension, which contains the first quarter of this year. 4) Measured sales, which contains quantitative numeric data that can be summarized. RESULTS The Data Filter Tool is developed as a plug-in of HydroDesktop. It contains three filter options. The Simple filter (Figure 2) option allows the user to choose data by six criteria, i.e., Theme, Site, Variable, Source, Method and Number of Observations. The Complex filter (Figure 3) option can execute an SQL statement to find the matched data by using a Query Builder (Figure 4). The All filter (Figure 5) option shows all the data in a database. What is more, the user is provided with 2

3 two ways to view an attribute table: Show All Fields in Sequence and Show Just Values in Parallel. The tool also has a right click context menu with the following options. 1) PropertyGrid, can show detailed attribute information of data series. 2) Save to Theme, saves the selected set of series to a new theme in the database. 3) Export, exports the selected set of series to a *.txt file. Figure 2. Simple Filter Option Figure 3. Complex Filter Option Figure 4. Query Builder Dialog 3

4 Figure 5. All Option The HydroAnalyzer plug-in, which aims to use OLAP cube (HydroCube) to retrieve important information from downloaded data, is now under development. At present it is a local computer OLAP tool rather than a web-based version. HydroCube is a multidimensional database stored in a single file. It contains compressed data and report layout descriptions that can be used for delivering reports on personal databases. While building the HydroCube, a shape-file that is created when data is downloaded using HydroDesktop, serves as an input. One date type field can be chosen for each time series, which are represented as dimensions in HydroCube. Totals and subtotals will be calculated within these dimensions. Only numeric fields can be used as values for the time series. As such, all qualitative values need to be aggregated and converted to a numeric field. A case study of water use in the United States has been conducted using HydroAnalyzer. The source data is shown in Table 1. OLAP was implemented with the Time (Year 2000) and Area (States) dimensions, the results are presented in Figure 6. The source data and summarized information can be viewed in the Data View tool (Figure 7). Table 1. Sample Data of Water Withdrawals by Water-use Category in 2000 and 2005 RecordID StateID RegionID WaterUseType Values (thousand acre feet) YearID DecadeID 1 Nebraska Region VII PUBLIC_SUPPLY s 2 Nebraska Region VII IRRIGATION s 3 Nebraska Region VII INDUSTRIAL s 4 Nebraska Region VII THERMOELECTRIC POWER s 5 Nebraska Region VII OTHER s Figure 6. OLAP Result 4

5 Figure 7. Data Viewer CONCLUSIONS Both hydrologic data filter and analysis tools have been developed to obtain useful data from spatial information databases. The OLAP technique was applied successfully to acquire information from downloaded hydrologic data. Compared to prior methods, this approach builds the OLAP cube more efficiently with a high performance standard, flexible step by step implementation, and query results visualized directly in maps with charts. ACKNOWLEDGMENTS The following individuals provided technical support for this study: Daniel P. Ames, Ping Yang, Jiří Kadlec, Harold (Ted) Dunsford Jr., Mark Van Orden, Dinesh Grover and Tevaganthan Veluppillai. I would also like to express my appreciation to those who reviewed this paper for AWRA. REFERENCES Bora Beran et al, OLAP Cube Visualization of Environmental Data Catalogs. Microsoft Technical Report MSR-TR Accessed Feb Matt Rodriguez, Map Making Web Application. Accessed Feb OLAP. Accessed Feb