Andrew Fayram Wisconsin Department of Natural Resources. Monitoring Coordinator

Transcription

1 Cth Catchy title! But not mine. Making Sense of Too Much Data Andrew Fayram Wisconsin Department of Natural Resources Office of the Great Lakes Monitoring Coordinator

2 Cth Catchy title! But not mine. Making Sense of Too Much Data Technology, Entertainment, and Design (TED) Talks Making sense of too much data (10 talks) It's the age of Big Data. But what, exactly, do we do with all this information? These talks explore practical, ethical and spectacularly visual ways to understand near infinite data.

3 Wikipedia Definitions Big data: data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process thedata within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set

4 Wikipedia Definitions Informatics: the science of information, the practice of information processing, and the engineering of information systems. Informatics studies the structure, algorithms, behavior, and interactions of natural and artificial systems that store, process, access, and communicate information

5 Wikipedia Definitions Analytics: thediscovery and communication of meaningful patterns in data. Especially valuable in areas rich with recorded information, analytics relies on the simultaneous application of statistics, computer programming, and operations research to quantify performance

6 Rate of Data Acquisition Lots of current databases are pretty big and will increase exponentially with the addition of real time data inclusion Sloan Digital Sky Survey: 200 GB of information per night!

7 Analysis of Big Data We can throw the numbers into the biggest computing clusters the world has ever seen and ltttiti let statistical algorithms find patterns where science cannot. We can analyze the data without hypotheses about what it might show. Chris Anderson Editor in Chief Wired Magazine Discovery science enumerates the elements of a system irrespective of any hypothesis on how the system functions. It is based on scales of complexity and volume of unprecedented in biological sciences, defying analysis by normal means of interpretation, presentation, and publication, discovery science depends on the integration of computational tools to store, model, and disseminate these exploding cascades of information Aebersold et al (Nature Biotechnology)

8 Some existing environmental databases: do we have big data? 1) WNDR Fisheries Management Database: 2,000,000 records 2) WDNR Watershed Monitoring (SWIMS Database): 8,000,000 records 3) ebird Database Cornell Lab of Ornithology oogyand National Audubon Society : 125 million records, 30 GB 4) Water Quality Portal (USGS + EPA): 200 million records, 300GB

9 Hypothesis Driven Science Ho: Phosphorus levels in the Sheboygan AOC are not below the target level Phosphorus levels within the Sheboygan Area of Concern (Esslingen Park) k)

10 Discovery Science? GLRI Action Plan: Focus Area 5: Accountability, Education, Monitoring, Evaluation, Communication and Partnerships This is an existing measure under the Government Performance and Results Act. The Great Lakes Index uses select Great Lakes ecosystem indicators (i.e., coastal wetlands, phosphorus concentrations, AOC sediment contamination, benthic health, fish tissue contamination, ti beach closures, drinking di water quality, and air toxics deposition) ) and is based on a 1 to 5 rating system for each indicator, where 1 is poor and 5 is good. Improvements in the index and measures would indicate that fewer toxics are entering the food chain; ecosystem and human health is better protected; fish are safer to eat; water is safer to drink; and beaches are safer for swimming.

11 Big Data TEDx Talk Dr. Peter Dodds PhD Massachusetts Institute of Technology Associate Professor at the University of Vermont College of Engineering and Mathematical Sciences vi&feature=player_detailpage

12 Couple of thoughts that struck me A hypothesis in a big data world is, this is interesting. An approach to big data is not to even ask questions but to find out what the questions should be.