Data organiza+on, data modeling, and databases

Transcription

1 Mark Schildhauer Na.onal Center for Ecological Analysis & Synthesis University of California, Santa Barbara ESA 2011 Credits: Taun.ngpanda, Anita363, Stonebird, NeilsPhotography, Rick Smit, Jschinker Data organiza+on, data modeling, and databases

2 Spreadsheets: primordial data entry tool of the digital age

3 Spreadsheets: The Good Quick on the draw (clickety-click and you re ready to fire) Always there in time (on most everyones computer) Smarter than he lets on (stats, pivot tables, VB scripts) Cleans up real pretty (graphics, fonts, colors, borders)

4 Spreadsheets: The Bad Also a fast shooter (click&fire; click&fire; click&fire) No scruples (delete row, click&fire, ctrl-x/ctrl-c, click&fire, re-sort, save) Talks a good story, but didn t get much education (e.g.

5 Spreadsheets: The UGLY Ill-mannered: takes your data prisoner; conflates raw data with summary data Gaudy: Use of visual cues color, font, borders to indicate critical metadata or other semantic tidbits Shifty: Cross-linking of worksheets sets up invisible dependencies Shiftless: Provenance is entirely lost The more complicated your Spreadsheet, the UGLIER it gets in terms of using with other software

6 Spreadsheets: Best Prac.ces Think before you collect your data Develop a MODEL for your data, and implement it Excel Use Excel to create a TABLE A TABLE has a formal definition coming from RELATIONAL ALGEBRA: a Set of VALUES organized into COLUMNS, with COLUMNS groups together in a TUPLE or ROW You can add ROWs and its still the same TABLE But if you add COLUMNS, it s really a different TABLE One COLUMN in every TABLE is identified as a KEY No two ROWs can have the same KEY, thus no two ROWS can be the same (i.e. need unique ID default is row-num; how else would you know if you have a duplicate?

7 Best Prac.ces Columns of data are consistent: only numbers, dates, or text Consistent Names, Codes, Formats (date) used in each column Data are all in one table computer works beper on single table than mul.ple small tables that require a lot of human interven.on Descrip.ve File Name

8 Spreadsheets: Best Prac.ces (SQL says it nicely): CREATE TABLE SEV_SmallMammalData (Date date, Site char(50), Plot integer, Species char(50), Weight float, Adult char(2), Comment char(50)); Name your Tables Name the Attributes in the Tables Type your Attributes Establish Key relationships among your Tables (PS don t forget to provide other CRITICAL METADATA; the above information is necessary but not sufficient ) That is Data Modeling!!

9 Best prac.ces evident here?

10 Cri.cal Metadata? Create robust metadata that is discoverable, and facilitates re- use of your data Use exis.ng standards whenever possible (FGDC/ISO 19115, Darwin Core, EML, Dublin Core, etc.) Time, Place, and ResponsibleParty always important Link to relevant thesauri and ontologies in order to beper constrain seman.cs Units needed for all APributes!!!? Provide references (URI s) to associated data catalogues, protocol/method specifica.ons, data downloads, project websites, etc.

11 A word about Formats There is no shame in using CSV (comma- separated value) format for archiving Tables: Every computer can read it It s been around forever It will be around for a long while more Databases typically require a specific applica.on to read their proprietary binary formats: DBF (DB IV), MDB (MS Access), or e.g. even XLS CSV in ASCII (watch out for dreaded LF or CRLF line endings!) to be expensive for storage, and not readily queryable

12 Terminological Soup Table = Relation = Worksheet = Data set Column = Variable = Attribute = Characteristic Row = Record = Tuple <> Observation (eek) Keys are used to Join or Merge Cell = Value = Measurement ( Observation ) Data Model = Schema Even if you are using a Spreadsheet/Worksheet, you should treat it like a Table

13 Databases & DBMS & RDBMS You can use Spreadsheets like a Database You can also store your data in R, or SAS, MATLAB, etc. and still be using it like a Database But this means you are conforming to best practices

14 So why the fancy- schmancy? Databases often ENFORCE good practice You are required to define TABLES, ATTRIBUTES, and their relationships (CONSTRAINTS) Also, Databases typically provide: Far better scalability (millions+ records) Far better features for subsetting/querying data Scripted language (SQL) with lots of arsome features Dramatic reductions in redundancy and potential errors in data entry through normalization

15 Quick start: Normaliza+on? A formal process for modeling your data Atomization (1NF) don t put compound values into single cells Separate LNAME, FNAME Separate ADDRESS into CITY, STATE, ZIP_CODE Don t proliferate COLUMNS; proliferate ROWS (go LONG, not WIDE) DATE NY LA TX Date Site Count 2001 NY LA TX 13

16 Quick start: Normaliza+on? A formal process for modeling your data (2NF) Identify entities in your data, and put these into separate tables E.g. Separate Taxonomic information and Geographic information into two tables Taxon Date Location Count Thalassoma bifasciatum 1984Jun12 Ruber 211 Thalassoma bifasciatum 1984Jul25 Ruber 1345 Thalassoma bifasciatum 1984Sep19 Ruber 976

17 What is a rela.onal database? Sample sites *siteid site_name latitude longitude description samples Samples *sampleid siteid sample_date speciesid height flowering flag comments Species *speciesid species_name common_name family order * Denotes the primary key A set of tables Relationships among the tables A command language to specify (DDL)and query (DML) these

18 Database Features: Explicit control over data types Date Site Height Flowering <dates only> <text only> < real numbers only> < y and n only> Advantages quality control performance

19 Rela.onships are defined between tables Date Site Species Flowering? Site La.tude Longitude 2/13/2010 A BOGR2 y 2/13/2010 B HODR y 4/15/2010 B BOER4 y 4/15/2010 C PLJA n A B C Mix and Match data on the fly Date Site Species Flowering? La.tude Longitude 2/13/2010 A BOGR2 y /13/2010 B HODR y /15/2010 B BOER4 y /15/2010 C PLJA n

20 SQL says it nicely: CREATE TABLE Site (Site_ID integer PRIMARY KEY, LocationName char(50), Latitude float, Longitude float, Elevation float) CREATE TABLE Phenology (Species char(50), SampleDate date, Site_ID integer REFERENCES Site (Site_ID), FloweringStatus boolean, AirTemp float) Name your Tables Name the Attributes in the Tables Type your Attributes Establish Key relationships among your Tables That is Data Modeling!!

21 Powerful Command Language called Structured Query Language or SQL (see-quel) This table is called SoilTemp Date Plot Treatment SensorDepth Soil_Temperature C R B C C R A N SQL PROJECT: Select Date, Plot, Treatment, Soil_Temperature from SoilTemp where Date = Date Plot Treatment Soil_Temperature C R B C 13.2 SQL SELECT: Select * from SoilTemp where Treatment= N and SensorDepth= 0 Date Plot Treatment SensorDepth Soil_Temperature A N

22 Spreadsheet vs. Database OK for simple, self- contained Charts,Graphs,Calcula.ons Handy for collec.ng raw data Flexible cell content type BUT: Hard to subset or sort Lack record integrity (can sort a colum independently of all others) Easy to use but harder to maintain as complexity and size of data grows (lots of repeats in records) Lacks provenance Works well with lots of data Easy to query and subset data Data fields are typed e.g., integers only are allowed in integer fields Columns cannot be sorted independently of each other (tuple) Normaliza.on reduces data entry and poten.al for error BUT: More to learn and harder to use than a spreadsheet

23 Types of Data Ø Focus here on Tabular Data Ø Other Data Types include: Ø Vector (e.g. time-series) Ø Matrix (e.g. life-history tables) Ø Hierarchical (e.g taxonomies) Ø Grid (e.g. raster) Ø Spatial Vector (e.g. point, line, polygon) Lots of specialized formats for storing and exchanging these various data types; But it s all fairly readily expressible in some tabular format

24 Conclusion } Be aware of Best Prac.ces when designing data file structures } Choose a data entry method that allows some valida.on of data as it is entered } Invest.me in learning how to use a database if your datasets are large or complex } Consider inves.ng.me in learning how to use databases if your data are small and humble } Consider inves.ng.me in learning how to use databases if you ever intend to share your data } Consider learning databases if you are < 30 y.o.

25 Handy References for BePer Data Structures Best Prac.ces for Preparing Environmental Data Sets to Share and Archive. September Les A. Hook, Suresh K. Santhana Vannan, Tammy W. Beaty, Robert B. Cook, and Bruce E. Wilson. hpp://daac.ornl.gov/pi/bestprac.ces pdf Some Simple Guidelines for Effec.ve Data Management. Elizabeth Borer, Eric Seabloom, MaPhew B. Jones, and Mark Schildhauer. Bull. Ecol. Soc. Amer., April 2009, pp

26 Mark Schildhauer Na.onal Center for Ecological Analysis & Synthesis University of California, Santa Barbara ESA 2011 Credits: Taun.ngpanda, Anita363, Stonebird, NeilsPhotography, Rick Smit, Jschinker Workflows & Other Tools

27 Data Analyses Conducted via personal computer, local server or cluster, grid, cloud compu.ng Sta.s.cs, model runs, parameter es.ma.ons, produc.on of graphs/plots etc. The line separating data manipulation from data analysis is thin or blurry, depending on your metaphorical preferences

28 Sta.s.cal Soxware R, SAS, MATLAB, SPSS, You can implement good data prac.ce in each of these Of course, these are also excellent for calculations, data analysis, quality assurance, subsetting data

29 Best Prac.ces: Analysis Reproducibility is at the core of scien.fic method can someone independently validate findings? Transparency enables others to understand how you arrived at your results Executability enables someone to re- run or re- use your analysis Conceiving of your analysis as a Scientific Workflow makes you think about how your work can be made more reproducible, transparent, and re-usable for others or yourself

30 Workflows in General Simplest form of workflow: commented scripts R, SAS, MATLAB all provide rich languages for specifying and execu.ng a huge number of data transforma.on, sta.s.cal analyses, and modeling constructs. When well- documented, this source code provides an excellent basis for reviewing, sharing, and re- doing an analysis.

31 Workflows in General Another simple form of workflow: flow chart Data import into R Quality control & data cleaning Analysis: mean, SD Graph production

32 Workflows in General Not only linear: analyses are typically mul.- step, branching, itera.ve, etc. Temperatur e data Salinity data Clean T & S data Explicit Inputs. from Transforma+ons Data import into R Quality control & data cleaning Analysis: mean, SD Graph production Data in R format Summa ry statistic s & Explicit Outputs

33 Example of SWF applica.on: Kepler Resul+ng output

34 Example SWF Tools: VisTrails

35 Conclusions about Workflows Minimally, should document your analysis (commented code; simple flow- chart) But emerging workflow applica<ons will: Link together disparate soxware packages for an executable end- to- end analysis Provide detailed informa.on about data and analy.cal provenance Facilitate re- use and refinement of poten.ally complex, mul.- step analyses Enable efficient swapping of alterna.ve models and algorithms into workflows Help automate tedious tasks or accomplish parameter sweeps

36 Mark Schildhauer & Jim Regetz Na.onal Center for Ecological Analysis & Synthesis University of California, Santa Barbara ESA 2011 Credits: Taun.ngpanda, Anita363, Stonebird, NeilsPhotography, Rick Smit, Jschinker GIS or geospa+al data?

37 Vector data: tables with spa.al info * We're familiar now with the concept of a data table - Columns as variables (characteris.cs, etc) - Rows as records (individual measurements, etc) * Now imagine that one of your variables encodes loca.ons, as either: - Points (coordinate pairs, e.g., lat/lon) - Lines (list of connected coordinate pairs) - Polygons (list of connected coordinate pairs where first pair = last pair) * Voila, you've got spa.al data! - This is called vector data

38 Vector data - GIS folks call a table with spa.al info a 'layer', and call each record a 'feature'. But there's no special magic. - Each 'feature usually corresponds to some en.ty - an individual plant (point), a river (line), a nature reserve boundary (polygon) - A layer usually groups similar en..es such as plant loca.ons/occurrences

39 Vector data - Addi.onal requirement: *geo*spa.al data also must specify its spa.al reference system, which tells us where coordinates actually lie on the earth, i.e. - datum: a specific mathema.c model of the shape and orienta.on of our lumpy, spheroid earth - projec+on: specific formula for represen.ng coordinates in 2D space (essen.al for mapping, and usually necessary for other tasks in most spa.al soxware) Also need to have informa.on about scale and origin. These are generally cri+cal pieces of metadata accompanying geospa.al data, and are typically stored along with the data in various ways.

40 * Think of a digital photograph Raster data: 2D matrices with spa.al info - a rectangular grid of cells ("pixels") each of some color - e.g. 12 megapixel = 4000 px wide by 3000 px high (if 4:3 aspect ra.o) * Now imagine the photograph was taken of earth from a satellite. We might _see_ features (e.g., a sandy desert plain sloping up to some mountains), but the data only encode values for pixels

41 Raster data: 2D matrices with spa.al info - We also need to know 6 numbers: - the coordinate pair (e.g., lat/lon) of the upper lex corner - size of pixels (in terms of distance on earth) in both X and Y dimension - number of pixels in both X and Y - as with vector data, we need to know other spa.al reference informa.on (e.g. datum) to know how the coordinates map onto the earth - then we can calculate what patch of earth corresponds to each pixel Voila, you've got spa.al data! - This is called raster data In the computer, just a matrix of numbers

42 Geospa.al Data Formats - Vector: by far most common is shapefile, but there are others - Raster: GeoTIFF is probably best default choice (= TIFF image format with spa.al reference info stored in internal metadata tags), but many other formats (including ASCII Grid format which simply stores space- delimited matrix below a 6- line header)

43 Some free, open source geospa.al sw * QuantumGIS (hpp://qgis.org/) - user- friendly graphical applica.on - read many kinds of vector/raster formats * R (see hpp://cran.r- project.org/web/views/spa.al.html) - (v,r) 'sp': provides consistent R representa.on of vector and raster data - (v) 'maptools', 'rgdal': read/write/convert various vector data formats; reproject with rgdal - (v) 'rgeos': lots of topographic opera.ons (distance, containment, inteersec.on, buffer, centroid, etc) - (v) 'geosphere': distance, centroid, midpoint, etc using lat- lon directly on spheroid - (r) 'raster': read, write, do lots of things with raster data

44 More * GDAL/OGR (hpp:// - powerful commandline tools for raster (GDAL) and vector (OGR) - "swiss army knife" for spa.al data - quickly reformat, reproject, transform, subset, summarize, etc GIS, Geographic Informa.on Systems, can be invaluable for rapidly accessing, querying, manipula.ng and visualizing geospa.al data. But remember, basically geospa.al informa.on is just another type of data

45 Conclusions about GIS GIS, Geographic Informa.on Systems, can be invaluable for rapidly accessing, querying, manipula.ng and visualizing geospa.al data. But remember, it is useful to remember that, basically, geospa.al informa.on is just another type of data, and the capabili.es to work with them are becoming available in a growing range of analy.cal soxware solu.ons.

46 FIN How many fingers, Winston? Orwell,