Total Survey Error: Adapting the Paradigm for Big Data. Paul Biemer RTI International University of North Carolina

Transcription

1 Total Survey Error: Adapting the Paradigm for Big Data Paul Biemer RTI International University of North Carolina

2 Acknowledgements Phil Cooley, RTI Alan Blatecky, RTI 2

3 Why is a total error framework needed? Large and important errors are inevitable for Big Data Big Data are inherent noisy The total error and its sources are not well-known or studied for Big Data N all does not imply error 0 Such errors lead to erroneous inferences, predictions, conclusions, and decisions Awareness of the errors is the first step to addressing their causes and reducing their effects

4 What is a total error framework? Identifies all major sources of error contributing to data and/or estimator inaccuracy Describes the nature of the error sources and how the errors could affect inference Maps the errors onto components of uncertainty (for e.g., bias and variance) Provides insights regarding how error components affect estimation and inference Suggests methods for reducing the effects of errors on inference

5 Total Error Framework for Traditional Data Sets Typical File Structure Record # V 1 V 2 V K variables or features Population units

6 Total Error Framework for Traditional Data Sets Typical File Structure Record # V 1 V 2 V K variables or features Population units total error = row error + column error + cell error

7 Possible Column and Cell Errors Typical File Structure Record # V 1 V 2 V K variables or features Population units Misspecified variables = specification error Variable values in error = content error Variable values missing = missing data?

8 Possible Row Errors Typical File Structure Record # V 1 V 2 V K variables or features Population units Missing records = undercoverage error Non-population records = overcoverage Duplicated records = duplication error

9 Shortcomings of the Framework for Big Data Big Data files are often not rectangular hierarchically structure or unstructured Data may be distributed across many data bases Sometimes federated, but often not Data sources may be quite heterogeneous Includes texts, sensors, transactions, and images Errors generated by Map/Reduce process may not lend themselves to column-row representations. 9

10 Big Data Processing Steps that Affect Total Error Generate data are generated from some source either incidentally or purposively Extract/Transform/Load (ETL) brings all data together in a homogeneous computing environment Extract data are harvested from their sources, parsed, validated, curated and stored Transform data are translated, coded, recoded, aggregated/disaggregated, and/or edited Load data are integrated and stored in the data warehouse 10

11 Big Data Processing Steps that Affect Total Error (continued) Analyze Data are converted to information Filtering (Sampling)/Reduction o Unwanted features and content are deleted; o features may be combined to produced new ones; o data elements may be thinned or sampled to be more manageable for the next steps. Computation/Analysis/Visualization data are analyzed and/or presented for interpretation and information extraction. 11

12 Big Data Process Map Generation Source 1 ETL Extract Analyze Filter/Reduction (Sampling) Source 2 Source K Transform (Cleanse) Load (Store) Computation/ Analysis (Visualization) 12

13 Big Data Process Map Generation Source 1 Source 2 Source K Similar to data collection errors in surveys; ETLdata may be erroneous or missing; data generating Extract units may be self-selected; meta-data may be lacking or absent Transform (Cleanse) Load (Store) Analyze Filter/Reduction (Sampling) Computation/ Analysis (Visualization) 13

14 Big Data Process Map Generation Source 1 Source 2 Source K Similar to data collection errors in surveys; ETLdata may be erroneous or missing data; Errors data include: generating Extract low signal/noise units may ratio; be lost self-selected; signals; failure metadata capture; may non-random be lacking or (or non- to absent representative) sources; metadata that are lacking, absent, or erroneous. Transform (Cleanse) Load (Store) Analyze Filter/Reduction (Sampling) Computation/ Analysis (Visualization) 14

15 Big Data Process Map Generation Source 1 Source 2 Source K ETL Extract Transform (Cleanse) Load (Store) Similar to data processing stages in surveys; Analyze includes creating or enhancing metadata; record matching; Filter/Reduction variable coding, (Sampling) editing, data munging or scrubbing, and data integration Computation/ Analysis (Visualization) 15

16 Big Data Process Map Generation Source 1 Source 2 Source K ETL Extract. Transform (Cleanse) Load (Store) Similar to data processing stages in surveys; Analyze includes Errors include: specification error creating or enhancing metadata, matching, Filter/Reduction in meta-data), (including, errors coding, matching error, coding editing, data munging (Sampling) error, or editing error, data munging errors, scrubbing, and data and data integration errors. integration Computation/ Analysis (Visualization) 16

17 Big Data Process Map Generation Source 1 Data are filtered, sampled or otherwise reduced. ETL This may involve further transformations Extract of the data. Analyze Filter/Reduction (Sampling) Source 2 Source K Transform (Cleanse) Load (Store) Computation/ Analysis (Visualization) 17

18 Big Data Process Map Generation Source 1 Data are filtered, sampled or otherwise Errors reduced. include: ETL This sampling may errors, involve selectivity further errors (or lack transformations of representativity), Extract of the modeling data. errors Analyze Filter/Reduction (Sampling) Source 2 Source K Transform (Cleanse) Load (Store) Computation/ Analysis (Visualization) 18

19 Big Data Process Map Generation Source 1 Source 2 Source K ETL Extract Similar to estimation and analysis error in surveys; includes Transform weighting, modeling, (Cleanse) estimation, graphing Load (Store) Analyze Filter/Reduction (Sampling) Computation/ Analysis (Visualization) 19

20 Big Data Process Map Generation Source 1 ETL Extract Analyze Filter/Reduction (Sampling) Source 2 Source K Similar to estimation and analysis Transform error in surveys; Errors include: modeling errors, includes (Cleanse) weighting, inadequate or erroneous modeling, estimation adjustments for representativity, computation and algorithmic errors. Load (Store) Computation/ Analysis (Visualization) 20

21 Other Big Data Analysis Errors Fan, Han, and Liu (2014) show that high dimensionality leads to three analysis issues : a. noise accumulation inability to identify correlates b. spurious correlations - false discoveries c. incidental endogeneity Cov(error, covariates) These issues are a concern even if the data could be regarded as error-free. Data errors can considerably exacerbate these problems. Current research is aimed at demonstrating this. 21

22 Summary Big data can be extremely complex and subject to selectivity bias, missingness and content errors Errors that apply to surveys can also apply to Big Data, including sampling Traditional approaches for describing errors in data bases may be too simplistic Distributed and unstructured data bases processed by Map/Reduce approaches create new opportunities for errors that may vary across applications A taxonomy with standardized definitions for these errors is needed 22