Decontextualizing + Assumptions = Fallacies: And It s Worse for Big Data

Transcription

1 Decontextualizing + Assumptions = Fallacies: And It s Worse for Big Data Michael Smithson Research School of Psychology The Australian National University

2 Setting the Scene This talk presents several commonplace myths about public data and problematic assumptions regarding models of such data. These problems are intensified and less corrigible in big data. The myths and assumptions fall under three headings: 1. Big data have integrity 2. Models of big data extract patterns that reflect underlying truths 3. Big data are better than small data

3 Big data have integrity? Accuracy? Data may be subject to recording distortion, recording errors, and/or measurement confounds. These may be worse for big data because big data often is an assemblage of multiple second-hand data sets taken out of context and used for purposes other than those originally intended. Distortion and error : Impact of shadow economies on official economic indicators (e.g., employment rates, inflation) Gaming the indicators (e.g., Australian universities and the ERA) Making it all up (e.g., the recent Canberra hospital records scandal) Measurement confounds: Differing or shifting criteria (e.g., definitions of crime, suicide in Catholic populations) Measurement contamination (e.g., webpage number of visits and dwell-times)

4 Big data have integrity? Precision? Big data often are sample data rather than population data, and the samples may not be representative of their referent populations. Nevertheless, decision makers and policy analysts usually treat sample data or estimates as though they are population data or estimates. Stability? Data often are not recorded just once, but re-recorded as better information becomes available or as errors are discovered. For example, in November 2012 the first official estimate of U.S. net employment increase was 146,000 new jobs. By the third revision that number had increased by 68% to 247,000. Completeness? Data collection schemes often are set up by groups who lack the necessary expertise. The Australian Transport Safety Bureau, e.g., collects lots of data on civil aviation flights that have resulted in an incident, but collects no data on incident-free flights.

5 Models of big data reveal underlying truths? Big data increasingly require automated data analysis, i.e., data mining. Data-mining has no quality control, beyond the assumptions built into its algorithms and post-hoc interpretations by humans. Spurious correlations There is no guarantee that a pattern (e.g., a correlation between two variables) uncovered by data-mining is meaningful or useful. Other unmeasured factors may render those correlations spurious. Autocorrelation may account for an apparent correlation over time. However, humans will make sense out of nearly anything.

6 Terms for uncertainty fall out of fashion in English-language books: As the plot below demonstrates, the terms ignorance", ignorant", unknown", uncertain", and falsehood" display a steady decline in relative frequency of occurrence in GoogleBooks, starting around 1830, and ending with a slight upturn at the start of the new century. logit Ignorance Ignorant Unknown Uncertain falsehood year

7 Terms for uncertainty fall out of fashion in English-language books: What is driving this? Could it be God? logit(god) year

8 Terms for uncertainty fall out of fashion in English-language books: It looks plausible; the correlations are very high. God Ignce. Ignt. Unkn. Uncrt. Falsh God Ignorance Ignorant Unknown Uncertain Falsehood Also, a search through the books containing such references reveals a potential link between mentions of these terms and references to God in the context of theological arguments.

9 Terms for uncertainty fall out of fashion in English-language books: However, The partial autocorrelation functions below suggest that all of these variables are strongly autocorrelated (AR(1) or AR(2) processes).

10 Terms for uncertainty fall out of fashion in English-language books: When autocorrelation is taken into account, the residual series no longer display strong correlations. The original correlations were inflated due to autocorrelation. God Ignce. Ignt. Unkn. Uncrt. Falsh God Ignorance Ignorant Unknown Uncertain Falsehood

11 Models of big data reveal underlying truths? Correlation ain t causation, but what if there s a time-lag? People tend to attribute causal status to X if X always precedes Y, and especially if they would like to attribute Y to X (e.g., my AFP experience). Nonstationarity Most data-mining procedures assume that the processes generating the data are stable over time (i.e., stationarity). This often may be untrue, and the changes in those processes unaccounted for.

12 Big data are better than small data? Big data can be big in at least three senses: Large samples, lots of different pieces of information, and more intense surveillance. These may not be unalloyed goods. Belief in small numbers Large samples indeed do give more accurate and precise estimates than small samples, ceteris paribus. But most non-statisticians don t know or understand this. Instead, they over-estimate how representative of a population a small sample is. More information can make us worse decision makers Several experimental studies have shown that people are more confident and make worse predictions when given additional, but irrelevant, data. This is especially problematic for unstructured fishing expeditions for data.

13 Big data are better than small data? Bigger data yield better models for predicting the future? One of the potential advantages of big data is long memory. Longer-term data-sets should give us a better model of the past, which in turn should enable us to better predict the future. But does it? Model inflation and nonstationarity pose problems here. Bigger data result in greater control? Bigger data clearly won t yield better control if underlying causes are not understood and/or are not malleable by us. Surveillance and data-gathering about people may destroy trust and privacy (which are means of control as well as social capital), and also may engender reactance among those under surveillance (and thus a loss of control).